"Gentrification is just the fin above the water," the San Francisco writer Rebecca Solnit once warned of the changes to her home town. "Below is the rest of the shark." The shark being a "hollow city" with an economy where "most of us will be poorer, a few will be far richer, and everything will be faster, more homogenous and more controlled or controllable".

Just over a decade later, her fellow citizens have been on the streets blocking Google's private buses, which ferry the company's workers from their expensive downtown pads to Silicon Valley offices. The tech industry rich have, it is argued, priced everybody else out of the city.

Gentrification is a western world phenomenon: brogue-heeled Brooklyn hipsters tarting up first Williamsburg and now Bushwick, leaving better coffee in their wake; Parisian bobos (bourgeois bohemians) pushing steadily eastwards from the Marais to Bastille to the 20th. Sydneysiders, meanwhile, are reclaiming the "Paris end of Macleay Street" from the backpacker brigade.

New Yorker Jane Jacobs would have been astounded. Her 1961 book The Death and Life of Great American Cities was a call to arms, a desperate defence of the positive qualities of urban villages in the face of the modernist's bulldozers that were intent on flattening large areas of Manhattan. Its neighbourhoods became drug- and crime-ridden ghettoes as "white flight" saw large corporations relocate to the suburbs, leaving the city close to bankruptcy by the 1970s. Similar protests were unable to save UK cities such as Manchester from flattening much of its historic inner area in the name of slum clearance.

Unlike the neutral-sounding "regeneration", however, gentrification has always had its negative connotations – which is why it is a term avoided by contemporary developers. The word was first coined in 1964 by Marxist planner sociologist Ruth Glass, for whom the rescue of Notting Hill and Islington streets by "pioneering" London bohemians with the cash to do up attractive old houses that banks wouldn't lend on went hand-in-hand with the displacement of long-standing, blue-collar communities who could no longer afford to live there. She identified the supplanting of net-curtained cheap lodgings by owner-occupiers' carefully stripped floorboards as a class struggle played out in three dimensions.

By 1988, rioters in New York's Tompkins Square Park were carrying placards reading "Gentrification is class war".

Gentrification's defenders have argued that repopulating the inner cities has been good for all, creating sustainably dense neighbourhoods that are not car-reliant, saving our architectural heritage, rebuilding derelict sites and introducing articulate new residents who then press for improved schools and services for all locals – rich or poor – in a kind of trickle-down aspiration.

Back in 2003, Columbia academic Lance Freeman argued that his quantitative research in Harlem shows that even low-income renters stay put and benefit from changes such as lower crime rates. Some of this, at least, may be true, but why should it take a middle-class invasion to improve poor people's environments?

Freeman's findings have been rediscovered recently by New York commentators in favour of gentrification, leading to this week's outburst by film-maker Spike Lee at an event in Brooklyn. "Why," asked the Brooklyn-born Lee, "does it take an influx of white New Yorkers in the south Bronx, in Harlem, in Bed Stuy, in Crown Heights for the facilities to get better?

"The garbage wasn't picked up every motherfuckin' day when I was living in 165 Washington Park. PS [Public School] 20 was not good. The police weren't around. When you see white mothers pushing their babies in strollers, three o'clock in the morning on 125th Street, that must tell you something."

Even if Freeman's claims are accurate, the crucial caveat here is that New York has many rent-controlled or rent-stablised apartments (in 2011, just over 47% of NYC apartments offered some form of rent protection [PDF]). In the UK, such controls have long been abolished, and Margaret Thatcher's right-to-buy scheme saw ownership of the best council-owned street properties shift en masse to the private sector. Precious little affordable housing has been built to replace these losses, and even Brutalist council estates such as Sheffield's Park Hill or London's Robin Hood Gardens are now being part-privatised in the name of regeneration.

This demonstrates nicely that you don't need Georgian mouldings or Victorian stained glass to find something to gentrify. Since the first conservation areas in England were designated under the Civic Amenities Act 1967, heritage has become a whipping-boy for those arguing that conservation favours the rich. If it does, that is not the down to the protection of architectural history per se, but to a property market that now places a premium on heritage properties after previously decrying conservation's restrictions on change. Where property values remain low, such as Liverpool's Welsh Streets, entire 19th-century neighbourhoods are still under threat of demolition.

With London, on the other hand, in the throes of "super-gentrification", fuelled by foot-loose international property investments, some of its outer suburbs are getting poorer as higher rentals spin the low-paid outwards. Government policies such as housing-benefit caps and the bedroom tax can only fuel this centrifugal force, while rising land values make social housing provision ever more difficult.

This drives out both the low-paid workers and the initial pioneers – the artists, designers and young entrepreneurs – who helped save our historic inner cities in the first place. It is a future that has long-term consequences for creative- and knowledge-based economies.

And it is happening faster than ever before. The vitally productive time-gap between artists using their own "sweat equity" to create studios in empty industrial buildings and their marketing as lofts to hedge-fund managers has dwindled from decades to a matter of months.

Collecting his 2014 gold medal awarded by the Royal Institute of British Architects last month, the historian and theorist Joseph Rykwert argued that "the price of property in city centres is making it impossible, particularly in the big cities, for any kind of social mix to take place. It's castrating the whole notion of city life."

It is entirely possible that the capital will end up resembling Paris or Sydney – where social problems and poverty are, on the whole, confined to the very edge of the city, and creativity is stifled by noise- and mess-averse new residents. Glass predicted as much: "London may soon be faced with an embarrass de richesses in her central area," she wrote in 1964, "and this will prove to be a problem too." The sharks are circling.

Or, as Lee so pithily put it: "There were brothers playing motherfuckin' African drums in Mount Morris Park for 40 years, and now they can't do it anymore because the new inhabitants said the drums are loud. My father's a great jazz musician. He bought a house in nineteen-motherfuckin'-sixty-eight, and the motherfuckin' people moved in last year and called the cops on my father … He doesn't even play electric bass! It's acoustic! Get the fuck outta here!"

• Robert Bevan is a writer on architecture and cities, and a regeneration consultant.