If there’s a supernatural estate agency in the afterlife, the ghost’s dream home would probably be something like the Winchester Mystery House in San Jose, California. It was built in 1884 by Sarah Winchester, heiress to the rifle fortune and, legend has it, she was told by a medium that it had to accommodate the spirits of those killed by her late husband’s guns. Thus, she employed an army of builders to continually extend the house for 40 years to no particular plan. You couldn’t wish for anything spookier: stained glass, odd-shaped roofs and turrets, corridors and staircases that lead nowhere, doors that open on to nothing. It makes no sense at all.

Before you say: “Someone should make a movie out of that!” they already have. It was called Winchester. Starring Helen Mirren, it came out last year and was rubbish. But the point is, by accident or design, houses are the scariest places of all. The movies know this very well. Just as old houses are adaptable to modern lifestyles, so haunted-house stories have accommodated our generational anxieties. And just as we prefer not to think about who lived (and died) in our house before we moved in, so haunted-house movies find a way of tapping into older, deeper fears. “When we go home and shoot the bolt on the door, we like to think we’re locking trouble out,” wrote Stephen King in his book Danse Macabre. “The good horror story … whispers that we are not locking the world out; we are locking ourselves in – with them.”

Those ghosts of terrors past are very at home in Hereditary – a terrifying new horror whose title hints at things we want to inherit (property) and things we don’t (the family “curse”). Anchored by a terrific performance from Toni Collette, Hereditary is a magnificent, mysterious tale of grief and guilt and dread. Everyone in this family seems to be shrouded in their own personal cloud: Collette has just buried her “difficult” mother; husband Gabriel Byrne is withdrawn, morose; the teenage son is a monosyllabic stoner; the young daughter scribbles angry drawings and snips the heads off dead birds.

The family home itself adds to the gloom: a large, wooden affair surrounded by birch forests, apparently in the middle of nowhere. Architecturally, it’s not your classic gloomy gothic – it’s more 20th century – but it is still a place of unease. It’s imposing, cavernous and too quiet, and severely short on lighting fixtures. Driving the point home, Collette is an artist specialising in detailed little scenes from her own life recreated using tiny model houses. Like Sarah Winchester, she plays out her neuroses architecturally.

This was one of the foundations of gothic horror: that the house was a reflection of its inhabitants’ troubled psyches and repressed secrets. Away from the presentable public areas, those dark corners and secret basements and attics corresponded to the spaces of our unconscious. Women are usually the victims, locked into prisons of the patriarchy, haunted by past wives and stern governesses and curtailed passions, as in Jane Eyre or Rebecca or Shirley Jackson’s The Haunting of Hill House, the basis for Robert Wise’s classic The Haunting.

So many of our haunted-house movies are still rooted in this gothic tradition, such as Guillermo del Toro’s Crimson Peak or the Daniel Radcliffe-led The Woman in Black, or Nicole Kidman’s The Others. Even Polanski’s Rosemary’s Baby, set in 1960s New York, still found a pocket of gothic gloom in the Dakota Apartments, where the patriarchy works its satanic tricks on poor, waif-like Mia Farrow. The urban familiarity only makes it worse.

You could almost see modern architecture as an attempt to exorcise all this, to dispel the dark corners and secret spaces and flood our living spaces with purifying light, but in the movies at least, you can never completely cast the devil out of the home. At best, you’re just sweeping it under the carpet and papering over the cracks.

Property prices being what they are, crumbling old mansions are also harder to come by in the modern era, but more pressing anxieties have come to haunt the domestic realm – such as economics. That was the case with The Amityville Horror, which has spawned a whole sub-genre of imitators and sequels. It’s the classic story (supposedly true – but aren’t they all?) of upwardly mobile newlyweds who think they’ve found a bargain fixer-upper and ignore all the warning signs: the scary attic windows; the fact that the previous tenant murdered his family with a shotgun; or that classic giveaway, the former Native American burial site. Amityville wasn’t a great movie but it clearly struck a chord with audiences in 1979, who were facing economic uncertainty and high mortgage rates. The same fears have come back to haunt the modern market. Lo-fi hit Paranormal Activity, released before the 2008 crash, played on ideas of an aspirational couple whose starter home comes with a hidden debt to repay.

Meanwhile, Tobe Hooper and Stephen Spielberg’s 1982 Poltergeist was set not in a scary old building but a sunny new one in a planned community (sadly built on the site of an old cemetery).

Six years later, Tim Burton’s Beetlejuice taught a similar lesson, albeit with a cheerier tone. It presents its ghost couple with a fate worse than death: having your perfect clapboard country home taken over by gentrifying yuppies, who set about giving it a lurid postmodern makeover. This decor paranoia still runs strong. Most viewers probably related to Darren Aronofsky’s Mother! less for its biblical-environmental allegories than its nightmare scenario of uninvited guests trashing your paintwork and sitting on your unbraced sink.

What makes Hereditary’s house all the scarier is that we never get a sense of its layout. That’s because there isn’t really one. The movie was not shot inside a house but on purpose-built sets. It’s the same trick Stanley Kubrick used in The Shining and, before that, Robert Wise in The Haunting: two seminal house horrors where our inability to mentally map the space adds to the unease. Conspiracy theorists have obsessively pored over the Overlook Hotel’s floor plan and found it doesn’t add up. Likewise with The Haunting, the sets were purpose-built to look slightly off, and filmed with distorting lenses. That “offness” of the house was there in Shirley Jackson’s source novel: “It had an unbelievably faulty design which left it chillingly wrong in all its dimensions, so that the walls seemed always in one direction a fraction longer than the eye could endure, and in another direction a fraction less than the barest possible tolerable length.” Jackson’s book also references the Winchester Mystery House. Might she have visited it?

Hereditary doesn’t pander to the cliches of the genre but it knows its history: there are no creaky floorboards, no peeling wallpaper. Haunted-house movies are now themselves haunted by what came before, and so few have surprises to spring. Movies such as Hereditary show that if you get the architectural wrongness right from the start, the house becomes part of the terror rather than just the venue for it.

Hereditary is in cinemas from 15 June