If his life unspools in the arch, neat fashion of one of his movies then the director Wes Anderson, who'll turn 45 this spring, is halfway through. "You had these film-makers, John Huston, Luis Buñuel, who more or less died on their sets. And they seemed happy. Now I wouldn't want to die young on one of my sets. But if I was a 90-year-old director…?" Snugly suited in olive corduroy, speaking in London before the release of his new film, The Grand Budapest Hotel, Anderson nods at the thought.

You wonder what Anderson's films might look like, sound like, should he still be going at 90. The action rendered upside down? Dialogue in an invented language? Since his third feature, 2001's The Royal Tenenbaums, the Texas-born director has inched ever further into a strange, spruce world that isn't really recognisable as our own. His fourth film, 2004's The Life Aquatic With Steve Zissou, charted an underwater mission of revenge against a glow-in-the-dark shark. His seventh, 2012's Moonrise Kingdom, followed a love-struck couple who met during an amateur performance of Noye's Fludde and eloped with too much luggage. (They were 12.) "Five degrees removed from reality," Anderson has called his world, probably undercalculating it.

His new film focuses on a hotel concierge called Monsieur Gustave (Ralph Fiennes) and a bellhop called Zero (newcomer Tony Revolori) who endeavour to recover an inheritance left to Gustave by a wealthy dowager. Set in the 1930s – "before the war," Anderson says, "the storm coming" – their efforts are complicated by the presence of SS-like troops, here known as the ZZ. The Grand Budapest Hotel isn't actually set in Budapest, or even definitely in Hungary, rather a composite European town that has cobbled streets, an English-language newspaper and a funicular train. "I wanna make a story about characters, their experiences," Anderson tells me, "but at the same time I wanna do something about their world."

We're chatting in a hotel room that's teeth-achingly overdecorated. Floral wallpaper, curlicued carpet, bedstead in burgundy and gold. It takes some effort to make out Anderson amid it all, his slim 6ft 1in arranged in a zigzag on a sofa. A famous manager of detail, he would never allow a place as messy as this to appear in one of his films. Everything has to be just-so.

A producer who often works with Anderson told me that on Fantastic Mr Fox, his 2009 animation of Roald Dahl's book, the director wanted to include a glimpse of a newspaper column written by Mr Fox. It would appear on screen for maybe a second – and Anderson put his head in a vulpine place and wrote 400 words of copy. More recently his girlfriend, an illustrator called Juman Malouf, was employed on The Grand Budapest Hotel to paint portraits of the actors in costume. Not for use on screen, instead to be hung around the cast and crew's hotel: a mood thing.

Fans hold that this amounts to work that is unusually rich and absorbing. Detractors call his movies fussy and mannered. As Anderson descends ever deeper into his special world, does he think he risks mystifying audiences? I suggest it might be difficult for a filmgoer to come to The Grand Budapest Hotel without having seen any of his previous films, to get a grounding.

"I would think someone coming to it fresh would just watch the movie," he says. "Maybe they would think: this is a little different from what I've seen before. But Grand Budapest Hotel is pretty straightforward storytelling, I think."

It is and it isn't. A massive dramatic finale in The Grand Budapest Hotel occurs off-screen. There are flashbacks, double flashbacks, triple flashbacks, and Harvey Keitel pops up in the middle of it – topless, bald – to orchestrate a prison break. Leading actor Ralph Fiennes plows through with gusto, but the knowing sort, as someone might play a parlour game or appear in panto.

Critics who saw Budapest at the Berlin film festival, where it premiered this month, have called it "vibrant and imaginative", "nimblefooted, witty", and as a sucker for Anderson's stuff since his early days, I'd agree. Still, I can imagine the writhing and groaning of his detractors, those sceptics who think, to borrow a phrase from New York magazine, the director has "become pickled in a world of his own creation".

In person Anderson is reserved and polite, a youthful-seeming fidgeter with an uncommon way of speaking. "Certainly I feel increasingly haggard," is how he frames his reply when I remark how fresh-faced he looks at 44.

He's a problematic interviewee. "There's a bit of a yes answer," Anderson says, when I ask something simple about his habit of writing screenplays in collaboration. "But I don't know how much is yes." The Texan very obviously doesn't want to talk about himself, but doesn't want to be rude either. His answers come coated in clammy, apologetic, Woody-ish reservation: "Well, um, often. Iguesswhatyou'dsayis. I'm usually. I wanna… I wanna say…"

He wrote the script for The Grand Budapest Hotel with the British artist Hugo Guinness, a friend. Guinness warned me I'd find Anderson shy. "He doesn't like a lot of attention. He wants to get on with the job of making films." During our conversation in London I think Anderson was unequivocal about himself exactly once. As a young man, he told me, "I was an idiot".

Anderson grew up in Houston, the son of a writer and an archaeologist. He was passionate about Star Wars, the New Yorker film critic Pauline Kael, and the "slightly heightened reality" of Dahl's children's stories. A mess of memories from his school days – appearing as an otter in a production of Noye's Fludde, fantasising about running away with a girl in his class – would ferment for decades before providing material for Moonrise Kingdom. Anderson remembers being nothing like the sparky 12-year-olds in that movie, however. "My experience was so insular," he says. "I didn't have any real perspective about where people were coming from. I didn't have a radar."

After studying at the University of Texas (philosophy, playwriting classes, piles of videos from Blockbuster) Anderson moved to California with his college roommate, Owen Wilson. Over 18 months they wrote a screenplay together, eventually making it – Anderson directing, Wilson starring – with the help of the influential producer James L Brooks. Bottle Rocket, a heist movie released in 1996, was understated, amusing, very original and a commercial flop.

Brooks, though, was convinced Anderson was an important new voice, later writing that the Texan saw things differently "than the billions of other ants on the hill". He helped Anderson get going on his second film, Rushmore (1998), a cruel and hilarious little gem starring Bill Murray as a misanthropic businessman whose midlife crisis is inflamed by a precocious schoolboy. Murray was wonderful in it. Anderson has always had a talent for finding and attracting bolstering names.

For his debut, Bottle Rocket, he convinced James Caan to take a role. (Caan, not long out of rehab, accidentally dislocated the shoulder of another actor on his first day.) Gene Hackman, Anjelica Huston, Ben Stiller and Gwyneth Paltrow agreed to play members of a family of geniuses in The Royal Tenenbaums. Edward Norton wore a too-tight scout kit for the duration of Moonrise Kingdom and George Clooney voiced Mr Fox. "A high catch rate," Anderson's long-standing producer, Jeremy Dawson, calls this, though the director himself, no show-off, says casting's not so difficult when you have the right email addresses.

Dawson has another take. "These films are no walk in the park to make. Yet you see how many people come back, time and again. [Bill Murray has appeared in all of Anderson's films since Rushmore; there are many other regulars.] Wes has a theory that because making a film takes up such a chunk of your life, the process should be enjoyable."

Shooting The Grand Budapest Hotel, the cast and crew lived together in a small hotel in Görlitz, Germany for the duration. The vocals for Fantastic Mr Fox were recorded between dinners at George Clooney's Lake Como villa. Anderson doesn't like his actors to live in trailers, or for anyone to commute. "Everybody can go home when it's over," he tells me. Dawson: "It becomes like summer camp or like school – you bond. And that, by osmosis, makes the films better."

Meanwhile Anderson is always absorbing, hoarding, collecting detail for use in future projects. "He remembers everything," says Dawson, which Guinness can confirm. "I don't think he's one of those people who works then relaxes – he can't stop. He's observant. He's got a very, very good memory and he soaks it all up."

Guinness once invited Anderson to his New York home. "And I seem to remember I had piles of porn videos on my desk." Later, when Guinness sat down to watch The Royal Tenenbaums, there they were, piled on one of the character's desks.

Back in 1998, when Anderson had finished making Rushmore, he sought out his old hero Pauline Kael, then in retirement, and arranged a private screening. "I don't know what you've got here, Wes," Kael told him afterwards. It seemed to preempt wider critical feeling about his work: unpersuaded.

Six films on, at a time when those other wilfully odd film-makers, David O Russell and Alexander Payne, are routinely awarded major prizes, Anderson sometimes merits inclusion in the best original screenplay lists. I ask him if he feels any envy. "You mean, like, why not me getting Oscar nominations? No, not really. I'm very happy just to do my thing. Happy just to keep this concern going. I think the resentful side of me will only come out when I'm unable to get a movie made."

His early mentor James L Brooks once made a comment both flattering and damning. When you have a voice as unique as Anderson's, Brooks said, "the voice must be served; all other exit doors, marked 'expediency' or 'solid career move', are sealed over". I wonder if Anderson is ever tempted to switch track. Does he long to chuck in the whimsy, forget Bill Murray cameos – instead cast The Rock in something and blow stuff up?

Another idiosyncratic director, Baz Luhrmann, recently admitted that he wouldn't mind having a go at Bond. What would Anderson, a childhood Star Wars fan, have said if he'd been approached to take on the recently mobilised reboot? His reply ("Well, you know…") comes layered in qualification ("… as a very abstract idea…") but there is an answer, buried.

"I could entertain what I would say in that situation. And it's not like anyone's been pursuing me to do a 'part five' of anything, by the way. But no. It's really simple for me… I'd rather do my own thing. That's what I like to do."

Anderson shifts on the sofa. "I've done a bunch of movies. And it's a luxury to me that they're all whatever I've wanted them to be."

The Grand Budapest Hotel is in cinemas 7 March