If you were asked to guess which prestigious film-making duo had spent their career scratching around desperately for cash, trying to wriggle out of paying their cast and crew, ping-ponging between lovers, and having such blood-curdling bust-ups that their neighbours called the police, it might be some time before “Merchant Ivory” sprang to mind. But a new warts-and-all documentary about the Indian producer Ismail Merchant and the US director James Ivory makes it clear that the simmering passions in their films, such as the EM Forster trilogy of A Room With a View, Maurice and Howards End, were nothing compared to the scalding, volatile ones behind the camera.

From their initial meeting in New York in 1961 to Merchant’s death during surgery in 2005, the pair were as inseparable as their brand name, with its absence of any hyphen or ampersand, might suggest. Their output was always more eclectic than they got credit for. They began with a clutch of insightful Indian-set dramas including Shakespeare-Wallah, their 1965 study of a troupe of travelling actors, featuring a young, pixieish Felicity Kendal. From there, they moved on to Savages, a satire on civilisation and primitivism, and The Wild Party, a skewering of 1920s Hollywood excess that pipped Damien Chazelle’s Babylon to the post by nearly half a century.

It was in the 1980s and early 1990s, though, that Merchant Ivory became box-office titans, cornering the market in plush dramas about repressed Brits in period dress. Those literary adaptations launched the careers of Hugh Grant, Helena Bonham Carter, Rupert Graves and Julian Sands, and helped make stars of Emma Thompson and Daniel Day-Lewis. Most were scripted by Ruth Prawer Jhabvala, who had been with them, on and off, since their 1963 debut The Householder; she even lived in the same apartment building in midtown New York. Many were scored by Richard Robbins, who was romantically involved with Merchant while also holding a candle for Bonham Carter. These films restored the costume drama to the position it had occupied during David Lean’s heyday. The roaring trade in Jane Austen adaptations might never have happened without them. You could even blame Merchant Ivory for Bridgerton.

Though the pictures were uniformly pretty, making them was often ugly. Money was always scarce. Asked where he would find the cash for the next movie, Merchant replied: “Wherever it is now.” After Jenny Beavan and John Bright won an Academy Award for the costumes in A Room With a View, he said: “I got you your Oscar. Why do I need to pay you?” As Ivory was painstakingly composing each shot, Merchant’s familiar, booming battle cry would ring out: “Shoot, Jim, shoot!”

Heat and Dust, starring Julie Christie, was especially fraught. Only 30 or 40% of the budget was in place by the time the cameras started rolling in India in 1982; Merchant would rise at dawn to steal the telegrams from the actors’ hotels so they didn’t know their agents were urging them to down tools. Interviewees in the documentary concede that the producer was a “conman” with a “bazaar mentality”. But he was also an incorrigible charmer who dispensed flattery by the bucketload, threw lavish picnics, and wangled entrées to magnificent temples and palaces. “You never went to bed without dreaming of ways to kill him,” says one friend, the journalist Anna Kythreotis. “But you couldn’t not love him.”

Stephen Soucy, who directed the documentary, doesn’t soft-pedal how wretched those sets could be. “Every film was a struggle,” he tells me. “People were not having a good time. Thompson had a huge fight with Ismail on Howards End because she’d been working for 13 days in a row, and he tried to cancel her weekend off. Gwyneth Paltrow hated every minute of making Jefferson in Paris. Hated it! Laura Linney was miserable on The City of Your Final Destination because the whole thing was a shitshow. But you watch the films and you see no sense of that.”

Soucy’s movie features archive TV clips of the duo bickering even in the midst of promoting a film. “Oh, they were authentic all right,” he says. “They clashed a lot.” The authenticity extended to their sexuality. The subject was not discussed while Merchant was alive and for some time after: “You have to remember that Ismail was an Indian citizen living in Bombay, with a deeply conservative Muslim family,” Ivory told me in 2018. But the pair were open to those who knew them. “I never had a sense of guilt,” Ivory says, pointing out that the crew on The Householder referred to him and Merchant as “Jack and Jill”.

Soucy had already begun filming his documentary when Ivory published a frank, fragmentary memoir, Solid Ivory, which dwells in phallocentric detail on his lovers before and during his relationship with Merchant, including the novelist Bruce Chatwin. It was that book which emboldened Soucy to ask questions on screen – including about “the crazy, complicated triangle of Jim, Ismail and Dick [Robbins]” – that he might not otherwise have broached.

The documentary is most valuable, though, in making a case for Ivory as an underrated advocate for gay representation. The Remains of the Day, adapted from Kazuo Ishiguro’s Booker-winning novel about a repressed butler, may be the duo’s masterpiece, but it was their gay love story Maurice that was their riskiest undertaking. Set in the early 20th century, its release in 1987 could scarcely have been timelier: it was the height of the Aids crisis, and only a few months before the Conservative government’s homophobic Section 28 became law.

“Ismail wasn’t as driven as Jim to make Maurice,” explains Soucy. “And Ruth was too busy to write it. But Jim’s dogged determination won the day. They’d had this global blockbuster with A Room With a View, and he knew it could be now or never. People would pull aside Paul Bradley, the associate producer, and say: ‘Why are they doing Maurice when they could be making anything?’ I give Jim so much credit for having the vision and tenacity to make sure the film got made.”

Merchant Ivory don’t usually figure in surveys of queer cinema, though they are part of its ecosystem, and not only because of Maurice. Ron Peck, who made the gay classic Nighthawks, was a crew member on The Bostonians. Andrew Haigh, director of All of Us Strangers, landed his first industry job as a poorly paid assistant in Merchant’s Soho office in the late 1990s; in Haigh’s 2011 breakthrough film Weekend, one character admits to freeze-framing the naked swimming scene in A Room With a View to enjoy “Rupert Graves’s juddering cock”. Merchant even offered a role in Savages to Holly Woodlawn, the transgender star of Andy Warhol’s Trash, only for her to decline because the fee was so low.

The position of Merchant Ivory at the pinnacle of British cinema couldn’t last for ever. Following the success of The Remains of the Day, which was nominated for eight Oscars, the brand faltered and fizzled. Their films had already been dismissed by the director Alan Parker as representing “the Laura Ashley school” of cinema. Gary Sinyor spoofed their oeuvre in the splendid pastiche Stiff Upper Lips (originally titled Period!), while Eric Idle was plotting his own send-up called The Remains of the Piano. The culture had moved on.

There was still an appetite for upper-middle-class British repression, but only if it was funny: Richard Curtis drew on some of Merchant Ivory’s repertory company of actors (Grant, Thompson, Simon Callow) for a run of hits beginning with Four Weddings and a Funeral, which took the poshos out of period dress and plonked them into romcoms.

The team itself was splintering. Merchant had begun directing his own projects. When he and Ivory did collaborate, the results were often unwieldy, lacking the stabilising literary foundation of their best work. “Films like Jefferson in Paris and Surviving Picasso didn’t come from these character-driven novels like Forster, James or Ishiguro,” notes Soucy. “Jefferson and Picasso were not figures that audiences warmed to.” Four years after Merchant’s death, Ivory’s solo project The City of Your Final Destination became mired in lawsuits, including one from Anthony Hopkins for unpaid earnings.

Soucy’s film, though, is a reminder of their glory days. It may also stoke interest in the movies among young queer audiences whose only connection to Ivory, now 95, is through Call Me By Your Name. “People walk up to Jim in the street to shake his hand and thank him for Maurice,” says Soucy. “But I also wanted to include the more dysfunctional side of how they were made. Hopefully it will be inspiring to young film-makers to see that great work can come out of chaos.”

• Merchant Ivory is showing in the BFI Flare festival at BFI Southbank, London, on 16 and 18 March

• This article was amended on 30 April 2024. An earlier version incorrectly said that Ismail Merchant and James Ivory’s sexuality was not discussed publicly until after Ivory won an Oscar for writing Call Me By Your Name.