Stephen Sondheim, who has died aged 91, was a leading light of musical theatre over the course of more than six decades, from the moment in 1957 when he achieved renown as Leonard Bernstein’s lyricist for West Side Story. He went on to establish a place for himself with intelligent, unconventional works such as Company (1970), Sweeney Todd (1979) and Into the Woods (1987), which brought him a following appreciative of the new departures he made, even if his chosen path was not obvious or easy.

Sondheim saw himself as heir to Oscar Hammerstein II, who had written the lyrics for Oklahoma! (1943) and South Pacific (1949). Many expected him to revive the floundering American musical, supplying new perceptiveness, relevance and verve.

Yet, although the half-century following Hammerstein’s death in 1960 was a golden age full of long-running musicals in the West End and Broadway, and despite his considerable gifts with words and music, Sondheim never scored a true hit show. Nonetheless, new productions and film versions continued for the rest of his life and beyond, and his best work may well last longer than the more profitable musicals of his time.

The old firms – Rodgers and Hart, Rodgers and Hammerstein, and Lerner and Loewe – comprised fruitful collaborations between composers and wordsmiths. Sondheim, on the other hand, wrote both music and lyrics himself. Some commercial flops resulted, but Sondheim took after Cole Porter in supplying both text and tune for a number of adorable hit songs.

The most famous, recorded more than 500 times – notably by Frank Sinatra – was Send in the Clowns, from A Little Night Music (1973), a nostalgic almost-operetta developed from Ingmar Bergman’s 1955 film Smiles of a Summer Night. Barbra Streisand took up I’m Still Here from Follies (1971), and The Ladies Who Lunch from Company. Who’s That Woman?, The Little Things You Do Together, Liaisons, Someone in a Tree, and Anyone Can Whistle were all good vehicles for singers.

But these numbers are better as out-takes than in their original dramatic contexts. However grownup his themes and coherent his dramas, Sondheim’s ambition to carry through the revolution that Hammerstein had achieved with Oklahoma!, the first musical where songs moved the plot forward and belonged to the character singing them, may have been unrealistic.

Born in New York, Stephen was the only child of a comfortably middle-class family. His mother, Janet (nee Fox, and known as Foxy), a dress designer, and father, Herbert, a clothing manufacturer, were partners in a New York fashion business. They divorced when Stephen was 10, and he ended up disliking his mother so much that in 1992 he did not go her funeral.

Foxy was friends with Hammerstein’s wife, Dorothy, and after the divorce she and Stephen moved to live near the Hammersteins, on a farm in Pennsylvania. During his adolescence, Sondheim would look to Hammerstein as a substitute father.

Sondheim had started playing the piano by ear from the age of four, before studying the piano and organ sporadically at school. Ravel was one of his early musical discoveries, which he eagerly shared with Hammerstein, giving him a recording of the Piano Trio as a birthday present. In 1990 Sondheim told an audience at the National Theatre: “Most popular music and most show music owes its origins to Ravel.”

From an early age, he showed his skill at mind games and crosswords, submitting a puzzle to the New York Times at the age of 14. (Later he subscribed to the Observer and entered its fiendish Ximenes puzzle contests. He did not win, though he was highly commended.) At Williams college, Massachusetts, he planned at 16 to major in maths, but a freshman music course inspired him to study that subject instead.

He wrote a musical called By George, and showed it to Hammerstein, who said it was terrible but talented, and showed him where he had gone wrong. This process turned into the unofficial yet fundamental course on which Sondheim learned to write musicals, adapting various plays, working on non-dramatic narrative and creating an original story.

At 18, Sondheim witnessed the staging of his first student work, All That Glitters – with his own book, music and lyrics. On graduation he won a $3,000 prize, paid annually for two years, went on to take private composition lessons in New York with the avant-garde composer Milton Babbitt, and also found work as a television scriptwriter. Lemuel Ayers asked him to write a show that would become Saturday Night (unperformed until the end of the century, as Ayers died unexpectedly).

The resulting material was good enough to convince Arthur Laurents, who was doing the book for West Side Story, that Sondheim should write the lyrics. And that marked his entry into the hall of fame. He became and remained a highly self-conscious intellectual artist, who knew precisely what he wanted to achieve. In the first of two books about his lyrics, Finishing the Hat (2010), he touched on the mysteries of his taste and its narrow range, key to his successes as well as his failures.

Sondheim musicals are best appreciated through original cast recordings. The songs give the essence of what motivated their creation. In 1977, Sondheim explained that hit tunes were irrelevant. Melody was merely a reflection of familiarity – as if all tunes were equal until dinned into a listener’s memory. His music was characteristic and very recognisable, whether or not it chimed with popular taste. Some thought it too influenced by his rhythmic facility and the fast syllabic flow of his clever lyrics.

West Side Story, based on Romeo and Juliet, was not the first Shakespeare adaptation to hit Broadway: it had been preceded in 1948 by Porter’s hugely successful collaboration with Sam and Bella Spewack, Kiss Me Kate, based on The Taming of the Shrew. But Bernstein’s music, the tough inner-city New York location and Jerome Robbins’ choreography struck a new note – and Sondheim’s song texts were a significant part of that novelty.

Two years later, Sondheim collaborated again as lyricist – this time with Jule Styne on Gypsy, based on the life of the stripper and burlesque star Gypsy Rose Lee.

In 1962, he displayed his talent as composer as well as lyricist with his successful Broadway breakthrough, A Funny Thing Happened on the Way to the Forum. Based on a classic Plautus farce, it enjoyed a two-year run. In Broadway terms, A Funny Thing was unusually economical, with no changes of costume or set. Having the incomparable Zero Mostel in New York and Frankie Howerd in London as the slave Pseudolus added real value.

Anyone Can Whistle (1964), with Lee Remick and Angela Lansbury, proved to be a flop, as did Sondheim’s collaboration with Richard Rodgers, Do I Hear a Waltz? (1965). But the cast recording of Anyone Can Whistle fared better, and Sondheim began to attract a cult following, enhanced by the modest but genuine artistic success of Company. He came to be hailed as the most significant innovator in musical theatre of his day.

Generally he fared best with London’s theatre critics, which perhaps compensated for his critical reception in the US. A footnote in Finishing the Hat claims: “Musicals are the only public art form reviewed mostly by ignoramuses. Plays, at least in [the US], are reviewed by people whose knowledge is comprised of what they read in Variety and gossip columns, and who know nothing, of course, about music.”

However, fans promoted and protected him. Commercial failure boosted his myth as suffering genius. He became a cause. It helped that he was backed by Cameron Mackintosh, a theatrical entrepreneur with deep pockets. Sondheim, British critics maintained, had created “the intelligent musical”. But was there something missing from his work?

He had a tendency to adopt elaborate dramatic structures and concepts. His shows tended to go limp at the interval, most notably Sunday in the Park With George (1984), contrasting the French painter George Seurat with how art was made a century later. He liked complex storylines laden with significance. However, aside from A Funny Thing, perhaps they lacked humanity and affection.

That may have stemmed in part from the competition that Sondheim met after 1975, when A Chorus Line (created by Michael Bennett, with music by Marvin Hamlisch) and Evita (with music by Andrew Lloyd Webber) swept in. Though A Chorus Line, the perfect backstage musical, was blatantly sentimental, it picked up a theme that Sondheim’s Follies (for which Bennett in 1971 created memorable choreography) had fluffed; while its initial run of more than 500 performances brought seven Tonys and it has been much revived since, it did not recoup its investment. And Lloyd Webber’s works were conceptually provocative in just the way Sondheim seemed to want to be.

“The point about Andrew,” he said, “is that people just happen to like what he likes.” Sondheim’s reputation as an innovator grew, but Lloyd Webber’s brash, bold, obvious vehicles underlined with their implacable popularity just how out of joint with popular taste Sondheim was.

He had an explanation: “What makes smash-hit musicals are stories that audiences want to hear. And it’s always the same story, how everything turns out terrific in the end and the audience goes out thinking, ‘That’s what life is all about.’ Unfortunately that’s seldom the kind of material that attracts me. In fact I never think of whether it’s reassuring until the whole show’s written.” Musicals had traditionally been the voice of hope, perhaps naive, perhaps foolish. Sondheim’s were about understanding. He rebelled against the dream-market ethos of popular theatre.

Among Sondheim’s close collaborators was the stage director Hal Prince, whom he had met at the premiere of South Pacific. Together they scored many prizewinning successes, from Company to Sweeney Todd, but also some later flops, including Merrily We Roll Along (1981) and Bounce (2003).

After Prince, Sondheim collaborated with James Lapine, who helped devise, write the dialogue and direct the first productions of Sunday in the Park With George, Into the Woods, jumbling up and analysing an assortment of fairy stories, and Passion (1994), set in the Italy of the Risorgimento.

He also worked with the librettist John Weidman on Pacific Overtures (1976), about the 19th-century westernisation of Japan, and Assassins (1990), about those aspiring to kill US presidents. He also collaborated with George Furth, Hugh Wheeler, James Goldman and Jerry Zaks. For brilliant orchestrations, starting in 1970 with Company, he turned to Jonathan Tunick.

Sondheim continued to find it easier to get his work staged in London than on Broadway. A portmanteau revue of his songs, Side By Side By Sondheim, was a big West End hit in 1976. Pacific Overtures and Sweeney Todd were programmed by English National Opera and Covent Garden in, respectively, 1987 and 2003.

Thanks to Mackintosh, Sondheim had a London theatre renamed after him in 2020, following that on Broadway to mark his 80th birthday in 2010. Sondheim was, too, the first holder of the post of Cameron Mackintosh visiting professor of drama at Oxford University, in 1990.

Sondheim’s nihilistic paean on social injustices, Sweeney Todd, was often revived, and in 2007 was adapted for the cinema by Tim Burton, with Johnny Depp in the title role and Helena Bonham Carter. The show’s relish for cruelty and grand guignol originated in the Christopher Bond play on which it was based. It had real comic energy, as David McVicar showed in his Opera North revival, staged in 2002. A decade later it returned to the London stage in Jonathan Kent’s production starring Michael Ball and Imelda Staunton.

Smaller London venues, the Menier Chocolate Factory and the Donmar Warehouse, gave Sondheim revivals with success. Film versions included Rob Marshall’s Into the Woods (2014) and, about to be seen in cinemas, Steven Spielberg’s West Side Story. In 2009 Lin- Manuel Miranda rendered that work’s lyrics into Spanish, and in 2020 it was reimagined, for Broadway, and this year has seen New York stagings of Assassins and Company.

When the latter opened, Sondheim was 40. It tells the story of a bachelor and his friends, and how he can never quite get married or genuinely involved with a woman. Sondheim had not had a long-term relationship of any sort when writing his musical comedy, and recalled that he asked a married female friend to explain what relationships were all about.

He came out as gay later in the 1970s, but did not live with a partner until the 90s. “It was difficult at times,” he said in 2010. “I went through what everybody goes through in their first serious relationship, which most people do in their late teens or 20s, and which I didn’t until I was 60. So it was difficult for me to learn; but it was also joyful.”

In 2017 he married Jeffrey Romley, who survives him, as does his half-brother Walter.

• Stephen Joshua Sondheim, composer and lyricist, born 22 March 1930; died 26 November 2021