In 1963, when he was 14 years old, Nick Broomfield briefly crossed paths with Brian Jones. “We were on the same train. I was travelling to school and he was going back to Cheltenham, where he grew up,” recalls the documentary-maker 60 years later. “He was sitting all alone in a first-class compartment. I just tapped on the door and, with some temerity, introduced myself.”

Jones, who was on the cusp of pop stardom with the Rolling Stones, the group he had formed the previous year, turned out to be warm and charming, chatting with the eager young fan for several minutes. “I was surprised at how friendly he was,” Broomfield recalls. “We chatted about trains, mainly. He told me that he loved trains and the line we were travelling on – the Great Western – was his favourite. I just remember thinking how very middle-class, well spoken, polite and accommodating he was.”

In the subsequent TV interviews and press conferences that attended the rapid ascendancy of the Rolling Stones, Broomfield’s fleeting impression of the musician was borne out, as he engaged effortlessly with the media, coming across as the most pleasant and articulate band member. Just six years after their encounter on the Cheltenham train, though, on 3 July 1969, Brian Jones was found dead in the swimming pool of his home, Cotchford Farm, in rural Sussex. The coroner’s report concluded that it was “death by misadventure”, and noted that his liver and heart were greatly enlarged by sustained drug and alcohol consumption. He was 27 years old.

The tumultuous seven-year period that Jones spent as a Rolling Stone, as well as the troubled childhood and adolescence that preceded them, are explored in depth in Broomfield’s latest film, The Stones and Brian Jones. Unlike many of his previous documentaries, in which he appears onscreen as a character in the unfolding story, Broomfield is present here only as interviewer and in a voiceover. The narrative unfolds slowly in an evocative, mesmerising and sometimes heartrending interweaving of first-hand testimony and rich archive footage: a portrayal of a gilded time of astonishing creativity, but also of self-destructiveness and inevitable tragedy.

“I’ve always wanted to do a film about the 60s, which were my formative period,” says Broomfield. “That chance meeting with Brian Jones on the train has stayed with me, not least because, back then, he seemed to have everything going for him. He was young, charismatic, incredibly gifted, and an integral part of a group that would define the time more than any other apart from the Beatles. He epitomised that dazzling 60s moment in many ways, which is so very different from now, and then he was suddenly gone. I can still recall the shock I felt on hearing the news of his death.”

The story of Jones’s speedy rise and tragic fall has been told in some depth before, not least in a 2019 documentary, Rolling Stone: Life and Death of Brian Jones. That film’s central premise is that Jones was actually killed in a violent argument over money with a builder, Frank Thorogood, who had been renovating his house and, as one contributor puts it, “leeching off” the hapless musician. In 2010, however, Sussex police, having conducted a review of the death, concluded there was no new evidence to contradict the original verdict.

Broomfield avoids what he calls “the murder conspiracies” altogether. “I felt they didn’t go anywhere,” he says, “and, in terms of the story, I just thought: what’s the point of me wasting a lot of time just to end up dismissing them?” Instead, his film is, at its core, a psychological study of a gifted, complex individual, ill prepared for fame and dogged by insecurity. And, though Jones’s early death saw him canonised for a time as a 60s icon, alongside other doomed figures such as Jimi Hendrix, Jim Morrison and Janis Joplin, Broomfield believes his creative genius is all but overlooked today.

Put simply, there would have been no Rolling Stones without Brian Jones. He formed the group, named them after a song by one of his idols, Muddy Waters, and honed their early sound in the style of the American blues musicians he worshipped. Before they found a manager, he was the one who rang venues in London and beyond to hustle for live gigs.

“Brian was the leader,” Bill Wyman, the Stones’ former bassist, tells me. “Early on, he even signed the management and recording contracts, and, as the most popular member of the band with girls, he received 75% of all Stones fan mail in those early years.” In Broomfield’s film, Linda Lawrence, who was Jones’s girlfriend from 1962 to 1964, recalls that, “Mick [Jagger] was in awe of Brian. He absolutely loved him. He wanted to be Brian, because Brian had all the girls and all the fan mail.”

In the months leading up to his death in 1969, though, the young man who had been the most angelic-looking and musically gifted Rolling Stone had become almost unrecognisable. His once pristine blond bob was unkempt and his features were bloated by his addictions. As is evidenced by fly-on-the-wall footage from Jean-Luc Godard’s 1968 film Sympathy for the Devil, he would often turn up at the studio, where the band were recording their seventh album, Beggars Banquet, too stoned and sleepy to play his guitar. Sometimes he didn’t turn up at all.

By June 1969, his self-destructiveness, coupled with mood swings and unpredictable behaviour, had become too much for his bandmates. With an American tour approaching, Jagger, Keith Richards and Charlie Watts visited Jones at his London home, where they broke the news that he was no longer a member of the group he had formed.

“The fact that he was expecting it made it kind of easier, I guess,” Richards said later. “He wasn’t surprised. I don’t really think he even took it all in. He was already up in the stratosphere.”

Abandoned by the group, Jones suddenly found himself alone, unmoored and, as the music writer David Dalton puts it in Broomfield’s film, “ostracised from his own band”. Three and a half weeks later, he was dead.

The Brian Jones that emerges from Broomfield’s film is a deeply troubled individual haunted by anxieties that often shaded into self-loathing. Broomfield locates the roots of those insecurities in Jones’s emotionally arid upbringing in the suffocating environs of postwar Cheltenham. His parents, Lewis, an aeronautical engineer, and Louisa, who played the organ in the local Baptist church, were upright, strict and bound by the bourgeois values of the time.

Though his father was also a piano teacher, he disapproved vehemently of his son’s obsession with jazz and blues music and, according to Wyman, constantly “badgered Brian to get a ‘proper’ job”. In the film, one of Jones’s teenage girlfriends, Pat Andrews, paints a much bleaker picture of his childhood. “His mother was very rigid. There was no fun, no laughter. I’m sure she loved him but I think that she didn’t know how.”

Throughout his short life, Jones sought, but never received, his parents’ approval, while simultaneously provoking and appalling them with his rebelliousness and utter disregard for convention or propriety. When he formed the Rolling Stones, aged 20, he had already fathered three children by three different girlfriends, and been banished from his parents’ house for bringing shame on the family. His first serious girlfriend, Valerie Corbett, became pregnant when both she and Jones were just 16, and the child was given up for adoption. In the film, Andrews, who was the mother of his third child, relates how, as Jones’s fame grew, he “just seemed to drift away”.

Broomfield links this pattern of behaviour to the emotional austerity of Jones’s childhood. “Brian was essentially a helpless child who craved his parents’ approval. After he was thrown out of their house at 17, he would somehow find families that would look after him. Basically, he would charm his girlfriends’ parents and move into their family nest like a cuckoo. Invariably, they would look after him until they found out that he had got their daughter pregnant.”

Many of the women who speak on film about Jones remember him fondly and are to varying degrees compassionate towards him – for a time, Andrews even ran his fan club. The most articulate and refreshingly unregretful is Linda Lawrence, who dated Jones in the eye of the hurricane that was the Stones’ early fame and notoriety. She is the mother of his fourth child, Julian. (Broomfield was in Ireland to show the finished film to Lawrence when he spoke to me.)

Not long after they first met, Jones and Lawrence moved in with her parents in Windsor. They were, she says, very much in love, and in the film she describes him as “a gentleman – he opened doors and was very gentle spoken”.

In July 1964 she gave birth to Julian. The following October, a young woman, Dawn Molloy, contacted the Stones’ management to tell them she was also pregnant by Jones. Charm and deception seem to have coexisted side by side in Jones’s psyche, the one, to a degree, camouflaging the other. In this instance, in return for £700, Molloy agreed not to make public the information that Jones was the father of her child.

* * *

As the Stones’ fame grew, and scenes of mayhem and unbridled sexual fervour attended their every live appearance, the power dynamic of the band shifted and charismatic frontman Jagger became the main focus of the media and fans’ attention. Jones’s jealousy and insecurity were compounded by the manoeuvrings of the Stones’ young manager, Andrew Loog Oldham, who was encouraging Jagger and Richards to write their own material in the collaborative manner of Lennon and McCartney. “Brian’s confidence suffered as a result,” says Wyman, who talks in depth in the film about Jones’s musical innovations and the audacious embellishments he brought to hit songs like Paint It Black. “He was a musical genius but he lacked the self-belief to write songs.”

Jones’s increasing sense of alienation was exacerbated by his heavy drinking and drug use. In a candid onscreen interview with Broomfield, the 60s French pop singer Zouzou, who was briefly Jones’s girlfriend in 1965, says: “He was doing so many stupid things all the time. Speed. We didn’t sleep for days on end…”

In 1966, backstage after a concert, Jones’s unruly lifestyle took another dramatic turn when he met Anita Pallenberg, a vivacious German-Italian actor and model who would become his lover, his soulmate and eventually his romantic nemesis. For the first time in his tangled love life, he had fallen for someone who was wilder, darker and stronger than himself. Jones’s biographer, Paul Trynka, tells Broomfield that the musician had been thinking of leaving the group, “but by teaming up with Anita he knew they’d be a real phenomenon – which they were”. Together they embarked on what Trynka calls Jones’s “last great ride”.

Initially, Pallenberg had a profound effect on Jones’s self-image, encouraging him to dress in a more dandified way and restoring his confidence within the band. “There was such erotic power to their pairing, and such glamour,” Trynka says. “He wanted to be seen as a main player in the Stones and she helped that happen.”

By 1967 they were as glamorous a couple as Mick and Marianne Faithfull, but their relationship, even by the wayward standards of the sexually heady time, was tempestuous and destructive. The film director Volker Schlöndorff, who cast Pallenberg in his 1967 film Degree of Murder, recalls how they often provoked each other to violence. “It was not a tender relationship,” he tells Broomfield. “They were crazed out.”

By then, Jones’s excessive drug taking and drinking were not just eroding his creativity, but affecting his ability to function. His relationship with Pallenberg deteriorated rapidly. “When she could not control him any more,” Schlöndorff says, “she just walked away to the next storm.”

In March 1967, Jones, Pallenberg and Richards travelled to Morocco together. When Jones fell ill, Pallenberg and Richards became lovers, taking flight together while he languished in hospital. The casual brutality of the betrayal wounded Jones deeply. It was, as Richards later put it, “the final nail in the coffin” of their already fragile friendship: “I knew he would never forgive me and I don’t blame him.”

So deep was Jones’s despair that his parents, shocked by his physical and mental deterioration, finally offered to help him. One of several moving passages in Broomfield’s film is a recording of Lewis Jones speaking about that painful time. “What I firmly believed was the turning point in Brian’s life,” he says, “was when he lost the only girl he really loved. In our opinion, he was never the same again… he became a quiet, morose and inward-looking man.”

As Richards later noted, 1967 was “a painful year”, not just for Brian Jones but for the Stones. In February of that year, Jagger and Richards were arrested in the famous drugs bust at Redlands, the guitarist’s home in rural Sussex, following an orchestrated campaign against them by the tabloid News of the World. The pair narrowly escaped a prison sentence. On 10 May, Jones was arrested for possession after a police raid on his London flat uncovered significant amounts of marijuana, cocaine and methamphetamine. On his release, he sent a telegram to his parents: “Don’t judge me too harshly. Don’t think badly of me.”

Brian Jones’s final appearance with the group took place at the filming of the carnivalesque extravaganza, The Rolling Stones Rock and Roll Circus. By the time they performed in the early hours of the morning, he was too drunk to play his guitar properly. “He was just gone,” Jagger later said of his bandmate’s decline, “and not in any condition to do anything. We had to baby him.”

It is left to Marianne Faithfull, who foresaw her own future estrangement from the Stones’ inner circle in Jones’s isolation and humiliation, to articulate what many contemporary viewers may be thinking as they watch: “I saw him as another person with incredibly low self-esteem, who needed help, not to be destroyed and ground underfoot.”

That, as this tragic tale makes clear, is not how rock groups operated in the 60s, where grand gestures of love and togetherness overlaid the essentially ruthless nature of the business of rock’n’roll. Just two days after his death, at a vast open-air concert-cum-wake in Hyde Park, the Stones remembered Brian Jones, their deposed leader, in front of 250,000 people. The concert began with Jagger reading from Shelley’s poem Adonais, while roadies released hundreds of white butterflies above the crowd. To Jagger’s right, though, the presence of a new guitarist, a callow young Mick Taylor, attested to the fact that the Stones had already moved on.

A few days later, Wyman and Watts were the only band members to attend Jones’s funeral in Cheltenham Cemetery. A regretful Watts later said of his bandmate: “He got much nicer just before he died. I felt even more sorry for what we did to him then. We took his one thing away, which was being in a band.”

* * *

Broomfield’s film is a kind of elegy for the era and for its first rock’n’roll casualty. It begins with slow-motion monochrome footage of a young Brian Jones onstage, beatific in the spotlight. Playing over it is an extract from a long-ago interview he gave to a German journalist. He is holding forth on, of all things, the inevitable fracturing of the Freudian bond between child and parent that occurs when the former tentatively starts to become an adult.

“A child is a thing to be loved,” he muses in his posh Gloucestershire tones. “A child is the manifestation of both parents and both parents see themselves in the child… he’s a reflection of their own personality. So, one day, when he grows up, he’s going to assert his own personality, which may well differ from the outlook and personality of his parents, who immediately feel upset… They feel they have lost him.”

The film ends with footage of Jones’s parents, looking lost and bereft, at their son’s very public funeral. Over the footage, Linda Lawrence reads from a letter to Jones from his father. It was discovered in a box of letters that had languished in the attic of her family house for 40 years after his death. It reads: “My dear Brian, we have had unhappy times and I have been a very poor and intolerant father in so many ways. You grew up in such a different way than I expected you to. I was quite out of my depth… I don’t suppose you will ever forgive me, but all I ask is for just a little of that affection you once had for me. This is a very private and personal note so don’t trouble to reply. Love, Dad.”

Broomfield tells me that it is impossible to know whether Brian Jones ever read, or even received, the letter.

• This article was amended on 12 May 2023 to replace an image that contained unverified caption information.

• Arena: The Stones and Brian Jones is on BBC Two and BBC iPlayer at 9pm on 15 May