As a struggling young singer-songwriter in 1960s Queens, New York, Paul Simon would often retreat to his parents’ bathroom. There, with the tiling giving the room an echo and the sound of running taps generating white noise, he’d sit strumming his guitar in the dark. This experience inspired The Sound of Silence – Simon and Garfunkel’s first US No 1 – and one of pop’s most memorable opening couplets: “Hello darkness my old friend / I’ve come to talk with you again.”

“The Sound of Silence was the first song I wrote which seemed to come from some place that I didn’t inhabit,” says Simon, now 82, over the phone from the city where he penned it. “At age 23, it was unusual, well beyond my age and abilities. Then it happened again throughout my writing. Bridge Over Troubled Water was another song that came mysteriously. So did a lot of Graceland. I wrote Slip Slidin’ Away in 20 minutes – usually it takes me a couple of months to get a song. There are other examples, like Darling Lorraine, of songs that came from some place else … A mystery, you could call it.”

Such classic songs flow through In Restless Dreams: The Music of Paul Simon, Alex Gibney’s three and a half hour documentary film, which explores the singer-songwriter’s long career and the making of his latest album, last year’s Seven Psalms. Simon says he approached the 70-year-old fellow New Yorker and gave him full editorial control. “I admire his work. I thought he would give a pretty accurate description of me from his perspective – but I didn’t think his perspective would be askew.”

Gibney has made award-winning documentaries on subjects such as Scientology and Enron corruption as well as Frank Sinatra, James Brown and Fela Kuti, but relished the opportunity to work closely with the artist his film calls “the greatest songwriter in the history of American popular music”.

“I’ve loved his work for many years, but I’m always looking for ways of going deeper than a kind of filmic equivalent of a Wikipedia entry,” says Gibney. “We had access to him making Seven Psalms and that’s very much an album about belief and mortality, issues with which I’m reckoning myself. It seemed an opportunity to go deep into his creative process and look at how what comes into the unconscious can be shaped into considered works of art. To do that with an extraordinary talent such as Paul was just a gift.”

In 2018, five years before Seven Psalms, Simon toured for the last time and was reportedly retiring. “I never said I was going to retire,” he says now, with a gentle New York accent. “I said I was going to stop, which I did. I thought that with that band and the repertoire we were doing we’d developed it as far as we could. It was enjoyable, but I wanted to find out what happens when you stop.” He did some travelling with singer Edie Brickell, his wife of 32 years. “Then I had a dream, and everything changed back to a new version of reality.”

On 15 January 2019, a voice in the dream told him: “You are working on a piece called Seven Psalms.” Over subsequent weeks and months, and as Covid closed the world down, he found himself waking in the early hours with lyrics that had come to him in dreams. As the album came together in Simon’s wooden cabin studio in Texas, Gibney was allowed unprecedented access, and filmed everything.

“Seven Psalms is an example of the whole [seven-part] piece coming to me in a unique way,” Simon says, after I mention that he’d actually sung about how “a vision softly creeping, left its seeds while I was sleeping” as long ago as The Sound of Silence. “I think there’s a connection between who I was as a kid, and my subconscious, and who I am now. It was very interesting and really quite pleasurable for a long time – until my hearing loss threw me off.”

While recording, he suddenly lost most of the hearing in his left ear. Gibney finds it “extraordinarily courageous” that Simon didn’t halt filming and allowed him to present such a moment of vulnerability to the world.

“It was scary, frustrating,” Simon admits. “You’re in denial and then you’re overwhelmed by this change in your life because you now have a disability. But even though it wasn’t pleasurable any more, I started to think that this was some new information that I needed to absorb into the piece. I started to focus on sounds, not from computers or synthesisers, but acoustic instruments used in unusual ways.”

Another startling moment in In Restless Dreams is an extraordinary, minute-long monologue in which his friend and collaborator Wynton Marsalis – trumpet legend and the first jazz artist to win the Pulitzer prize – reveals the things they talk about, everything from “being divorced, having children” to “race relations in the United States, the direction the country is going in” to “Bach v Beethoven, Duke Ellington’s musical output in 1962, what does it take to write a song …” There’s another sequence where the pair passionately debate the merits of leaving mistakes in recorded music; Marsalis argues that making something too perfect loses the soul.

“He said that to me because I’d sang the whole Seven Psalms to him in a car as it was being made,” Simon says. “He’s like: ‘Man, this is soulful. You’ve gotta record it like this!’ And that’s true when you’re sitting in a car, but not when you’re listening to a recording. He’s right about not taking away the humanity, but if I sing a note flat will it be worse if I go back and sing it better? I always say: ‘The ear goes to the irritant.’ When I hear something I don’t like in a piece of music, I want to change it.” But, he says, “this idea that I’m a perfectionist is not correct”.

Simon, who was an English literature major and quit law school after one semester, had wanted to be a singer-songwriter since he was 13. He experienced the big bang of rock’n’roll in the 1950s, when teenagers suddenly had their own culture. “It’s impossible to convey how exhilarating it was,” he says, his voice quickening with glee. “Little Richard, Jerry Lee Lewis, Fats Domino, Elvis Presley, Johnny Cash, Ray Charles, doo-wop groups from New York, Philly, Chicago, New Orleans. An enormous amount of musical information that came in a very few years exactly at the age when you are really open to it, absorbing and falling in love with it.”

In Restless Dreams tracks Simon’s beginnings from meeting Art Garfunkel in Parsons Junior High, harmonising together over Everly Brothers songs, having a minor 1957 hit as Tom & Jerry with Hey, Schoolgirl (which Simon wrote when he was 15), to his early 60s songwriting apprenticeship in New York’s Brill Building, working alongside Carole King. Gibney sees this period of “elbow grease, discipline and developing the craft” as crucial to Simon’s subsequent ability to translate the more mystical elements of his creativity into something catchy, tangible and universal.

Gibney says Simon talked affectionately about the periods he spent playing English folk clubs in 1964 and 1965, which he found most welcoming after struggling for bookings back home. The singer refers to those travels in Seven Psalms’ Trail of Volcanoes, written six decades later. “The two years I lived in England were amongst the happiest times of my life,” Simon says. “I wrote a couple of songs that became hits with Simon and Garfunkel [including Homeward Bound, believed to be inspired by Widnes station]. I learned how to do a set, perform, talk to an audience. I was 23, 24 and was living away from a home in a country that seemed magical to me. It was an extraordinary time – Carnaby Street, the Beatles and the Stones, the Who, mods and rockers and then the folk movement. Martin Carthy, Davey Graham, the Ian Campbell Folk Group, Ewan MacColl and Peggy Seeger. Then contemporaries like Bert Jansch, John Renbourn and later Sandy Denny and Jackson Frank. I met Kathy [Chitty, his first love, who inspired Kathy’s Song and many others]. It really was a beautiful time.”

Judith Piepe, a passionate folk champion/activist/broadcaster who’d fled Nazi Germany and would often put up folk singers, gave him a room in her east London flat and got him on the BBC’s TV daily religious show, Five to Ten. He performed The Sounds of Silence (as it was first titled), which had initially appeared on Simon and Garfunkel’s poorly selling 1964 US debut, Wednesday Morning, 3AM.

Ears were pricking up, although Simon was still often performing in pub backrooms where, he has said, promoters would tell audiences: “Shut up! You’ve ’ad yer bingo! Now give the turn a go.” Meanwhile, in New York, producer Tom Wilson was overdubbing the electric guitars and drums that in January 1966 would take the retitled The Sound of Silence to the top of the US charts, a place they soon knew well.

Gibney’s problem was a dearth of UK footage. “Paul wasn’t famous then,” he explains. “People weren’t following him with cameras, so thank God for the precious photographs we found. Weirdly, we found some footage of Judith Piepe walking into a club, and if you look very closely she passes a poster for Paul Simon. Incredible. So we were trying to re-create a dreamlike version of that period he felt free, alive and open.”

Simon and Garfunkel’s 1970 breakup presented a different challenge: how to portray a split rooted in what Simon calls an “imbalance of power” between songwriter (Simon) and co-singer (Garfunkel), plus conflicts over the latter’s new film career. “I did talk to Art, but he wouldn’t agree to talk to me with microphones on,” Gibney explains. “Nevertheless, we dug deep into the archives to find some rather pointed remarks from Artie to give a balance. It was delicate and difficult but that was an instance where there was an amount of back and forth between me and Paul. He was willing to dig deeper and go further in terms of talking about it.” The documentary features Simon’s candid admission: “Maybe it was my Freudian trauma. My mother said to me once: ‘You have a good voice, but Arthur has a fine voice.’”

In Restless Dreams subsequently chases Simon’s solo musical adventures, to Jamaica (where he recorded 1972’s early white reggae hit Mother and Child Reunion with members of the Maytals), to Brazil for 1990’s The Rhythm of the Saints and most famously to apartheid-era South Africa, where – after hearing a bootleg tape of township music – he worked on 1986’s Grammy-winning, 16m-selling Graceland, amid allegations that he had broken the cultural boycott of the time.

“The boycott was intended to stop white, western musicians performing in apartheid South Africa and making money,” Gibney argues. “What Paul was doing was amplifying that music and letting the whole world hear its power. People suddenly started thinking: ‘This music is wonderful. Why is South Africa treating people who make it this way?’ And then the world changed.”

In Restless Dreams includes Michael Lindsay-Hogg’s incredible footage, much of it unseen, of a 1987 concert in Harare, Zimbabwe. Exiled South Africans Hugh Masekela and Miriam Makeba are in Simon’s all-African band and a racially integrated audience of Zimbabweans and visiting South Africans are filmed dancing joyously together, which would have been illegal across the border.

“When I look at the footage, it’s nostalgic and a little bit sad because almost everyone in the band is gone, including Hugh and Miriam,” sighs Simon. “That band was magical, and then being able to go back and perform in Johannesburg [in 1992] was extraordinary. You know, in Africa I met friends I never would have met who stayed friends my whole life. It was a privilege for me as a white American to participate in other cultures. I’d say that Zimbabwe concert, the two concerts I did in Central Park – with Artie [the 1981 reunion attended by half a million people], then on my own – and the concert in Hyde Park for the 25th anniversary of Graceland are the peaks of my performing career.”

Which hasn’t stopped yet. The hearing loss precludes playing with a full band, but he is finding ways to perform acoustically. Two weeks ago, he played seven songs with two guitarists at a fundraiser for the Stanford Initiative to Cure Hearing Loss, his longest performance in five years. “I’m hoping to eventually be able to do a full-length concert,” he says. “I’m optimistic. Six months ago I was pessimistic.”

He’s also written two new songs. “One of them, a duet with Edie, is different from anything I’ve written. I might just put it out into the ether, see where it goes.” He says he has no interest in developing the “extraordinary stadium spectaculars” of some of his contemporaries. “I’m interested in relearning how to write songs, like I did in England, and developing new acoustic sounds.

“Maybe I’m something of a lone wolf in that respect,” he chuckles. “But I’m kinda interested in the conclusion of where my thinking in music finally ends up.”

• In Restless Dreams is in UK cinemas on 13 October – Paul Simon’s 83rd birthday – and available digitally and on Blu-ray from 28 October