A long, long time ago – five decades to be exact – America was roiled by wrenching generational showdowns, massive street protests, and a blazing array of social justice movements. Now, half a century later, similar events and dynamics dominate the public conversation. So, perhaps, it’s poetic that precisely five decades have elapsed since a song that captured all that cultural turmoil, American Pie, became a smash hit. “It’s a song that spoke to its time,” said Spencer Proffer, who has produced a comprehensive new documentary about the song, titled The Day the Music Died. “But it’s just as applicable now.”

In fact, American Pie has only gained in fans and expanded in meaning as it has hit successive generations and generated fresh covers. Over the years, it has been interpreted by artists from Madonna (who created a commercially triumphant, if aesthetically limp, take in 2000) to Garth Brooks to Jon Bon Jovi to John Mayer. Throughout the years, journalists have subjected the song to a Talmudic level of scrutiny, while its songwriter, Don McLean, has doled out dribs and drabs of insight into his intent. By contrast, the new documentary offers the first line-by-line deconstruction of the song’s lyrics, as well as the most detailed analysis to date of its musical evolution. “I told Don, ‘It’s time for you to reveal what 50 years of journalists have wanted to know,’” Proffer said. “This film was a concerted effort to raise the curtain.”

In addition, it offers an emotional account of the tragic event that McLean used as his jumping off point for the larger story he wanted to tell.

The event, which McLean dubbed “the day the music died”, shattered the pop world of its day and had a formative effect on the songwriter. On a frigid night in 1959, a small plane carrying Buddy Holly, Ritchie Valens and JP Richardson (The Big Bopper) crashed in a corn field in Clear Lake, Iowa, minutes after take-off, killing everyone on board. The documentary begins with that event, traveling back to the Surf Ballroom, where the stars played their final show. The film-makers scored a coup by bringing on camera a man who saw that fateful concert, as well as the man who owns the aviation company that rented the doomed plane. More, it features a moving interview with Valens’ sister Connie, who we see thanking McLean for immortalizing her brother in song.

The first part of the film covers McLean’s early life, including his time as a paper boy in the suburb of New York City where he grew up. In an extensive interview for the film, McLean talks about delivering the paper that carried news of the crash, something he alludes to towards the start of the song’s lyrics. At the time, Buddy Holly was his musical idol. If his death instigated the song’s words, a more personal loss altered the course of McLean’s life. When he was 15, his father died suddenly of a heart attack. “That had a profound effect on him,” Proffer said. “He has carried the death of his father in his soul.”

In his grief, McLean threw himself into music, developing a talent promising enough to earn him gigs in the folk clubs of Greenwich Village as a teenager. He found a role model in the Weavers, particularly in Pete Seeger, whom he befriended. The primacy of storytelling in the group’s songs, as well as their socio-cultural grounding, served as a template for certain aspects of American Pie. From Seeger, he also learned the value of the singalong. One clear draw of American Pie is its chorus, which anyone can mimic. The simplicity of its melody echoes children’s music. “It’s like a camp fire song,” Proffer said. “Everyone is invited to sing.”

Some of the song’s lyrics even quote nursery rhymes, including “Jack be nimble/Jack be quick.” The cover of the American Pie album underscored the connection by featuring McLean’s thumb in the forefront to reference another nursery rhyme about Little Jack Horner, who “put in his thumb/and pulled out a plum”.

At the same time, the song’s message couldn’t be more adult. “For me, American Pie is the eulogy for a dream that didn’t take place,” says the song’s producer, Ed Freeman, in the film. “We were witness to the death of the American dream.”

“The country was in some advanced state of psychic shock,” McLean says on camera. “All this bedlam and riots and burning cities.”

The extremity of it all made McLean want to shoot for the moon, creatively. “I wanted to write a song about America, but I didn’t want to write a song about America like anybody ever wrote before,” he says.

That was no small goal considering the number of songwriters at the time who were fashioning their own odes to the disillusion of the American dream. They ranged from Paul Simon with American Tune (which imagines the Statue of Liberty sailing out to sea) to Dion’s version of Abraham, Martin and John (which poignantly addressed the assassinations of Lincoln, Luther King and Kennedy).

McLean’s desire to stand out from the other singer-songwriters who dominated music at the time had a careerist incentive as well. His debut album, Tapestry, released in 1970 hadn’t made waves and his small record company, MediaArts, had little faith in him. Even so, the big statement song he devised to turn that around arrived in a form that defied the most basic edict of a hit – that it last no longer than three minutes. American Pie snaked on for eight and a half minutes, and was stuffed with a fever dream’s worth of cryptic imagery.

In fact, McLean wrote even more verses than the final song held. “He just kept writing,” Proffer said. “If it was over eight minutes, it could have been 16.”

In that sense, it shares something with Leonard Cohen’s Hallelujah. In both songs, verses were written by the author and discarded (though many more were ditched in Cohen’s case). Both songs have also gained in stature and impact over the years. (Coincidentally, Cohen’s song is also the subject of new documentary titled Hallelujah: Leonard Cohen, A Journey, A Song). Yet, at the core, they differ fundamentally. “Hallelujah is a spiritual study,” Proffer said. “American Pie is a sociological study.”

Often, it’s a coy one. The lyrics brim with coded references to kings, queens, and jesters, along with a host of cultural figures that, together, turn it into a virtual pop quiz: “Name that reference!” The result has made the song especially involving, teasing the listener to solve its puzzle. “Every time you listen, you think of something else,” Proffer said.

In the film, McLean dismisses some of the most common speculations about his reference points. Elvis wasn’t the king in question. The “girl who sang the blues” wasn’t Janis Joplin, and Bob Dylan wasn’t the jester. In 2017, Dylan commented on his alleged reference to Rolling Stone: “A jester?” he said. “Sure, the jester writes songs like Masters of War, A Hard Rain’s Gonna Fall, It’s Alright, Ma.” I have to think he’s talking about somebody else.”

As fanciful as some of McLean’s lyrics may have been, its core reference to the “day the music died” turned the song into a history lesson for those born too late to remember that event as crushingly as McLean did. Even when the song first appeared, over a decade had passed since the crash, the equivalent to a thousand years in the fast-paced life of pop.

One of the most interesting sections of the documentary offers a granular dissection of the evolution of the song’s arrangement. It didn’t find its true groove until they brought in session keyboardist Paul Griffin, who has played on seminal recordings by everyone from Dylan to Steely Dan. His piano parts brought a gospel fervor to the song, as well as an extra pop bounce. Hooks like that helped a song of daunting density and length become beloved by millions.

To deal with its length, McLean’s record company had a clever idea. The first half of the song appeared on the A-side of the single, while the second was consigned to side B. The result turned the A-side into a cliffhanger the listener had to see through to the end. The subsequent demand forced AM radio stations to play both sides. At the same time, FM radio – whose mandate was to go deeper and play longer – was reaching its commercial apex at the time. Issued at the end of 1971, American Pie hit No 1 by January of 72, where it stayed for a full month. For 49 years, it held the record for the longest song to hit No 1 – until Taylor Swift’s 10-minute cut, All Too Well, broke it.

Interestingly both songs have a certain anger. But, with the passage of time, McLean’s piece has morphed considerably in the public consciousness. Today, it is sometimes performed, and interpreted, as if it were some kind of rousing sequel to The Star-Spangled Banner. In the film, one fan describes it as a song that makes “you pause and be grateful for everything that you have”.

Garth Brooks says in the film that it’s a song “about that drive of independence, that drive of discovery … of believing anything is possible”.

Both views couldn’t be more baffling, given the wall-to-wall sadness and disgust of the actual words. In fact, American Pie ends with “the father, son and the holy ghost,” so appalled by the state of the country that even they – the ostensible saviors of mankind – cut and run for the coast. “People aren’t thinking about what (the song) really means,” Proffer said. “They’re thinking about how it makes them feel.”

If such reactions wildly decontextualize the song, the film can serve to re-contextualize it. More, it aims to extend its legacy by featuring new versions of the song sung by someone of the current generation (the 24-year-old British singer Jade Bird) as well as artists from another culture (singer Jencarlos and producer Maffio, who created a version in Spanish). “It’s exciting to know that something that happened 50 years ago can resonate to later generations,” Proffer said. “Through listening to the song, people get a glimpse into what life was like then and what it came to be today.”

• This article was amended on 19 and 22 July 2022. American Pie held the record as the longest song to reach number one for 49 years, not 39 years as an earlier version said; the session keyboardist Paul Griffin’s name was misspelled as Griffith; and references to the song Abraham, Martin and John were changed for clarity.

• The Day the Music Died: The Story of Don McLean’s American Pie is now available on Paramount+