Jason Lee knew he was in trouble when he stepped on the set. The year was 1992, Sonic Youth were at their peak and he was starring as a doomed skateboarder in their latest video. As a music obsessed, pro skateboarder with acting aspirations, he felt he had a point to prove. To add more pressure, it was for the song 100% – the band’s classic ode to a murdered Black Flag roadie – and the video was being co-directed by one of his skateboarding friends (some guy called Spike Jonze).

“I was really trying my hardest to focus,” says Lee. “I was like pretending to be Robert De Niro on the set, really trying to get into it and make it count and make it real and believable.”

Admittedly, lying on some grass with a white sheet pulled over you isn’t exactly the Russian roulette scene from The Deer Hunter, but for someone with limited range it was still a push. “I don’t know Shakespeare, I can’t do accents, I didn’t study, I don’t read many plays, I’ve never been in a play. I’m just a skateboarder from Orange County who fell into acting because I like watching movies,” says Lee, down the phone from his home in Los Angeles.

Lee believes his career has been a series of happy accidents. He became a pro skater during the massive knee-pad boom in the 80s. He charmed Clerks director Kevin Smith at an audition, became friends with him and ended up with life-changing roles in Mallrats and Chasing Amy. Cameron Crowe loved him so much he made him a jaded rock star in Almost Famous. Then he transferred to TV at a time when moving from film to small screen was still seen as a step down. But he landed on his feet and wound up in one of the biggest network hits of the last two decades as the titular star of My Name Is Earl.

Now at 50 years old, he’s a father of five dealing with the reality of home schooling. There are still hints of his slacker past – anecdotes are peppered with the occasional “duuuude” – but like other skaters, such as Jonze, Ed Templeton and Mark Gonzales, Lee found art as he reached middle age.

In the past decade, Lee has moved away from the screen and taken up residence behind a tripod. He got interested in photography after being on set and wanting to “understand things a little bit more as an actor”. He bought a camera, and a light meter, asked a few questions and started shooting. “I started just roaming America and documenting its weirdness and its strange beauty. That’s been going on now for 14 years,” he says.

He has released a series of photography books that cover the freewheeling yearly American road trips that he takes either by himself or with his children. “I love road tripping. I love America,” he says. “I’m fascinated by all its shapes and sizes and forms and all the shit that just kind of gets left behind.”

The result is a mix of expansive landscapes and smaller snapshots of a beautiful but declining nation that recall the work of William Christenberry and the Dutch Paris, Texas cinematographer, Robby Mueller.

His latest book In the Gold Dust Rush – shot in New York state, Texas, Arizona, New Mexico, Colorado, Utah, Nevada, Oregon and northern California – has echoes of Peter Bogdanovich’s The Last Picture Show, documenting small town America as it erodes. Old petrol station signage sits on mounds of dirt, one-horse towns silently stew as Lee’s camera surveys the scene. Outbuildings look like they could be blown over by a stiff breeze. “America is beautifully shitty,” says Lee. “That’s the lay of the land. That’s what you might find in the desert, that’s what our suburbia looks like.”

Lee knows all about suburbia: he grew up in Orange County. His family lived in a 70s tract home with a skate ramp out front and BMX bikes. He skateboarded to school, went to the liquor store and played video games. “It was just that typical 70s and 80s American suburban upbringing; staying out all night, causing trouble, going to the arcade and getting slices of pizza at the strip mall,” he says.

You can see how his being cast as Brodie Bruce, an overeducated, underemployed smart-arse slacker in Mallrats, worked so well. He got the role through his then girlfriend, Marissa Ribisi. Her mother knew the casting director and asked if he could get an audition. It wasn’t a done deal though: Smith insisted he came out before the other cast members to nail his lines.

Just before the cameras rolled he had another on-set epiphany. “I was in a world that wasn’t mine, it wasn’t skateboarding,” he says. “I had all these people around me relying on me and I’m in Minnesota in the middle of nowhere shooting in some empty mall.”

Getting through it was his proudest career moment; it’s the role he’s most happy with to this day. So what does he think of the idea that the 90s “nerd masculinity” culture that Mallrats was part of developed into toxic online misogyny: could Brodie be the kind of person who believes in QAnon and supports Trump?

There’s a pause. “Brodie would not support Trump. No,” he says, emphatically. Did it annoy him that he became synonymous with the “slacker” tag? “I was closer to Brodie than not, right? I was proud of the movie … it’s an ultimately redemptive, sweet dick-and-fart-joke movie. I don’t know that it’s got some underlying message that needs to be picked apart.”

We’re speaking less than a week after the rioting at the Washington Capitol, so it’s understandable that Lee isn’t too happy with the suggestion the character he’s most proud of playing is being called a Trump supporter. There’s also the fact that as well as skateboarding and slacker culture, the other unavoidable “S” in Lee’s story is Scientology. According to a 2015 Gawker piece by Lee’s former partner Carmen Llywelyn, Ribisi introduced Lee to the organisation in the 90s – and blamed it for their breakup.

Surely any organised religion would be anathema to a dick and fart joke-loving skateboarder? Why did Lee decide to join the Church of Scientology? “Everybody wants answers,” says Lee. “Everybody wants to feel less depressed or less anxious or they want to try to understand some issue that they might have with themselves.”

He adds: “Ultimately, it was just not for me. That’s it. I wasn’t really involved going back many, many years now.”

The association stuck though – during My Name Is Earl’s run there was a theory that the show was riddled with Scientology subtext. His work with other prominent Scientologists, including Beck, Ribisi’s brother Giovanni and Tom Cruise (in Vanilla Sky), added more fuel to the fire. When Lee and his family moved to Denton, Texas, in 2016, he gave an interview assuring the residents that he was no longer a Scientologist and that he wasn’t moving there to build a church.

“Normally, I don’t give a shit,” he says. “But people were like, ‘Well, I hear he’s buying up all the buildings and he’s gonna do this to the town and he’s trying to do that.’ That was more to just say, ‘Whoa, guys, we’re not opening churches, we’re not tearing down buildings, we’re not trying to ruin anything, we love it here.’”

Lee and his family moved back to California after four years in Texas – he married the Turkish-Australian actor Ceren Alkac in 2008. The break from Los Angeles was due to a desire to change scene, try photography and refresh after a long run on My Name Is Earl, he says. “Back in the Earl days we shot as many as 20 episodes a season – that was a fucking long, long haul. Earl was an amazing experience but certainly a grind, it was a lot of work. I think that had something to do with it.”

Lee is currently trying to get a dark comedy (which he will star in and executive produce) off the ground. There’s a planned Mallrats sequel that he says will prove Brodie isn’t a Trump supporter once and for all. But going back to acting isn’t a case of unfinished business, Lee insists. He’s had more success than he ever imagined when he stepped on to that Sonic Youth set.

Anyway, he’s found something else to occupy his time. “Some people go hiking, some people go on hunting expeditions or whatever, I jump in my car with my camera gear, drive around and shoot photos,” he says. “That’s what I like to do.”

• In the Gold Dust Rush is published by Stanley/Barker.