Boots Riley doesn’t do soundbites. He speaks in dense, meaty paragraphs full of ideas that, in the course of explicating themselves, branch out into sub-ideas which then branch out into sub-sub-ideas. In dealing with such an interview subject, a reporter will sometimes follow a long response by restating the gist of what’s been said, looking for confirmation that everyone’s on the same page. When I take this tack – suggesting that his 2012 album and newly released feature film, both titled Sorry to Bother You, offer two perspectives on a single story – he rebuffs me.

“I mean, that makes for a cute, concise kind of slogan,” he concedes, during a candid interview outside of a screening room in New York’s posh Tribeca neighborhood. “I wish I was that type of person, who was better at marketing. But I don’t like that; that’s actually not how I think of it.” He goes on to explain that while the two works draw influence from the same set of themes, neither was intended as an adaptation of the other. He will not be mistaken.

Just as his work defies easy classification, so too does Riley himself refuse to be placed in a tidy media narrative. He’s a dedicated radical with the street cred to prove it, a wicked humorist oscillating between surreal satire and gallows slapstick, a rapper alone in a genre he created, an authority on class theory who loves to party, and now, a film-maker wending his way through an industry he doesn’t quite consider hostile territory, but doesn’t trust, either. Messy as the task might be, we’re all going to have to figure out what to make of Riley, because he doesn’t intend on going anywhere any time soon.

At age 47, two and a half decades out from the release of his first album as ringleader of the Oakland hip-hop collective the Coup, Riley has steered his career closer to the mainstream than ever without compromising his deeply held political convictions. No easy feat, as shown in Sorry to Bother You, his debut as director. The film is an incendiary look at the trials and tribulations black men face in an economy exploiting and demeaning them before casting them aside. Riley transmuted his own frustrations from dealings with record label suits into a fairytale nightmare of late capitalism: Cassius Green (Lakeith Stanfield), a telemarketer, finds that his commissions soar when he puts on his “white voice” and tumbles into a corporate behemoth with darker intentions even beyond their stated mission of sanitized slavery. With ribald humor and outrageous provocations – there’s a nearly nude recitation from the Berry Gordy-produced film The Last Dragon, and a number of jokes revolve around a specific aspect of a specific animal’s anatomy – Riley has cloaked a subversive anti-establishment screed in allegory.

“Science fiction is where a lot of radical writers go to hide,” Riley explains, “because you can conceal your ideas in metaphor, sometimes to the point where it doesn’t matter at all. If metaphor is so much that that’s all there is … OK, so originally, George Lucas was going to follow American Graffiti with Apocalypse Now, but his version was going to frame the Viet Cong as the protagonists. We follow them as they go into the US-occupied territory and encounter their Kurtz. But he couldn’t get it funded, and his budget was way, way lower than Coppola’s ended up being. So he said, ‘How about we put it in space?’ And so Star Wars was born. The most popular piece of pop culture after the Bible was written against American imperialism, and nobody gets that.”

Riley has spared no effort to ensure that his call to arms doesn’t get drowned out by the peals of laughter. Steeped as the film is in an antic sensibility, the themes of oppression and organized resistance aren’t shuffled off to the subtext. Cassius bears witness to the insidious process by which institutions nudge the unthinkable into the realm of acceptability bit by bit, and joins a rebel faction putting Riley’s fiery ideologies into practice.

“I’ve done a lot of fighting to stay in Oakland, with my music,” Riley is proud to say. “At different times in my career, people are like: ‘You should really go to LA, because that’s where you can have this meeting and that meeting …’ But I also wanted to stay in Oakland, because to me, it afforded me more power. The power to be part of organizing campaigns, the power of knowing people and being known by my people. Someone says, ‘You need your car fixed,’ I know the guy. I think that helps for an artist to know their community. I fought for myself to stay there. I should’ve fought harder for everyone to stay there. I should’ve been part of a campaign for rent control and higher wages.”

A seriousness and concentration creep over Riley’s face when the conversation turns to his hometown, which also provides Sorry to Bother You with its vibrant setting. He’s fiercely defensive of Oakland – he has to be, as the city’s integrity and culture have recently come under attack from developers and other agents of gentrification. The Silicon Valley boom that cannibalized San Francisco has spread its plague across the Bay to drive up the cost of living and put neighborhood spots out of business.

Riley breaks it down: “For me, the version of the southern plantation-style mansions that we have in Oakland, as far as the wealth is concerned, they’re a little hipper. The style is sometimes self-deprecating as part of its hipness. The style tries to be really basic, recall the time when someone could order a 25-cent cup of coffee. I find it kitschy, the whole PBR thing. But for me, gentrification isn’t about what kind of people are moving in, it’s about the people who are forced out. Go around Oakland, and parts of it are a fancy ghost town. Places where friends used to live, you see reminders of what was once there. It’s like the ghost town from Martian Chronicles. You can tell something’s going on.”

Principled, outspoken, and partisan, Riley doesn’t act like the kind of person who’d make a go of it in showbiz. The nature of the movie industry is such that finagling any sizable amount of money means getting in bed with a company owned by a gargantuan media conglomerate, but Riley found an open-minded collaborator in Megan Ellison and her Annapurna Pictures distributor. They presented a happy medium, just big enough to give Riley the nationwide rollout he felt his picture deserved, and not so big that suits would start clucking over content biting the hand that feeds. Still, he held on to a healthy cynicism:

“Annapurna is, genuinely, some of the coolest people in Hollywood … but the whole indie capitalists versus big capitalists thing, that preference comes from a lack of class analysis,” he states. “There are practical things that happen when someone’s independent, they can make the choices they want to make. You don’t need a meeting of 200 people that ends with everyone finding something they least disagree with. But, still – it’s like this: I’d rather have the local IPA, because it tastes better, but it’s not ethically superior.”

Cinema – the largest, most money-reliant sector of the entertainment world – hasn’t historically been kind to dissenters. (As the old wisdom goes, if you want to send a message, try Western Union.) Riley has arrived at a career juncture in which, for the like-minded rabble-rousers that came before him, left only the options to sell out or give in to obscurity. But the fury of Sorry to Bother You has struck a chord with a furious nation, as luminaries from the music and film communities have lined up to sing Riley’s praises while black test audiences cheer in cathartic release. Riley’s insistence on being Riley, whoever that might be, paid off; instead of angling towards palatability or chasing trends, he’s reshaping the business in his own image.

“Soon, in about six months, you’ll start to see the Sorry to Bother You-style movies, except with dinosaurs and the Rock,” he deadpans. He doesn’t sound all that pleased about it, however, and why should he be? He’s currently promoting a film about the ruthless efficiency with which white industries dilute, reproduce, and commodify black excellence. In the same hot seat as Cassius Green, Riley has concluded that for both himself and the culture at large, the only path to progress is by keeping it real – that is to say, by inciting action in our own communities on a realistic scale. Ever the enigma, Riley reveals one last contradiction with his parting words: that a jaded man can still feel firmly, earnestly hopeful.

“There are movements happening that aren’t just online: Black Lives Matter, and before that we had Occupy. Even in the cold, marketing-minded sense of ‘What are people into right now?’ it’s all making us feel a little more free … If we want more radical movies, we’ve got to first have more radical movement out in the world. That’s where it starts.”

• Sorry to Bother You is released in US cinemas on 6 July with a UK date yet to be announced