Sean Penn walks into a low-key restaurant not far from his house and slides undetected into a seat at the back. It is a misty day in Malibu, 15 miles north of LA, and the actor appears in familiar, I-don’t-care guise: beat-up leather jacket, brushfire hair (which, in light of the views he will air about female vanity in Hollywood, I feel bound to report may be more vibrant than nature intended), with the jittery energy of a man who has stood for too long next to very loud speakers. The 57-year-old greets me affably, to my surprise: while Penn may be long admired for his acting, he is denigrated for almost everything else. He is known to be pugnacious, short-tempered, long-winded, a man who yells at photographers and smokes on TV. And now he has written a lunatic novel. “It’s the most fun I’ve had professionally,” he croaks and leans back in his chair.

A few days before our meeting, Bob Honey Who Just Do Stuff was the subject of a full-page newspaper ad in which the publishers, rather cleverly, ran lines from the book’s worst reviews and asked readers to make up their own minds. The novel, which follows the adventures of its eponymous hero as he goes around selling septic tanks and murdering old people, is a satire of American consumer culture and the cult of image, and for weeks its excesses have been gleefully documented. Penn writes as if every word has been put through a thesaurus: for “face”, read “dermal mask”; for “quiet”, “lingering symphony of silence”; a man with good hearing has “extraordinary auditory augmentation efficiency”. So it goes on, until we reach: “Effervescence lived in her every cellular expression, and she had spizzerinctum to spare.”

Cherrypicking these extravagances has provided us with a jolly good laugh, but here’s the odd thing: taken as a whole, there is something quite thrilling about Bob Honey, a hallucinatory effect that makes the novel at times almost feel like a piece of performance art. Linger long enough in the world of Penn’s prose and one starts to lose all purchase on reality, a sort of brain slide that Penn insists was intentional. “It was meant to be challenging,” he says. “Meant to have you stop-the-rhythm to go and look something up.” If he talks in a word fog, it is because he prefers “to be understood by osmosis. If I’m being brief, I assume [people] will fill in, or if I am not being brief” – the far greater likelihood, on the evidence of our chat – “the tangents would be connected by the other person.” He hoped the book might work along similar lines.

The result, while largely insane, does provoke an emotional response in the reader and for that reason alone might be judged a success. “I wrote the Sean-by-osmosis version in about 30 days,” Penn says, after which “I rewrote for two years”, before selling it to Simon & Schuster, where “I got lucky; Peter [Borland] totally got the book. I felt I was being supported by a really smart reader.”

Was he edited?

“He wasn’t looking to change anything,” Penn says cheerfully. “But he did send me great notes which said, ‘I don’t understand this. Do you want me to?’” (In most cases Penn replied “yes”; in some, “not so much”.) The suggestion he was indulged because of his celebrity is one he finds baffling. “No, because I wouldn’t have done it any other way. When people listen to the advice they get, it usually robs the soul of the piece. In the movies, they’re not thinking about whether it has impact on them; they’re thinking about whether it will impact the box office.”

This rather quaint view of publishing – as an industry devoid of commercial principles and certainly not the kind of business that might judge a book by a famous actor to have greater value as a car crash than as something more credible – is the main source of Penn’s enthusiasm for novel writing, which he sees as a less mediated form of expression than acting. But why would he trade in something he is indisputably world class at – Penn has won two Oscars, for Milk and Mystic River – for something he is indisputably not? The answer is burnout, what he calls “my great source of alienation from my peers, actors who would rather sell a movie than make one”. Where these actors are “brands”, and where even a movie he loves “is the sum of compromises at its best”, Bob Honey, by contrast, is “all mine. It’s been said you never complete a film, you abandon it, and I didn’t abandon this. I stand by every line exactly how I wrote it.”

Has he seen the worst reviews? “I have a sense of them. It’s keep-your-day-job shit. A bunch of children, schadenfreude, whatever. And in the end, some of the best laughs I’ve got have been the one-liners that have come out of the worst reviews.”

Through long espousal of leftwing politics, including vocal protests against the Iraq war and support of Hugo Chávez and Raúl Castro, Penn is used to being unpopular. Does it hurt to be mocked? “Well, mocked means the person’s an idiot.” Penn smiles crookedly and shutters his lids. “I’m not vulnerable at all.”

There are two images of Sean Penn that rival the memory of his greatest screen roles. Up there with Carlito’s Way, or Dead Man Walking, or his role as Harvey Milk – the 2008 biopic of California’s first gay elected official and a performance for which one could almost forgive Penn any amount of bad behaviour off-screen – there is the shot of him in a boat rescuing survivors of Hurricane Katrina, and the one of his handshake with Mexican drug lord El Chapo.

The latter, taken two years ago when Penn interviewed El Chapo for Rolling Stone, brought down on his head almost as much opprobrium as Bob Honey, for its perceived amateurism and for the way it seemed to apologise for a man with so much blood on his hands. Penn remains unrepentant. “Just because someone writes about stuff doesn’t mean they know about it. I know fathers who don’t know their own families. Most of these journalists on this subject know fucking nothing and most never read my piece. They read the excerpts. Then you get a talking head like a Don Winslow [on CNN], I’ve read his books cover to cover and he’s a functioning moron, and people think he’s something. That’s their resident expert. It’s like Henry Kissinger is their resident expert.”

The gung-ho activism is more complicated and, on closer inspection, much more admirable than the caricature suggests. If there is precedent for Penn wading into a job about which he knows nothing and nailing it through sheer bloody-mindedness, it is here, in his experiences with aid.

In 2010, Penn went to Haiti in the wake of the devastating earthquake and set up the J/P Haitian Relief Organization, an agency that has, by all accounts, been enormously successful and helped thousands of Haitians find shelter and medical relief. In a piece in the New York Times in 2011, by which time Penn had been in Haiti for almost a year, veteran aid workers lined up to praise him. The question of what he was doing in Haiti in the first place has roots in his inability to find equilibrium at home. The most persuasive passages in Bob Honey are those in which Penn writes of his character’s insomnia. When I ask if this comes from experience, he takes a big breath. “So this goes back as far as I, or my mother, can remember. I was born at 3.45am and I generally fall asleep at 3.45am. I’ve been to Zen centres, done sleep studies, everything. I could never get myself to go to sleep. I’d get up, sometimes, with no sleep at all. And I was getting older, so I wasn’t able to push through this thing mentally.” Eventually a doctor prescribed Ambien: “But it doesn’t work like it used to. It doesn’t keep me asleep, but I get to sleep and at least get three hours. Then you’re watching the clock for the time you have to get up, and it’s getting closer and closer.”

Some of Penn’s wilder behaviour makes more sense, perhaps, when looked at in the context of a man who has never had more than three unbroken hours’ sleep.

“Yuh. Judgment issues,” he says, raising his eyebrows in what is almost certainly a reference to the time, in 2009, when a court made him attend anger management classes after an altercation with a photographer.

It probably didn’t improve your temper, I suggest.

“No. Nor the thoughts that lead to a temperamental response.”

One result of this is that Penn often feels himself to operate best in crisis mode. He quotes a line from Cutter’s Way, a movie from 1981 starring Jeff Bridges and John Heard in which a woman kills herself to get away from her alcoholic husband, and the final line of which is: “You know, the routine grind drives me to drink. Tragedy, I take straight.”

“There’s been a lot of that in my life,” Penn says and he relates it, obliquely, to how he is able to function effectively in a destabilised environment such as Haiti after the earthquake. “I think tragedy can also be urgency. I find myself going calm, and being demanding. I can bring an American arrogance to anywhere in the world.” He smiles. “Look, in the movie business you tell your crew: ‘I need this set done yesterday.’ When I got to Haiti and I saw how NGOs worked, it would be like: ‘This patient needs this, now, and this is the person to get in touch with.’ Then I had six other people asking me six other things. When I circled back half an hour later to the first one and said, ‘Did you get it?’ they said, ‘We’re waiting for a response, we sent an email.’ I said, ‘You sent an email? Give me your phone, I’m going to call.’ And it drove me fucking mad.”

Pretty quickly, Penn’s outfit became known as an agency that got things done quicker than the NGOs and he calls it “the easiest kind of work to do in an emergency, because it’s emergency related”. It was triage. He nods. “It’s the easiest, commonsense work. There’s a kid in the street, a truck’s going to hit it, do you know that the sidewalk is clear of paedophiles? No. I’ll make my apologies later.”

It is fascinating that the self-important urgency of a Hollywood movie set can be translated to a disaster zone, but think about it, Penn says: “It’s the same business, the stakes are higher.” There’s even overlap with the sanitation logistics of shooting a movie. “You’ve gotta get porta potties for people.”

Was he surprised when allegations were made last year that aid workers, including Roland Van Hauwermeiren, the head of Oxfam in Haiti, used sex workers? “I don’t know why it wasn’t a bigger scandal earlier. I was reading about it in the Haitian press when I was in Haiti in the beginning. Roland Van Hauwermeiren was probably the best country director of an organisation there. Of course, you’re not the best anything if you’re looking the other way on that stuff. Or, worse yet, if you’re participating.”

In his own organisation, Penn had a policy of “no tolerance” right down to the rule that “if you smoke one joint, you’re fired”. In fact he ended up relaxing the drugs rule for his indigenous workers, because “how am I controlling that when my local workers went back to their homes at night? I stopped being an idiot with that.” For the imports, mostly Americans or Brits, however, “You don’t come to this country to break its laws.”

And when someone in his organisation did break a rule? Well, Penn says, “There’s no organisation, including my own, where you don’t have something happen. And you deal with it; you certainly want to take it transparent.” The cover-up, he says, is the biggest problem. “I’ve known of situations where big organisations discovered something and did an investigation, and fired somebody – and because they didn’t want a lawsuit from that person, another organisation would hire them; it’s like passing on priests. In our organisation, I’d get on the phone to whoever the violator was and say, ‘If I hear you are ever trying to get a job in one of these organisations, I’m coming to your home town, and I’m going to rent out the movie theatre and show a documentary about you and make sure your parents are there.’”

Why does he think an aid worker, who presumably starts out with good intentions, gets to the point of exploiting desperate women this way? “I can have theories,” Penn says, “but they’re the same theories one would apply to someone who abuses their children. How do you do that?”

In light of these life-or-death stakes, Penn is unsympathetic to what he seems to regard as more trivial abuses closer to home, particularly those involving celebrity exposure. A week before we meet, he appeared in the gossip columns for dining with Charlie Rose, the disgraced former TV host fired from PBS and CBS for inappropriate behaviour around female staff. Penn is not, it is safe to say, on board with #MeToo, which he describes as “not intellectually honest”, “a movement led by mania” and full of “self-aggrandisement and venting”. At the end of Bob Honey, he writes a poem defending Charlie Rose and appearing to lament the demise of Louis CK.

Clearly, I say, Penn thinks his friend was mistreated.

“He says one thing happened; other people said another thing happened. I wasn’t there. What I do think is we maybe should be very careful; for example, are we now going to say we will never celebrate Thomas Jefferson again: slave holder, that’s all you are? Charlie Rose provided one of the only sophisticated dialogue programmes, and I don’t know what the percentages are, but I know that there are some lies that have been told publicly about people; I know of some serious omissions. I’m talking about women towards men.” This is a personal issue for Penn, who in 2016 won a court-appointed apology from the director Lee Daniels, who erroneously implied the actor was guilty of domestic violence.

Romantic relationships don’t seem to be Penn’s strong point. The worst bits of Bob Honey are those involving the character’s ex-wife, a harridan called Helen, about whom he is so vicious, I had wondered whether Penn’s own ex-wife might have commented on it. Penn gives me a dry look. “Which ex-wife?” he says.

Oh. I had momentarily forgotten about Madonna, to whom Penn was married from 1985 to 1989. “The mother of your kids,” I say. (Penn and Robin Wright, who were married for 14 years, have Dylan and Hopper, both in their 20s.)

“No; these characters are amalgamations of a lot of things and she’s not a character in the book.” Penn pauses. “I don’t think the book would be either one of my ex-wives’ cup of tea. They both thought that I talked too much in ways that were too complicated.” He laughs. “I don’t have a lot of conversations with my most recent ex, but my first ex, we have a close friendship.”

What does he think Madonna would say about the book? “I think if she was reading it, she’d say, ugh, I’ll read this some other time.”

Anyway, Penn says, returning to the subject of #MeToo, “I don’t ever want to be in a movement. I don’t trust any movement. The anti-war movement did not sustain. We’re back in wars.”

But going back to his friend Charlie, surely the lesson of this is: don’t wander around naked in front of your staff if you don’t want to get sacked. “I don’t disagree with that,” Penn says. “I go back to my original thing, which is that if that’s what the individual that happened to is asking for, then they have a legal case to go through, or the company has a decision to make. But when the company is making that decision under the pressure of… a movement that is led by mania, I get worried for both [sides].”

There is a libertarian streak in Penn that is impossible to miss. I repeat a wince-making line from Bob Honey, which Penn almost certainly considers to be a liberal statement: “Getting older in America is tough on a woman; seeing what she’ll do to avoid it is tough on a man.” I assume Penn approved of Frances McDormand’s unadorned style at the Oscars this year?

He takes a big breath: “When my kids were growing up, there was a thing called the surgery channel and I would watch this thing and get into hysterical laughter. One slice and you’re a human body! We’re the fucking same! But where I really became a giggling 12-year-old girl was volunteering for cosmetic surgery: you have someone who’s anaesthetised, and they’re doing the most vile things to them, and I would get so hysterical. You bring up Frances McDormand. What do you do to be more beautiful than that? That fucking woman walks into a room and you don’t know if you want to kiss her, eat her farts or bite her toes off, you just want to be around that energy.” As opposed, he says, to “the vanity that errs on the side of violence against oneself, physical violence. It’s crazy.”

But there are market forces at play. You’re still viable as a leading man at 57; a woman is not. “But isn’t it changing? There’s Meryl out there. Also, in terms of the social/sexual part of it, I think more and more – I mean my ex-wife just married a 30-year-old guy she met two months earlier. [Wright is reported to have married a French fashion PR manager earlier this year.] You see that dynamic shifting. When I would try to console a woman in the business who maybe wasn’t getting the opportunities I was because they were a woman, I’d say, well, which parts are you not getting? Because, man or woman, most parts are shit. Most movies are shit. If you look at the really talented women – Julianne Moore never has trouble getting a job. The opportunities are there.”

Does he understand why his countrymen voted for Trump?

“I have a pretty good idea. We became celebritised,” he says. “I don’t buy People magazine. But when people complain that someone took a picture of them or that their privacy is being invaded, I’m like, yeah? Fuck you. I just spent my entire adult life with you telling me that because I want to express myself as an actor, that that comes with the territory. Well, it comes with the territory for you, now, and you bought those fucking People magazines. You made that happen.” He is jabbing his finger at me across the table. “When I go like this, I’m not at my best,” he says. “So two sides that are not at their best, it gets very shallow.” Penn has a disdain for the idea of the personal brand. “Branding is being! Branding is being! The algorithm of modern binary existentialism,” he writes in Bob Honey, a line I read out to him now.

“Yeah,” says Penn, lowering his eyes and looking sage.

What does “the algorithm of modern binary existentialism” mean? He snaps his eyes open. “Well. Binary – two components. When you’re taking an algorithm that’s going to identify the two components, and, there are, I think I talk about in the book also… it’s America today.” There follows a long story about how easy it is to buy components for a bomb in the US.

“I’ve just gone off on the word binary. But I think it’s self-explanatory, in there. Huh.”

Anyway, he says, the reason people voted for Trump is “because I think there’s a mental health problem in this country. It’s rage over reason. There’s a lot going on that was going on in Nazi Germany; we’re finding scapegoats, groups to hate, whether it be of a race, or so-called lefty Hollywood, so that’s part of why I want to write something – I only have my sense of humour to offer. I didn’t want to just be angry all the time.”

And he isn’t. There is an underlying humour to Penn’s rants that is often lost in translation and while I find much of what he says preposterous, arrogant and occasionally delusional, to my surprise, I also find him likable and funny – a genuine American eccentric.

As we get up to leave, I can’t help laughing. “Sorry. I keep thinking about the fact that Madonna thought you were boring.”

“I love her,” Penn says, and laughs. Is she a good egg? “Such a good egg.”

We go outside to his pick-up truck and he introduces me to Flyboy, his six-month-old golden retriever. After Penn has gone, I call a friend with a degree in philosophy. “Does ‘the algorithm of modern binary existentialism’ mean anything?” I ask. After the laughter has died down, he says, “I think he means… in modern life everything is black and white. But existentialism refers to a specific tradition in philosophy; he surely just means existence.” No matter; Penn has a lot more to say. Before leaving he told me: he is hard at work on Bob Honey, the sequel.

• Bob Honey Who Just Do Stuff by Sean Penn is published by Simon & Schuster at £14.99. To order a copy for £12.74, go to guardianbookshop.com or call 0330 333 6846.

Commenting on this piece? If you would like your comment to be considered for inclusion on Weekend magazine’s letters page in print, please email weekend@theguardian.com, including your name and address (not for publication).