It's lunchtime in Richmond, Virginia when a perky Jamie Bell calls. He's there filming Turn, a new American TV drama in which he plays a farmer heading up a team of secret agents during the revolutionary war, and it's "absolutely freezing", he says. But playing war games in the cold is a piece of cake compared to the week he spent with Lars von Trier.

Since 2000's Billy Elliot, Bell's Bafta-winning breakout role, he's been notching up meaty turns with directors of choice, from the indies (David Gordon Green, Kevin MacDonald) to the big guns (Steven Spielberg, Peter Jackson). Von Trier was always on his list. Bell loves his work, calling him "one of the most experimental film-makers who's ever lived," and initially met him a decade ago when he starred in Thomas Vinterberg's Dear Wendy, which Von Trier had written. It was about as typical a Von Trier meeting as he could have hoped for.

"I'd just got to Copenhagen to begin shooting," he says. "Thomas was showing me around: 'We built this set here, this is where the production design room is.' Then we had to walk past this outdoor pool and there was Lars with a towel in his hand. He dropped the towel, he shook my hand – he was stark bollock-naked – and he said, 'Do you want to come for a swim?' And I said, 'I'm all right man, I just got here, I've still got my suitcase, so I'll wait.'"

In 2012, Von Trier sent him the Nymphomaniac screenplay. Bell's character, K, is an S&M dominator visited by Charlotte Gainsbourg's nymphomaniac, who, as such, is forever in need of fresh fulfilment. Their scenes are tough and cold and hard, and we know nothing of K other than what occurs in that room. "I had no idea how to do it," says Bell. "I had no idea what this guy's life was. There's no lead-up, no backstory, there's no emotionality to him. It felt very scary to me, but I wanted to be a part of it, and I felt that the challenge of not knowing how to do it was part of the fun."

Knowing nothing of S&M culture, Bell hung out in a friend's LA sex shop, getting a feel for the clientele, familiarising himself with the world and ridding it of taboo. "I went to his store, which is actually rather nice, not dingy, and I asked him questions like: 'What do they buy, what are they looking for, what are the characters like that come in here?' And there's not one person that looks like they might be into a certain kind of thing. I discovered that I don't have to try to be somebody different, I don't have to get into a different look. This is just a thing that people are into. The pursuit of some weird subculture, that doesn't exist. It is within our culture."

Bell had "lots of help" on the set from bondage, domination, sadism and masochism [BDSM] professionals. "They showed me how to use whips and how to tie certain knots," he continues, "because I tie Charlotte Gainsbourg up a lot in the film. I had to feel comfortable with Charlotte – or not feel comfortable with her, because sometimes I didn't."

'It's a weird experience. But it's part of the character, and you somehow manage to justify in your head that it's what you're supposed to do. But he did ask me to do things that I said I wouldn't do'

That, it seems, is something of an understatement. He knew he'd be out of his comfort zone, but was nevertheless unprepared for Von Trier's working methods. Also, he'd arrived halfway through the film, and felt a little alienated. "It was intense, it was scary," he says. "I felt out of my depth a lot, I felt like I wasn't doing a good job."

Why? "Because his process is just so different to what I've been used to. With Lars, you literally get to the set and it's: 'Action!' And I lose my mind. Because I have no idea what I'm doing. For the last few years I've been working with very conventional film-makers. I'm used to rehearsals, finding beats and positions with directors. Lars doesn't do any of that. He records you not knowing what you're doing, to a degree. I hadn't said hello to Charlotte Gainsbourg before I started hitting her in the face. I didn't even say hello to her. I really didn't, I swear to God, I didn't say a word to Charlotte Gainsbourg the entire time we were making that film. When they said, 'OK everyone, this was Jamie's last day, he's wrapped, that's him done on Nymphomaniac,' and everyone gives a little round of applause, in my very short goodbye speech I said, 'Charlotte, I have no idea who you are but I'm sure you're a lovely person. Maybe I'll see you another time.' I literally hadn't said anything else to her."

It seems impossible that that could have been the case, but Bell explains: "When you're on a set with Lars, you're filming or you're trying to figure out what you're doing. Charlotte's worked with him three times; she's used to it. She did once give me a look of, 'Don't worry, this is just the way it works.' Maybe there was some comfort in that look she gave me. But we did not converse between takes, we had no relationship. Bizarrely, I saw her in Los Angeles a few months ago in a grocery store. And I wanted to say hello to her but I couldn't. I thought, 'The last time I saw you I was pulling your panties down around your ankles, tying you to a couch and smacking your arse. And now you're buying milk.' The change of context is too much for me, I can't handle it. And I walked out without saying hello to her because I felt so embarrassed."

Was filming in such an alien environment psychologically tough? "Umm... I was being asked to do things that weren't particularly comfortable, like drop some coins into a glove then smack her in the face a few times. That is a weird day at work. 'How was your day at work, honey?' 'Well, I smacked Charlotte Gainsbourg in the face a bunch of times.' It's weird. It's a weird experience. But it's part of the character, and you somehow manage to justify in your head that it's what you're supposed to do. Now, he did ask me to do things that I said I wouldn't do. I didn't want to do them."

Like what? "I was asked if I would insert my fingers into a body double. Not Charlotte Gainsbourg, a pornography actress. And I told him I didn't feel comfortable doing that and I'd rather not. Which was totally fine." He didn't mind? "No, absolutely not. His object isn't to frighten people. I never felt frightened by him. He would joke a lot, he's very loving actually. I felt his soul was very warm."

Bell speaks fondly of Von Trier throughout, and despite having felt untethered, he says his Nymphomaniac experience was fulfilling and rewarding. "You always want to challenge yourself and do things that frighten you. It was such a departure in terms of character to me, and as an actor you can only deliver what you believe to be truth. Once I got on the plane home it was, 'I had no idea what I just did. The experience was great, I can't wait to tell my new wife [actor Evan Rachel Wood] what happened.' But I left it behind in Germany."

Bell's since seen the film and is happy. "I think it's a bold, brave, unflinching movie, a great piece of Lars von Trier experimentation and human commentary," he says. "I got everything I wanted from it."

Nymphomaniac Parts I & II are in UK cinemas 22 Feb ahead of a full release in March