When the Beat Drops, a new documentary directed by Jamal Sims, chronicles a close-knit circle of black gay men in Atlanta who lead double lives. By day, they are mild-mannered hospital workers, teachers and corporate employees. By night, they rock a furiously energetic, hyper-athletic style of dancing known as “bucking”.

It’s a style of dancing that’s both hidden and mainstream – simultaneously a part of everything from underground clubs to Beyoncé videos and dance routines, including her recent Coachella extravaganza. (Sims, a noted choreographer, has worked with Beyoncé, Madonna and many other stars.) “It’s still very underground,” Sims says of bucking. “Although Beyoncé did it, and artists are doing it, people don’t know what it is, exactly.”

When the Beat Drops is the latest in a long line of dance films depicting the underground, from Paris is Burning’s view of voguing in New York to Rize’s dynamic portrayal of crumping in Los Angeles. “I was inspired by Rize in particular because I’m from Los Angeles, and the Dave LaChapelle movie really did inspire me,” says Sims.

Bucking, also known as “j-setting”, was originally invented in the 1970s by the Jackson State J-Settes, the dance team for the marching band at Jackson State University, a historically black university in Mississippi. “A woman named Shirley Middleton really created the style,” says Sims. “She was a Jackson State University J-Sette. She was the one who put down her baton and started bucking on the field.”

Bucking is an incredibly intense, high-energy dance form, and just watching it on the screen can make your heart rate skyrocket: it looks like the ab workout to end all ab workouts. “Bucking is really from the pelvis,” Sims explains. “It’s the thrusting of the pelvis … the girls, the drill teams, whenever they start to thrust their pelvises forward, that is the basis of bucking. They’ve incorporated jazz or hip-hop, and their hips thrust forward, even if they are standing in place, gyrating.”

The style was originally invented by women, but bucking was soon adopted by gay men in the south, who took what they saw on the college sports field and moved it to the club, dancing to house music and hip-hop. Sims first discovered the phenomenon in 1996 on a visit to Atlanta. “I went to Traxx – that was the big club in Atlanta,” he says. “I was dancing on the dancefloor, and saw these guys come in and they started battling. As a professional choreographer and dancer, I couldn’t believe I didn’t know that this style existed. I thought I pretty much knew what was going on. I decided then that I wanted to know more.”

When the Beat Drops tells the story of bucking through the personal stories of Atlanta dancers and leaders with names like Flash, Napoleon and Anthony, also known as Big Tony. Perhaps the biggest revelation in the movie – which received its California premiere this week at the Frameline film festival in San Francisco – is that the men involved in “bucking” are a real, organic community. You can’t tell from the aggressive, take-no-prisoners way they battle on the dancefloor.

“These guys are like brothers to me,” says Anthony in the film. “We have our fights like anybody else, we go through breakups … we are there for each other, God knows we love each other.”

On the dancefloor, though, where bucking teams battle, it’s all about ruthless competition. When the Beat Drops climaxes with a breathless tournament in Atlanta. The teams are incredibly disciplined – they practice for several hours a day. Each team sports its own uniforms and distinct styles.

“You could almost tell which team it was even if their faces were covered, just by the way they danced,” says Sims. “Some of them are acrobats – they’re extremely acrobatic, even though a lot of them have never trained. They were self-taught. I connected with that; I was a self-taught dancer. I never had a dance class prior to working professionally. When you learn something yourself, sometimes it’s really raw, and the technique might be really challenged or not correct, but for you it works. These guys are doing backflips and tucks and splits and things. That’s just them teaching themselves.”

At any given time, if the music changes, the captain of the team will instantly switch to a new routine. “They will have over a hundred routines in their head of any routine they can do,” marvels Sims. “The guys are trained to know what they are about to do. It’s really intense. You have to remember all these bits of choreography; it’s not just like you go to the club … For a dancer, we rehearse one week for one number. Can you imagine having a hundred routines in your head that you choose from?”

At the end of the day, despite the waves of mainstream interest, bucking still flies below the radar. “I still feel it’s very underground,” Sims says. Even when you go to New York City, it doesn’t exist in the clubs. That’s because it’s really a southern thing.”

• Frameline, the San Francisco International LGBTQ Film Festival, continues until Sunday