A woman seeks to learn information about her birth parents in China – are they alive? Do they ever think about her? A man wants to confirm that his former pet bearded dragon, which he gave up for adoption, is OK. Another woman, a doctor, has thought for decades about a 10-year-old girl she tried to save on one of her first days of residency, the victim of a drive-by shooting – “how is she?” A young man hopes for guidance in this messy, confusing, exhausting business of being alive.

These are a few of the clients – some hopeful, some skeptical, all yearning for something – who consult psychics in plaintive, open-hearted fashion in Look Into My Eyes, documentarian Lana Wilson’s remarkable new film on a group of mediums in New York. The film opens with a collage of delicately recorded sessions between different psychics and clients, recalling the Showtime series Couples Therapy in its non-judgmental, radical window into people’s innermost insecurities, longing and pain, their nagging questions and uncertainties.

A few pairings seem to reach some type of spiritual insight – one female psychic convincingly tells someone that her late grandmother is in the room, and still disapproves of her husband; a Black psychic tells a Black man, grappling with the knowledge that his ancestors were bought and sold, that his enslaved great-great-great-great grandfather does not want him carrying his burden in the present. Others reach no such helpful conclusions. But all feel, regardless of one’s spiritual convictions, like an act of care, however theatrical. “Sometimes just deep listening and sustained attention is enough,” Wilson told the Guardian. “Sometimes that alone is meaningful.”

Featuring dozens of individual sessions and time with seven mediums, Look Into My Eyes takes a neutral perspective to the oft-denigrated practice of psychics. “You can come into this film a total skeptic, you can be a diehard believer,” said Wilson. It doesn’t matter: “This is about how we as human beings try our best to connect with, witness, and heal each other. And how we all need outside witnesses to better understand ourselves.” Though most of the clients and mediums believe, to an extent, in a spiritual realm and afterlife, the film is first and foremost “an exploration of these very human needs we all have that aren’t supernatural at all”.

Wilson’s work often operates at the intersection of private pain and public performance – her previous films include The Departure, following a Japanese punk-turned-Buddhist monk who counsels people against suicide, and prominent documentaries about Taylor Swift (Miss Americana) and Brooke Shields (Pretty Baby). A “lifelong skeptic” of psychics and religion, she first became interested in what would become Look Into My Eyes eight years ago, after visiting a medium on a whim the morning after the 2016 election. Like many that day, she was “a bit shellshocked”, depressed, terrified, heartbroken. Entering the room for a $5 reading, Wilson felt immediately that she could see her internal state more clearly. By simply laying that state out, she felt comforted. “It wasn’t about believing her or not believing her,” she said. “It was just like, I feel a little soothed now. I think because it was a brief, intimate connection with a stranger, and that’s really rare, and very powerful when it happens.”

Like the seven mediums in the film, all of whom reveal deep, longstanding wells of grief, that psychic told Wilson how much pain she takes on in her sessions. You wouldn’t believe the kind of situations people come in here with, she said. People enter the booth at real crossroads in their lives. “I had trivialized what psychics were and what they did,” said Wilson, but “that cued me into the fact that, you can believe or not believe, but people are reaching out to psychics at moments of real distress.”

Production did not begin until 2020, during the pandemic, when many New Yorkers were experiencing both very real distress and unprecedented acts of kindness from strangers. Wilson, producer Kyle Martin and their team visited more than 150 (self-proclaimed) psychics in all five boroughs and on Zoom. They visited storefront mediums, “a huge part of the geography of New York City, our little neon-lit storefront psychic shops”, and, over time, gravitated toward people doing longer sessions that often braided spiritual practice with psychotherapy. “The seven people in the film are all totally sincere about what they do,” she said, though not without their own doubts, over their ability to commune with something unseen or if there’s anything there at all. Are they really helping people? Are they good at this job? Does it even matter?

Look Into My Eyes includes just as many fumbles – bombs, to use the parlance of improv, which more than one person compares it to – as it does successes. “I was fascinated by the whole range of that experience – when they’re connecting, when they’re not, when something hits the nail on the head, when it’s a total miss, all of it,” said Wilson.

Not coincidentally, most of the professional psychics are also semi-professional or aspiring performers, and drawn to film, TV, theater, books, art. Most acknowledge a performance element to their work, either in presentation or in taking an instinct and leaning into it. But neither the participants nor Wilson see performance at the cost of authenticity. “It’s both performance and authentic,” said Wilson. “I don’t know, honestly, where the line is. I think that we all naturally are slightly different in different situations or in different roles in our lives.” Like documentarians, psychics are curating an emotional experience for another person. “I’m making non-fiction cinema – it is constructed on the one hand, and also real, or at least has this authenticity at the core. And I think that’s not unlike psychic sessions,” said Wilson.

“It’s a fundamental truth about psychics and art in general: they’re both artificial and real,” she added. “But an artificial connection can be just as meaningful as a real one.” You could believe there’s really a ghost of someone’s grandmother in the room, or that the psychic and client are just engaged in a shared performance of remembrance, or even that the act of unpacking one’s emotional burden with someone else is cathartic in itself. Look Into My Eyes supports all interpretations, or none; the connections stand for themselves. “We all tell ourselves stories about things we can’t prove and believe in things we can’t actually see,” said Wilson. “It’s not about proving, or if the belief is ‘real’ or not. It’s about, what does that belief do for you? What does that mean for you?”

As humans, “we have this need to be seen, to connect to each other that’s so deep. When it happens, it feels like magic.”

• Look Into My Eyes is out now in the US with a UK date to be announced