Viggo Mortensen is a formidable creative presence in the movies: taking on complex work as an actor with directors such as David Cronenberg and Lisandro Alonso, investing the star capital he earned with his turn as Aragorn in Peter Jackson’s Lord of the Rings trilogy, and doing a workmanlike job as wise-guy driver Tony Lip cured of his picturesque 1960s racism in the egregious Oscar-winner Green Book.

Now he has written and directed his first movie, and it’s a really valuable work, beautifully edited and shot, with a wonderful performance by the veteran actor Lance Henriksen: a sombre, clear-eyed look at the bitter endgame of dementia. Mortensen takes a determined walk across the hot coals of family pain; the drama shows how the condition, with its outbreaks of anger and fear, locks the sufferer into disjointed memories that cannot be expressed or made sense of, a mute ecstasy of loneliness. But it erases other painful memories of wrongdoing that, through a mysterious, sickening quantum, get displaced all too vividly into the minds of the grownup children and carers. They are not even allowed the relief of anger, because dementia behaviour has to be forgiven.

Henriksen plays Willis, an ornery, snowy-haired farmer and widower in the cold expanses of upstate New York: he is a lion in winter, or maybe a junkyard dog in winter. Homophobia is the one of his many attitudes that have now come obsessively to the fore because his son John (played with reticent restraint by Mortensen) has come out as gay. Now the old man has just about accepted that he cannot look after himself any more, and has come to stay with John and John’s husband, Eric (Terry Chen), in their California home. John’s sister, Sarah (a typically strong performance by Laura Linney), stops by with her family for a lunch – which, as Willis yelps and snarls his bigoted insults and sneers, becomes a group martyrdom of tongue-biting silence and subject-changing smiles from the older generation, while the teens are derisive and unafraid.

But a whole past flows beneath this stressed present like an underground stream: that of Willis’s memories – and John and Sarah’s. Sverrir Gudnason plays young Willis: nervy and insecure with a thwarted need for love that curdles into abuse; Hannah Gross plays his sensitive young wife, who cries at LP records of Chopin; and Bracken Burns plays Jill, the woman for whom he leaves Gwen, a wary stepmother to the resentful and bewildered children.

Gudnason shows that Willis was not always a villain: he wanted to bond with his son (though not his daughter) through hunting, and Willis is thrilled that John shows a talent for it, at least at first. But their relationship deteriorates and Willis gives John a scar above the lip which, worryingly, appears to match an ancient scar of his own. A cycle of abuse? Now John has given up drinking, perhaps because it is a pleasure that only fuels his rage at Willis.

With his memories of the farm and its horses and its vision of frontier masculinity, I think Mortensen has probably absorbed the influence of Larry McMurtry. Tellingly, Willis is shown watching Hawks’s Red River, with John Wayne, on TV – and maybe, via McMurtry’s script for The Last Picture Show (about Red River), the McMurtry DNA has indirectly arrived at Mortensen’s work in the present day. There is real passion and tragedy in these vivid flashback memories, triggered by moods, shapes, sounds. Just occasionally, there is black comedy. Cronenberg has a cameo as a proctologist who has to give the ageing Willis a rectal exam, the cue for all sorts of bad-taste wisecracks about his son’s sexual identity. “This is strictly routine,” says the doctor. “For you maybe,” snaps Willis, supine in his hospital gown.

With some self-effacement, Mortensen has conceded the performer’s alpha prerogative to Henriksen. It’s the right decision: Henriksen’s Willis, in all his self-defeating cantankerous arrogance, is so commanding. But I wondered if Mortensen could or should have shown us more about John, more about what he has gone through to arrive at this strenuously calm, diplomatic unresponsiveness. Could he have broken out more, shown more anger? Either way, this is a very substantial achievement.

• Falling is released in cinemas on 4 December.