The Act of Killing won the documentary prize at the Baftas last week and is the favourite to win the much-coveted Oscar. I watch many documentaries on behalf of the BBC each year and I go to festivals. I'm a doc obsessive. By my own, not quite reliable reckoning, I've been asked by fans to show The Act of Killing on the BBC at least five times. I've never encountered a film greeted by such extreme responses – both those who say it is among the best films and those who tell me how much they hate it. Much about the film puzzles me. I am still surprised by the fact that so many critics listed it among their favourite films of last year.

For those who haven't seen the film, it investigates the circumstances in which half-a-million Indonesian leftists were murdered in the 1960s, at the instigation of a government that is still in power. You might think this is a recondite subject, worthy of a late-night screening for insomniacs or atrocity buffs on BBC4, but, no, the film-maker Joshua Oppenheimer has made the subject viewable by enlisting the participation of some of the murderers. He spent some years hanging out with them, to his credit luring them into confessions. But he also, more dubiously, enlisted their help in restaging their killings. Although one of them, the grandfatherly Anwar, shows mild symptoms of distress towards the end of the film, they live in a state of impunity and it is thus, coddled and celebrated in their old age, that we revisit them.

So let me be as upfront as I can. I dislike the aesthetic or moral premise of The Act of Killing. I find myself deeply opposed to the film. Getting killers to script and restage their murders for the benefit of a cinema or television audience seems a bad idea for a number of reasons. I find the scenes where the killers are encouraged to retell their exploits, often with lip-smacking expressions of satisfaction, upsetting not because they reveal so much, as many allege, but because they tell us so little of importance. Of course murderers, flattered in their impunity, will behave vilely. Of course they will reliably supply enlightened folk with a degraded vision of humanity. But, sorry, I don't feel we want to be doing this. It feels wrong and it certainly looks wrong to me. Something has gone missing here. How badly do we want to hear from these people, after all? Wouldn't it be better if we were told something about the individuals whose lives they took?

I'd feel the same if film-makers had gone to rural Argentina in the 1950s, rounding up a bunch of ageing Nazis and getting them to make a film entitled "We Love Killing Jews". Think of other half-covered-up atrocities – in Bosnia, Rwanda, South Africa, Israel, any place you like with secrets – and imagine similar films had been made. Consider your response – and now consider whether such goings-on in Indonesia are not acceptable merely because the place is so far away, and so little known or talked about that the cruelty of such an act can pass uncriticised.

The film does not in any recognisable sense enhance our knowledge of the 1960s Indonesian killings, and its real merits – the curiosity when it comes to uncovering the Indonesian cult of anticommunism capable of masking atrocity, and the good and shocking scenes with characters from the Indonesian elite, still whitewashing the past – are obscured by tasteless devices. At the risk of being labelled a contemporary prude or dismissed as a stuffy upholder of middle-class taste, I feel that no one should be asked to sit through repeated demonstrations of the art of garrotting. Instead of an investigation, or indeed a genuine recreation, we've ended somewhere else – in a high-minded snuff movie.

What I like most about documentary film is that anything can be made to work, given a chance. You can mix up fact and fiction, past and present. You can add to cold objectivity a degree of empathy. You will, of course, lie to reluctant or recalcitrant participants, in particular when they wish not to divulge important pieces of information. And trickery has its place, too. But documentary films have emerged from the not inconsiderable belief that it's good to be literal as well as truthful. In a makeshift, fallible way, they tell us what the world is really like. Documentaries are the art of the journeyman. They can be undone by too much ambition. Too much ingenious construction and they cease to represent the world, becoming reflected images of their own excessively stated pretensions.

In his bizarrely eulogistic piece defending The Act of Killing (of which he is an executive producer), Errol Morris, the documentary maker, compares the film to Hamlet's inspired use of theatre to reveal dirty deeds at the court of Denmark. But Hamlet doesn't really believe that theatrical gestures can stand in for reality. Nor, we must assume, did his creator. A more apt analogy than Morris's might come from Shakespeare's darkest play, Macbeth. What would we think if Macbeth and his scheming wife were written out of the action, replaced by those low-level thugs paid to do bad business on their behalf? We might conclude that putting them centre stage, in the style of The Act of Killing, was indeed perverse and we'd be right.

There are still half-forgotten, heavily whitewashed atrocities from the last century, such as the Bengali famine allowed to occur during the second world war through the culpably racist inattention of British officials; the never wholly cleared-up question of Franco's mass killings; or the death of so many millions in the 1950s as a consequence of Mao's catastrophic utopianism. Those wondering how to record such events will no doubt watch The Act of Killing, but I hope they will also look at less hyped, more modestly conceived depictions of mass murder. In Enemies of the People (2010), the Cambodian journalist Thet Sambath goes after the murderers of the Khmer Rouge. He finds Pol Pot's sidekick, but it is the earnest, touching quest of Sambath himself that lingers in the mind, rather than the empty encounters with evil-doers. Atrocity is both banal and ultimately impossible to comprehend.

Writing in 1944, Arthur Koestler was among the first to gain knowledge of the slaughter of eastern European Jews and he estimated that the effect of such revelations was strictly limited, lasting only minutes or days and swiftly overcome by indifference. Koestler suggested that there was only one way we could respond to the double atrocity of mass murder and contemporary indifference and that was by screaming.

I'm grateful to The Act of Killing not because it's a good film, or because it deserves to win its Oscar (I don't think it does), but because it reminds me of the truth of Koestler's observation. What's not to scream about?

Nick Fraser is editor of the BBC's Storyville documentary series