Georg Ratzinger, older brother of former Pope Benedict XVI, led the famous millennium-old choir in the southern city of Regensburg between 1964 and 1994.
He was "impulsive, even fanatical," a "merciless" teacher when "imposing his idea of musical discipline," the current choirmaster, Roland Büchner, told German weekly Die Zeit.
"Afterwards, he could be the gentlest person in the world. Some pupils saw him as a model, others feared him as someone who would beat them," he added.
Ratzinger, now aged 93, has said he knew nothing about the violence at the school.
But Büchner, 63, recalled that under Ratzinger's baton, "a system of fear ruled" among members of the choir.
"It wasn't 'just' about slaps, but genuine physical abuse. People raged, there were injuries," he said.
A report published on Tuesday detailed some 547 cases of abuse at the Regensburg school, site of one of the biggest such scandals to have struck the Catholic church in Germany.
The actual number of cases could be higher than 700, report author Ulrich Weber said.
From 1945 until the early 1990s, choirboys were subjected to abuse ranging from deprivation of food to physical blows and sexual assault including rape.
Most of the cases described in the report fall under the statute of limitations, meaning the 49 alleged perpetrators will not be prosecuted.
As well as Ratzinger, former Regensburg bishop Cardinal Gerhard Ludwig Müller is implicated indirectly in the report.
Müller, who Pope Francis recently removed from a powerful Vatican office as head of the Congregation of the Doctrine of the Faith, faces criticism dating back to his term as bishop of Regensburg from 2002 onwards.
Müller is accused of botching an internal investigation into the abuse in 2010 and failing to engage with the victims.