Interior Minister Hans-Peter Friedrich said the working methods of the federal authorities needed a "fundamental rethink" after it emerged that key case files were shredded.
"What happened, of course, is unacceptable and that is why there must be consequences," Friedrich told public radio a day after he announced that domestic intelligence director Heinz Fromm, 63, would take early retirement.
"I think that we need to take a very critical look at what happened at the Office for the Protection of the Constitution, and I will," he said referring to the domestic intelligence agency.
Fromm had been under fire since November when it emerged that a far-right trio calling itself the National Socialist Underground (NSU) claimed a murder spree with 10 victims, most of them shopkeepers of Turkish origin.
The case broke only when two members of the NSU were found dead, one having shot the other before killing himself and the other, a woman, turned herself in.
Investigators initially assumed criminal elements from the Turkish community were behind the rash of killings.
The intelligence chief finally threw in the towel after an interior ministry official told a parliamentary committee last week that files on informants among neo-Nazis linked with the NSU had been shredded.
Friedrich said that Fromm's successor would be named in the course of a "calm discussion about reforms and changes" at the agency.
Despite his resignation, Fromm is to testify on Thursday before the parliamentary committee probing failings in the investigation of the murders, as will the staff member who ordered the files be destroyed.
The committee's chairman Sebastian Edathy said further resignations were possible and called on the agency to pinpoint what went wrong during the criminal probe.
"The more you try to sweep under the carpet, the greater the likelihood that you trip over the carpet," he said.
The probe will attempt to determine whether undercover officers had direct contact with the NSU and what the agency's destruction of the records was meant to cover up.
German Turkish community leader Kenan Kolat said the intelligence authorities were in a "shambles".
"Our faith in the German security services has dropped to nearly zero," he said, adding that the scandal showed that "the state and certain organs of the state are blind to the extreme right."
Germany was stunned to learn that a neo-Nazi cell of killers was apparently able to operate with impunity for years and that the authorities only stumbled upon the prime suspects when two were already dead.
The Bild newspaper said the shredded files showed that there was a list of 73 names of possible contacts for the security authorities in the far-right scene including the two NSU members later found dead.
The government's appointed representative for the victims of the murders, Barbara John, said the suspicions raised by the file-shredding would be difficult to snuff out.
"Now everything is plausible in terms of possible complicity of the agency. We all have to shudder at the thought," she told the Tagesspiegel newspaper.
Media outlets joined the calls for an overhaul of the security services and the agency's use of informants.
"The loss of public confidence has reached a level which makes corrections imperative," the Tagesspiegel said, calling for greater parliamentary scrutiny.
Bild added, "Fromm's resignation must be the trigger for a fundamental reform of the intelligence service."