The agreement was scuppered on Wednesday after negotiations between the two chambers of the German parliament - the Bundestag and the Bundesrat - broke down in a parliamentary mediation committee.
The Bundestag, controlled by Chancellor Angela Merkel's centre-right coalition, had voted in favour of the bill, but it was blocked by the Bundesrat, where a centre-left alliance of the Social Democratic Party (SPD) and the Greens holds sway.
The opposition threw the deal out, arguing that it was too lenient on tax evaders.
Under the terms of the agreement, which would have come into effect on January 1, Germans who have parked undeclared assets in Swiss bank accounts would have been able to regularise their holdings while retaining their anonymity.
Swiss banks would have deducted taxes from German clients and transferred the tax revenues to Berlin, allowing the clients to remain anonymous. It was meant to negate the need to buy stolen bank account data from clandestine sources, which many German regional tax offices had resorted to over the past few years.
The mediation committee passed a declaration calling on the government to begin new negotiations "for a fair tax agreement."
Bundestag MP Thomas Strobl, of Merkel's Christian Democratic Union (CDU), said it had been "a difficult session."
The SPD and the Green Party cast doubt on government estimates that the deal would bring in up to €10 billion of tax revenue in 2013 alone. The opposition also criticized the fact that German tax evaders would still have had the chance to get their fortunes out of Switzerland by January 1, 2013.
The Swiss legislature had already passed the agreement, and Eveline Widmer-Schlumpf, President of the Swiss Confederation and head of the Finance Department, expressed regret that Germany had not ratified the deal.
She said the failure meant that "the unsatisfactory status quo with chance finds of illegally appropriated CDs" would have to be maintained for now.
Berlin and Bern have been embroiled in a spat over tax since 2010, when German authorities raided branches of Credit Suisse bank in 13 German cities after buying data on suspected tax frauds.
Switzerland reacted angrily, saying the data were stolen in violation of its banking secrecy laws.
As much as €180 billion in German assets are hidden in Switzerland, according to unconfirmed media reports.