The proposed designation for Algeria, Tunisia and Morocco aimed to make it easier for Germany to deny asylum requests from their citizens on the grounds that they were safe in their home countries.
The designation, which presumes a government does not systematically persecute opponents, would also aim to deter citizens of the three Maghreb countries countries from illegally heading to the EU in future.
Germany has taken in one million refugees and migrants since 2015, under a generous policy which Merkel has defended in the teeth of criticism including from new US President Donald Trump who called it a "catastrophic mistake".
But the "safe countries" plan was rejected by the leftist Greens and Linke parties, which had voiced concerns over the rights records of those countries, including on the treatment of homosexuals.
Interior Minister Thomas de Maiziere said the decision marked "a bad day in our efforts to stop illegal immigration", charging that "especially the Greens carry the responsibility".
He said that "today's 'no' can only be explained by party politics" ahead of September elections.
The interior minister of the conservative southern state of Bavaria, Joachim Herrmann, branded the vote "an entirely wrong signal".
"We have to be in a position to quickly and easily reject asylum requests that are not based on political persecution but purely on economic reasons," he was quoted as saying by news agency DPA.
Refugee rights group Pro Asyl welcomed the defeat of the bill which it said would have paved the way for fast-track asylum denials, saying that each case deserves careful consideration and the possibility of an appeals process.