Merkel's Christian Democrats (CDU) won more than 39 percent of the vote in the eastern state Saxony, according to preliminary results, meaning it will need to find a new coalition partner.
The Alternative for Germany (AfD) party, which backs the dissolution of the euro, won around 9.6 percent, according to the initial results.
The result, if confirmed, would see AfD exceeding even the high expectations that had been placed on it ahead of the election. Pollsters had predicted a result of around 7 percent for the party.
The CDU's current coalition partners in Saxony, the pro-business Free Democrats, failed to pass the five-percent threshold to re-enter the parliament. The far-left Linke party, which has roots in East German communism, won around 19 percent.
Merkel's partners in the national "grand coalition" government, the centre-left Social Democrats, won around 12.5 percent, the preliminary results showed.
The Greens won more than five percent, and the far-right, anti-immigrant National Democratic Party of Germany also just scraped back into the Saxony parliament, one of only two in which it has members.
The AfD, which leaped into the European Parliament in May but narrowly missed out on entering the German parliament in last September's general elections, was set up by economics professor Bernd Lucke, a former CDU member, early last year.
It also wants an end to EU bailouts and for Germany to return to its once beloved Deutschmark.
"It's time for a new party in Germany," Lucke said on ZDF television as the first results were coming in.