Martin Schulz's Social Democrats (SPD) came out ahead with 37 to 37.5 percent of votes in the western state of Lower Saxony, home to scandal-tainted Volkswagen, according to estimates released by public broadcasters ARD and ZDF.
Merkel's CDU party clinched 35 percent, followed by the Greens with 8-8.5 percent, and the liberal and pro-business Free Democrats (FDP) with 7 to 7.5 percent.
The far-right Alternative for Germany (AfD), fresh from winning its first seats in the national parliament, scored 5.5 percent, giving it a presence in 14 of Germany's 16 states.
The SPD victory will come as a relief to former European Parliament chief Schulz, who has gone into opposition after being humiliated in last month's general election and who oversaw three regional election losses this year.
The Lower Saxony vote took on outsized significance as it came just three weeks after a general election that handed Merkel's conservative bloc its worst result in decades, while marking a breakthrough for the far-right AfD.
Merkel still won a fourth term but in order to form a government she must now forge an alliance with the Greens and the FDP, a political poker game that could drag on well into 2018.
Commentators had warned that a setback in Lower Saxony would weaken Merkel's bargaining position as she begins the negotiations in Berlin on Wednesday.
News weekly Der Spiegel said the state poll did nothing to boost Merkel and sent a "bad starting signal" for the talks.
But the defeat was "bearable", it added, considering the CDU's successes in other regional elections this year.
Lower Saxony was forced into a snap vote after the ruling coalition of SPD and the Greens collapsed when it lost its wafer-thin majority due to the defection of a lawmaker to the CDU.
Opinion polls had given the CDU a clear lead early on, but that advantage evaporated in the final stretch.
The SPD's state premier Stephan Weil hailed the poll win as a "roaring success" after a "catch-up race".
Schulz was equally effusive, hailing an "unprecedented election battle" and praising voters for giving the SPD "a tail wind" as the party heads to the opposition benches in Berlin.
It remains to be seen whether the SPD and the Greens secured enough seats to continue governing together in Lower Saxony.
By 1800 GMT, the latest results gave the Greens around 8.5 percent of votes, followed by the FDP at some seven percent.
The CDU's general secretary, Peter Tauber, said all three parties meeting for coalition talks in Berlin had "lost ground" in Lower Saxony, but he downplayed its significance.
"It remains a regional election and we have a clear task in Berlin to build a stable government," he said.
After campaigning for the CDU in Lower Saxony, Merkel must now turn her attention back to cobbling together a coalition with two very different partners - an alliance that is untested at the national level.
If they fail to reach an agreement, Merkel will have to call fresh elections.
Little mood for concessions
The unlikely grouping has been dubbed "Jamaica" because the parties' colours match those of the Caribbean country's flag - black for the conservatives, yellow for the FDP, and green.
In the coming weeks, their leaders will haggle not just about ministerial posts but also red-line policy issues that are sometimes diametrically opposed.
Among flashpoint topics will be questions surrounding the more than one million asylum seekers who arrived in Germany since 2015.
The CDU's Bavarian allies, the CSU, have signalled a tough stance on immigration to win back voters who have drifted to the far-right Alternative for Germany (AfD).
They would find an ally in Christian Lindner, the leader of the FDP, who made a comeback after a dismal previous governing stint in the shadow of Merkel.
Lindner on Friday insisted that there must be a "clear time limit for the residency of war refugees in Germany".
"On war refugees, the aim is not integration in Germany, but their return to their homeland after the end of the war to help rebuild it," he told regional newspaper group Suedwest Presse.
But the Greens are more welcoming towards refugees and also proponents of European "solidarity" - while the FDP has said it will not accept German taxpayers footing the bill for any attempts at shoring up the EU.
The Greens will push issues like fighting climate change and promoting renewable energy, likely to be opposed by the more pro-business parties.
All players are highly reluctant to make major concessions, said Oskar Niedermayer of Berlin's Free University: the FDP because it has previously wilted in Merkel's shadow, the Greens because they face their environmentalist base and the CSU because it must win Bavarian elections next year.
"So, I'm still very, very sceptical," said Niedermayer.
"But of course it is also clear that all sides are under great pressure.
Because the alternatives - a minority government or fresh elections - are something the German people do not want."