Merkel's conservatives won 34.5 percent, according to exit polls by public broadcaster ARD, which would be their worst result ever in the state. The CDU's coalition allies the pro-business Free Democrats (FDP) won 6.8 percent, leaving them well short of a majority.
The centre-left Social Democrats (SPD) garnered 34.5 percent, the environmentalist Greens took a record 12.1 percent, and the socialist Left party likely managed to get its first seats in the state legislature with 5.6 percent.
“This election evening is bitter for both CDU in North Rhine-Westphalia and me personally,” said the state's conservative premier Jürgen Rüttgers, who hinted he might try to pursue a coalition with the SPD or Greens in the coming days.
“We'll have to wait and see who's ahead at the end of the night. North Rhine-Westphalia needs a stable government and that cannot happen with extremist parties,” he said, referring to the possibility the SPD could link up with the Greens and the hard-line socialists from The Left.
The head of the SPD in the state, Hannelore Kraft, hopes to succeed Rüttgers as premier, but it was not immediately clear if her preferred coalition with the Greens would be able to eke out a majority.
“I'll think we'll come out ahead in the end and it will be enough for red-green,” Kraft said, referring to the two parties' colours.
The CDU plunged in the polls since the last election in 2005, when they scored 44.8 percent. But the SPD, which governed the state for the previous four decades, has also lost support, having scored 37.1 in 2005.
The result, if confirmed, will make it considerably harder for Merkel to push key reforms through the Bundesrat, the upper house of parliament that represents Germany's 16 federal states. It could stymie efforts including sweeping tax reforms she believes are needed to revitalise Europe's top economy.
The timing of the election could hardly have been worse for Rüttgers' Christian Democrats, who have governed the state with the FDP since 2005. Most Germans oppose the €22.4 billion ($28.6 billion) in loans Berlin agreed to offer debt-wracked Greece only two days before the state election.
A poll published Saturday showed that 21 percent of NRW voters said the Greek bailout would affect their ballot decision, according to a YouGov survey for the daily Bild.
The outgoing government in North Rhine-Westphalia (NRW) mirrored the same centre-right coalition Merkel has in Berlin, making the poll a damaging referendum on her government eight months after she won re-election.
FDP leader Guido Westerwelle, who is also Merkel's vice-chancellor and foreign minister, called the poll result a "warning shot" by voters that would have clear consequences at the national level.
"Citizens need to know that we have heard their message. We must redouble our efforts to win back their lost confidence in our work," he said.
The state is also home to the Ruhr rustbelt region, which has seen economic misery deepen during last year's deep recession.
Underlining the election's importance, Merkel scheduled 15 personal appearances in NRW and staged a media blitz this week to defend the aid to Greece. But the chancellor has also faced criticism in Germany and abroad of dragging her feet over the loans for Athens and thereby exacerbating the crisis.
Beyond control of the state legislature, the dominance of Merkel's coalition in the Bundesrat upper house is now history.
Currently, the conservatives and the FDP hold 37 of the 69 seats in the Bundesrat, just over the 35 votes needed for an absolute majority. Losing North Rhine-Westphalia deprives the centre-right of six seats.
That will effectively axe a drive by the FDP to cut income taxes by €16 billion from 2012 - a move many conservatives, and many voters, oppose as fiscally irresponsible in light of Germany's parlous public finances.
It will also give the centre-left the power to block health care reforms planned by the coalition, and to restore an initiative to mothball the country's nuclear reactors against the wishes of the Merkel government.
The vote's impact will be long-lasting as well as it is the only state election planned this year.
"Scarcely one of the big projects that the conservatives and the FDP promised seven months ago in their coalition agreement will - if the polls are correct - survive this Sunday," the Tagesspiegel am Sonntag newspaper said.