The environmentalist Greens and the centre-left Social Democrats (SPD) managed an unprecedented political upset in the wealthy southwestern state, which is normally a stronghold for Merkel's Christian Democratic Union (CDU). Official results showed that the Greens won 24.2 percent while the SPD took 23.1 percent.
The incumbent CDU and their pro-business Free Democratic (FDP) allies managed 44.3 percent between them, with the conservatives winning 39 percent and the FDP 5.3 percent.
Merkel's Christian Democrats have governed Baden-Württemberg since 1953, but anger over her nuclear policy in light of the Japan crisis as well as decisions on Libya and the euro angered voters in the run-up to the poll.
“We've achieved a historic election victory,” said Winfried Kretschmann, who is likely to become Germany's first state premier from the Green party. “I'd like to thank those that voted for us - especially those voting for us for the first time.”
In another state election in neighbouring Rhineland-Palatinate, the Social Democrats looked set to stay in office, but will have to share power with the Greens.
The SPD won 35.7 percent of the vote, the CDU 35.2 percent, and the Greens 15.4 percent. The FDP, which won only 4.2 percent, failed to clear the five-percent hurdle to win seats in the Rhineland-Palatinate state legislature.
The outcome will increase the pressure on Germany's already embattled Foreign Minister Guido Westerwelle, the FDP leader, although analysts said Merkel's centre-right coalition was expected to survive.
“This is a difficult evening for us. We're naturally disappointed by the election results,” Westerwelle said in Berlin. “Energy policy was decisive. It was a referendum about atomic energy and we have gotten the message.”
But beyond a crushing blow to morale in Berlin, the double state defeat will make it even harder for Merkel to pass legislation in the Bundesrat upper house and likely prompt fresh calls for her to shore up her conservative credentials.
But analysts said Merkel's coalition was expected to survive, mainly because the opposition is still too weak at the national level and her party lacks a viable challenger to her position.
"The CDU has no one," news weekly Der Spiegel wrote on its website. "It is chained to Merkel, at least until the next scheduled federal election (in 2013)."
Campaigning was dominated by the nuclear catastrophe in Japan, where officials Sunday discovered high radiation levels in water leaked from a stricken reactor at the Fukushima plant.
Calling Japan's crisis a "turning point," Merkel suspended for three months an earlier decision to extend the lifetime of Germany's nuclear reactors, four of which are based in Baden-Württemberg.
Nuclear power is highly unpopular in Germany, and an estimated 250,000 people took to the streets across the country on Saturday to protest against the government's energy policy.
But the elections confirmed that voters saw Merkel's atomic reversal as an electoral ploy that cost her conservatives support while boosting the anti-nuclear Green party.
Baden-Württemberg's outgoing conservative state premier, Stefan Mappus, also faced massive protests against the unpopular rail project Stuttgart 21.