Exit polls published as voting closed at 1700 GMT showed the CDU scoring a historic low of 27.5 percent in stronghold Baden-Wuerttemberg, behind the Greens.
The anti-migrant AfD meanwhile obtained double-digit scores in all three, garnering as much as 23 percent of the vote in the eastern state of Saxony-Anhalt.
The elections are the biggest since a record influx of refugees to Germany, and are largely billed as a referendum on Merkel's decision to open the country's doors to people fleeing war.
"These elections are very important... as they will serve as a litmus test for the government's disputed policy" on refugees, Duesseldorf University political scientist Jens Walther told AFP ahead of the polls.
Surveys in the run-up to the vote showed that support for the CDU and its junior coalition partner Social Democratic Party (SPD) dropping while the populist Alternative for Germany (AfD) was steadily gaining momentum and expected to record a surge in backing in all three states.
Merkel's Christian Democratic Union (CDU) was bracing for one of its poorest showings in years, particularly in its traditional stronghold of Baden-Wuerttemberg, with a poll published late Thursday by ZDF public television showing support plummeting by 10 percentage points to 29 percent -- putting it for the first time behind the Greens -- while the AfD snatched 11 percent.
Guido Wolf, the CDU's leading candidate in the southwest, has described Sunday's as the "most difficult election campaign" the party has had to run.