Merkel's Christian Democrats (CDU) scored 30.6 percent, according to ARD public television, with her junior partners at the national level, the Free Democrats (FDP), winning 8.3 percent - not enough to retain power in the northern state.
However, the opposition - combining the centre-left Social Democrats and ecologist Greens - also failed to gain sufficient support to form a government, with 29.9 percent and 13.6 percent respectively.
This left as a strong possibility a so-called "grand coalition" between the CDU and SPD, which many believe could be the final result of the national elections due in September or October 2013.
The big winners on the night were the Pirates, an upstart party that has shaken up the staid world of German politics with a campaign based on more transparency in the political process and internet freedom.
For the third consecutive regional election, they breached the five-percent mark needed to enter the state parliament, winning 8.2 percent of the vote.
But for the FDP, although they lost more than six percent compared to the last election in 2009, it was a better-than-expected result, given that they are polling nationally at around three percent.
Turnout was low, with around 60 percent of the 2.2 million registered voters casting their ballot, compared to more than two-thirds in 2009.
The socialist Left party failed to clear the five-percent hurdle, scoring around 2.4 percent. A party representing the state's small Danish minority also fell below the threshold, with 4.5 percent.
The parties will now engage in days of horse-trading before the final make-up of the state parliament is determined.
However, the election will have little impact on the make-up of the Bundesrat, the upper house of parliament where Germany's 16 states are represented, and Merkel's personal popularity remains high.
The vote in Schleswig-Holstein is seen as a dress rehearsal for a much more significant election in the bellwether state of North Rhine-Westphalia (NRW) on May 13.
As Germany's most populous state with 18 million people and a major industrial base, NRW's centre-left minority government fell after just 22 months over a budget dispute.
The state historically plays a big role in federal politics - in 2005, a lost vote in NRW prompted then Chancellor Gerhard Schröder to call a snap federal election which he then lost to Merkel.
A poor result for the FDP there would be seen as possibly destabilising the ruling coalition nationally, around 16 months before the federal poll.
But their performance in Schleswig-Holstein will at least take some of the pressure off their leader Philipp Rösler, who is also economy minister.
"It's a turning point. We were not dead," said the FDP's parliamentary chief, Rainer Brüderle.