The SPD won around 50 percent of the vote in the northern German city-state, compared with just 20 percent for Merkel's Christian Democrats (CDU), exit polls on public television showed.
"That's a very, very impressive result," said the SPD's mayoral candidate Olaf Scholz. "We will stick to our promises made before the election."
In the last election in the port city in 2008, the centre-right CDU's share of the vote was 42.6 percent, more than double Sunday's score, while the SPD was on just 34.1 percent.
"This is a painful day for the CDU in Hamburg," outgoing Mayor Christoph Ahlhaus said. "The likely result is a bitter defeat."
The early state election became necessary after the Greens abandoned an unusual coalition with the Christian Democrats after the bland Ahlhaus took over from popular former Mayor Ole von Beust last summer.
The result will return the SPD to power in a city that - apart for the last 10 years of CDU rule - has traditionally been a Social Democratic bastion.
Once the SPD has formally formed a government, the opposition at the federal level will have more leverage to block Merkel's legislative agenda in the upper house of parliament, the Bundesrat.
"This was a historical result, not just for us, but also for others," a triumphant SPD Chairman Sigmar Gabriel said in Berlin. "It shows what the Social Democrats are capable of."
Local issues, however, featured heavily in the Hamburg campaign and surveys show the SPD's lead over the CDU in Hamburg is not replicated nationwide, where the conservatives remain the most popular party.
The election is the first of seven state polls in Germany this year, with the most important in Baden-Württemberg on March 27, where the opposition hopes to unseat the CDU after 58 years in power.