The result sparked calls from victorious conservative CDU-CSU bloc for the centre-left Scholz to change course or pave the way for new elections.
The opposition is ramping up the pressure just months ahead of key regional elections in several eastern states where the far-right is projected to romp to victory.
Scholz's Social Democrats (SPD) scored its worst result in history at 14 percent, third behind the far-right Alternative for Germany (AfD) on around 16 percent, and well behind the conservative CDU-CSU bloc's 30 percent.
The Social Democrats' coalition partners Greens recorded 12 percent while the liberal FDP took five percent.
"This is a very bitter result for us," said Kevin Kühnert, general secretary of the SPD, adding that his party will have to examine what went wrong in its mobilisation.
The opposition conservatives argued it was time for the coalition to rethink its policies.
"The government must really reflect on this result... a political change is needed in Germany," said CDU chief Friedrich Merz.
"Things can't go on as they have in the last two and a half years," he added.
At the European level, preliminary projections from the parliament showed the three main centrist parties were set to maintain a clear majority, albeit slightly reduced, with 401 out of 720 seats up for grabs.
The centre-right European People's Party (EPP) came first with 186, followed by its two main partners - the centre-left Socialists and Democrats on 133 and the centrist Renew Europe on 82.
However, the far-right across the EU surged - and even triggered a political earthquake in France where President Emmanuel Macron has called a a snap election.
READ MORE: European election results - Centre holds but far-right makes gains
A fresh start
CDU general secretary Carsten Linnemann said Scholz should call for a "vote of confidence" following the dismal result.
The coalition must either change course "or clear the path for new elections," said Linnemann.
Voters have sent a clear message to the coalition to "drastically change (its) politics -- on migration, on the economy and on climate protection," wrote Germany's best-selling daily Bild.
"We call that a fresh start. But because the (coalition) probably lacks strength and insight to make such a fresh start, the option would be early elections," it predicted.
Scholz's three-way government has been lurching from crisis to crisis since it took power in December 2021, after his SPD defeated former chancellor Angela Merkel's conservatives.
Just months into the job, it has had to contend with Russia's invasion of Ukraine and an ensuing energy crisis that plunged Germany into recession.
While the economy is now eking out better news, recovery is still fragile.
Voter discontent remains high over issues from climate laws to spending cuts, while crime and immigration are growing concerns for the public.
The Greens in particular took a hit over plans to switch Germany over to renewables that require heavy outlays in investments.
Scholz unpopular
Public opinion also remains split over Scholz's coalition handling of the Ukraine war, with some firmly behind its decision to provide armaments for Kyiv, while others fear the backing would drag Germany into war.
A new party BSW led by former Die Linke politician Sahra Wagenknecht, which campaigned to halt weapons deliveries to Ukraine from the West, garnered a stunning 5.7 percent, further splintering the vote, which had a turnout of around 65 percent, up from 61.4 percent in 2019.
Scholz himself is also deeply unpopular, with just 37 percent of approval ratings.
The conservatives and the social democrats have over the decades taken turns to lead the polls in each election cycle.
But the AfD's strong showing could be a cause for concern for the mainstream parties, particularly as it came despite a series of scandals surrounding the far right in the run-up to the polls.
The AfD's result marks a gain from its 2019 showing, when it garnered 11 percent. It came even though the party had to ban its leading candidate Maximilian Krah from campaigning because he is being investigated for suspicious links to Russia and China.
Comments made by Krah minimising the crimes of the Nazis' notorious SS had also prompted the AfD's expulsion from the far-right group within the European Parliament.
Co-leader of the AfD Alice Weidel said the results showed that "people have become more critical of Europe".
Fellow co-chief Tino Chrupalla told AFP "the people ... see that we need an alternative in this country".
Comments