The centre-right alliance led by Friedrich Merz topped the polls with 28.5 percent of the vote, and is already eyeing up the SPD as its preferred coalition partner.
Scholz and his allies were still smarting from the result, as senior party figures began positioning themselves for the next government.
"The ball is in Friedrich Merz's court," said SPD co-leader Lars Klingbeil at a party press conference.
Germany needed the SPD to "to be able to act quickly and make decisions," said Klingbeil. "The world is turbulent and decisions do not wait."
Another alliance between the two parties would be the fourth "grand coalition" in the past two decades, with the Social Democrats always playing second fiddle to the CDU/CSU.
But negotiating a coalition with the CDU will be far from easy. The election campaign underlined significant programmatic differences between the SPD and the conservatives.
READ ALSO: What kind of coalition government should German expect after election?
'Big loser'
The SPD came out of the election as the "big loser", Der Spiegel magazine said.
"The Germans are obviously done with Olaf Scholz", who was "seen by many as the face of the political and economic crisis in the country".
At the party's headquarters in Berlin, named for iconic SPD chancellor Willy Brandt, activists watched in stony silence as the results of Sunday's election were announced.
Speaking to his supporters after the vote, Scholz conceded a "bitter" defeat.
"The election result is poor and I bear responsibility," said Scholz, who will hang on in office until his successor can be named.
The time it will take to form the next government will depend on often arduous coalition negotiations, which can drag on for weeks if not months.
For the two would-be partners, distancing themselves from the legacy of former CDU chancellor Angela Merkel, who three times led a grand coalition, would be "in everyone's interest", said Jacob Ross, a fellow at think tank DGAP.
Scholz, who served as vice chancellor and finance minister under Merkel, had failed to craft a different image to his former boss, whose time in office was now viewed "extremely critically" by voters, Ross said.
The Social Democrats had shown some willingness "to improve internal security and reduce the number of asylum applications", two key demands of the conservatives after a string of recent attacks.
But the talks could prove "difficult" if those on the Social Democrats' left flank dig their heels in, Ross said.
READ ALSO: How different groups of Germans voted in the federal election?
'Very different'
"I have personally decided that I will leave the party if the SPD makes Friedrich Merz chancellor," Social Democrat supporter Philipp Bauer, 44, told AFP outside the party HQ on Sunday.
"I can't support that," said Bauer, who works as a lawyer in Berlin.
Going into government with Merkel was one thing, but joining Merz's coalition would be "very different", Bauer said.
Social Democrats within government have however signalled their willingness to continue in office with the opposition leader and presumed next chancellor.
SPD Defence Minister Boris Pistorius, the country's most popular politician, who was touted by some within the party as an alternative to Scholz at the start of the election campaign, said Sunday the party was "ready for talks" and offered to play a "leadership role".
Klingbeil, who acted quickly to claim a key role as the SPD's parliamentary leader after the vote, said it was for the conservatives to "seek dialogue".
The Social Democrats "need programmatic changes, we need organisational changes" before heading back into government, Klingbeil said.
But the final decision could be made in an internal vote of the rank-and-file, he added. "The members of the SPD will decide."
By Clement KASSER
Comments