The four-time winners came to Russia as one of the favourites but became the fourth side this century to lose in the group stage after winning the previous tournament following a 2-0 defeat against unfancied South Korea.
Sweden topped the group after a 3-0 win against Mexico, who join them in the next round despite the heavy defeat.
Germany came into the match against South Korea knowing a win by two clear goals would guarantee them a place in the last 16 but repeatedly failed to create clearcut chances in Kazan.
Coach Joachim Löw's men wasted a series of chances in a tense match that saw a pale version of the world champions sent packing at the first hurdle.
Joachim Löw. Photo: DPA
Germany's team of superstars could only look on in shock in the final minutes of the match as US referee Mark Geiger called for the video assistant referee to intervene after the unmarked Younggwon Kim had bundled his shot past German keeper Manuel Neuer from a corner early in stoppage time.
The goal was awarded because Toni Kroos' touch had played him onside, and minutes later Germany were 2-0 down after Neuer ran up the pitch in a desperate attempt to get a goal.
A long punt upfield saw Tottenham forward Son Heungmin chase a clearance to fire into an empty net.
Löw caused a surprise by leaving midfield attacking stalwart Thomas Müller on the bench for the first time since 2012 after his underwhelming displays against Mexico and Sweden.
But by the end of a frustrating opening half, Löw was left desperately studying his options.
Germany suffered the first of several scares after South Korea won a free kick on the quarter hour when Sami Khedira's boot connected with the head of Jung Wooyoung.
From 30 yards out, it should have been fodder for World Cup-winning goalie Neuer, but after fumbling Jung's dipping drive Neuer scrambled desperately to clear as Son raced to pounce.
Germany spurned a series of chances soon afterwards.
Marco Reus, who hit a crucial leveller in the 2-1 win over Sweden, saw his drive deflected and Mesut Ozil saw his effort deflected off a Korean boot and out for a corner, after which goalkeeper Hyeonwoo Jo smothered desperately when Goretzka's header found Hummels trying to pounce on the loose ball.
Germany resumed with more urgency after the interval, but despite edging closer when Löw introduced Mario Gomez just before the hour the big Stuttgart striker headed straight at Jo from Kimmich's cross.
Striker Timo Werner then met another Kimmich ball from the left byeline with a right-foot volley that skewed agonizingly wide of Jo's upright.
Timo Werner attempting a goal. Photo: DPA
Die Mannschaft's urgency was not matched by their normal efficiency and soon legs started to tire.
Löw played his final card when he introduced Müller for Leon Goretzka.
But the late changes failed to spark Germany into life as Korea hung on for a famous win that, unfortunately for the impressive Asians, saw them bow out after Sweden beat Mexico 3-0.
Defender Mats Hummels admitted Germany's group stage exit was a “bitter” shock.
“It is a really, really bitter evening for us,” Hummels told German broadcaster ZDF. “The situation is very difficult to put into words.
“We did not put the ball in the net, even though we had plenty of opportunities, including myself in the 86th minute — I have to score from chances like that.”
“We had plenty of opportunities, we broke our necks trying out there today,” he added.
Germany qualified for Russia with a perfect record of 10 wins, but Hummels said the last time the team had played well was when they wrapped up qualification late last year.
“We put ourselves in this situation after the Mexican game,” he admitted after the Germans lost their opener 1-0 to Mexico. The last game we delivered a good performance was in the autumn of 2017.”
Sweden coach Janne Andersson spoke of his pride after his side topped the group with second-half goals from Ludwig Augustinsson, Andreas Granqvist and an own goal from Edson Alvarez.
“We've done a fantastic job today with this match,” he said in Yekaterinburg, ahead of a last-16 clash against either Brazil, Switzerland or Serbia.
“I'm so incredibly proud, almost moved as the thought of how we performed on the pitch, so disciplined, I think we've grown since our last match.”
Angela Merkel's spokesman was quick to respond to the bitter result, tweeting “not our World Cup – how sad. There will be other competitions that we will celebrate at.”
Elsewhere, as the first round draws to a close, five-time winners Brazil need a point from their final Group E game against Serbia to guarantee progress.
Victory will confirm Brazil as group winners, but coach Tite is warning that under-fire superstar Neymar cannot be expected to win the World Cup on his own and is not yet at full fitness.
The world's most expensive player, who battled back from a broken bone in his foot to be fit for Russia, took plenty of flak despite scoring in the 2-0 victory over Costa Rica on Friday.
Both Brazil goals came in added time and Neymar wept after netting their second, but was earlier accused of diving after the video assistant referee overturned a penalty awarded to him when replays showed there was minimal contact.
“There is an excessive responsibility on him in terms of success and that is not the way to go,” said Tite.
“Maybe we need one more match and he will be in his full form physically and technically, because he is outside the normal standards.”
Should Neymar and Brazil stumble in Moscow, Switzerland are hoping a win over Costa Rica in Nizhny Novgorod could catapult them into the last 16 as Group E winners.
The Swiss are second in the table and level on four points with Brazil, but have an inferior goal difference.
“I think we should get this done, we want to top the group,” said Swiss head coach Vladimir Petkovic.
The Swiss camp is promising no more controversy after FIFA fined three players for their “double eagle” hand gestures to represent the Albanian flag in the politically charged victory over Serbia.