Germany ran riot as captain Philipp Lahm, Sami Khedira, veteran striker Miroslav Klose and Marco Reus all got on the score-sheet to cancel out Giorgos Samaras' and Dimitris Salpingidis' second-half strikes for the Greeks.
Germany will now play England or Italy, who meet in Sunday's Kiev quarter-final, in the semi-final at Warsaw's National Stadium on Thursday as they inflicted Greece's heaviest defeat at a European championships.
It eclipsed the Greeks' 3-1 defeat at the hands of Czechoslovakia in 1980.
"I would like to congratulate the Germans, they were really good," said Greece coach Fernando Santos.
"They certainly have the game worked out. We should have had a few more chances, but they had more possession and put us under pressure."
"That was a class performance from the team to reach the semi-finals for the fourth time in a row," said Löw with the Germans having reached the semi-finals at both the last two World Cups and lost the Euro 2008 final.
The Germans dominated at Arena Gdansk with 700 first-half passes completed compared to the Greeks 70 and the Germans' ball possession never dropped below
Santos brought in Grigoris Makos for suspended captain Giorgos Karagounis, who scored the winning goal against Russia which put the Euro 2004 winners in the last eight, with midfielder Kostas Katsouranis named as skipper.
Löw pulled off a pre-match triple surprise by leaving out striker Mario Gomez, plus forwards Lukas Podolski and Thomas Müller in a bold move.
"We can be proud of the team, it was clear that the Greeks could make something out of nothing. We failed to take an early lead, but we did not panic. There were a few changes and I thought I had to change things after three wins," said the German coach.
Klose won his 120th cap to come in for Gomez to claim his 64th international goal leaving him just four short of Gerd Müller's all-time Germany record.
Borussia Dortmund-bound Reus capped his first Euro 2012 appearance with a goal at Müller's expense to win only his seventh cap while Bayer Leverkusen's Andre Schürrle took Lukas Podolski's place on the left wing.
The Germans had the ball in the net after only four minutes as Khedira's shot was only parried by Greek goalkeeper Michaelis Sifakis, who was soon beaten, but Schürrle was flagged for offside.
Celtic's Samaras earned the first booking when he ran his boot down the back of Germany's Bastian Schweinsteiger's ankle on 14 minutes while Sotiris Ninis forced Germany's Manuel Neuer into a rare save on 31 minutes.
With the Germans enjoying 70 percent ball possession, the goal they had been threatening came on 39 minutes when Lahm burst through the left side of the Greek defence and fired home from the edge of the area.
It was a carbon copy of the goal Lahm scored against Costa Rica, the first of the 2006 World Cup, and his fifth for his country on his 90th appearance.
Greece coach Fernando Santos added an extra forward after the break with Sotiris Ninis making way for Fanis Gekas and midfielder Giorgos Fotakis on for defender Giorgos Tzavellas.
The switch paid off when Samaras slipped his marker Jerome Boateng to bundle the ball home from the cross of Salpingidis, who outpaced Lahm on the right flank on 55 minutes.
But Germany reclaimed the lead thanks to a volley from Real Madrid's Khedira whose late run connected with Boateng's cross to leave Sifakis with no chance.
Klose added to his impressive tally with a header from a Özil free-kick on 68 minutes, while Reus scored the fourth when Klose latched onto Özil's pass, forcing Sifakis into a save.
But the ball fell to the Dortmund man, who hit the net on 74 minutes. With time almost up Salpingidis scored from the penalty spot after Boateng's foul, but it was too little, too late.
German Chancellor Angela Merkel had the last laugh as she praised her team for their 4-2 win over Greece which brought them a place in the semi-finals of Euro 2012.
Earlier, the German leader had been heckled by some Greek fans when the stadium camera at the Gdansk Arena showed her on the big screen as she sat next to UEFA president Michel Platini.
The heckling came in a highly-charged atmosphere, the match played out against a backdrop of economic crisis in the Eurozone with Germany seen as pressurising cash-strapped Greece to choose a programme of tough austerity to rein in ballooning debts.
"It was a great match," beamed Merkel afterwards as she basked in a win courtesy of goals by Philipp Lahm, Sami Khedira, Miroslav Klose and Marco Reus lifted the team through to a match against either England or Italy.
Merkel admitted that she was nervous at the start but once Germany settled "I had the impression we dominated. It was a great match."
"Congratulations - keep it up," smiled the Chancellor, who raced to Poland from a meeting with top Eurozone economy leaders in Rome.