The winning goal came in the 108th minute when Japan captain Homare Sawa put in a superb long-range pass and substitute Karina Maruyama drilled her shot past Germany goalkeeper Nadine Angerer.
Having won both the 2003 and 2007 editions, this is the first time Germany have lost at a women's World Cup since they were beaten 3-2 by hosts the USA in the quarter-finals of the 1999 tournament.
Japan have now got beyond the quarter-finals for the first time, as their previous best performance was a last-eight appearance in 1995 in Sweden.
Japan will now face either Australia or Sweden, who meet on Sunday, in the semi-final in Frankfurt on Wednesday.
Having won Group A with three victories, Germany were bidding for a third consecutive title, while Japan came in as Group B runners-up after losing their final pool game to England.
Germany suffered an early setback when key midfielder Kim Kulig headed over the bar and limped off with a knee injury after an awkward landing on three minutes.
Coach Silvia Neid was forced into bringing on defender Bianca Schmidt and left-back Linda Bresonik pushed up into midfield.
It was predominantly German one-way traffic in the opening stages in front of a sold-out crowd of 26,067.
Powerful midfielder Celia Okoyino da Mbabi, who ran herself ragged over the course of the night, had a shot parried away by Japan's goalkeeper Ayumi Kaihori on 13 minutes.
Captain Kerstin Garefrekes saw her effort fly across the face of the goal on 22 minutes.
Japan finally came into the match, and their clearest chance was when forward Yuki Nagasato fired wide with half an hour gone as the Germans dropped off their early pace.
Mexico referee Quetzalli Alvarado finally showed the first of three second-half yellow cards in the 55th minute as the Japanese produced some loose tackles
Japan midfielder Aya Miyama fired over twice with two long-range efforts as normal time faded and Germany forward Inka Grings had an appeal for a penalty turned down in the final five minutes after she was shouldered off the ball.
After 10 minutes of extra time, Grings found the ball at her feet in the area but fired wide to make way for attacking dynamo Alexandra Popp.
The game was finally decided in the second period of extra time when Japan stunned the home crowd with the crucial goal to reach the last four.