Ajax striker Arkadiusz Milik gave the hosts the lead on 51 minutes before substitute midfielder Sebastian Mila netted a late second at Warsaw's National Stadium to inflict Germany's first defeat in a qualifier for seven years.
The Republic of Ireland's 7-0 drubbing of minnows Gibraltar in Dublin earlier in the day means Germany are now fourth in Group D with Poland top, Ireland second and Scotland third.
Joachim Loew's Germany have the chance to get their campaign back on track when they host Martin O'Neill's Ireland in Gelsenkirchen on Tuesday, the same night Poland host Scotland.
"We are obviously disappointed, we were too reckless with our chances," admitted Loew.
"Our structure was good and we created a few opportunities, but Poland took their chances. We have to focus on Tuesday now, react well against Ireland and gather our strength."
Poland earned their maiden victory over Germany at the 19th attempt.
"We are delighted, you could see in the stadium that the supporters have been waiting a long time for this victory," said Poland defender Lukasz Piszczek.
"Our counter-attack game worked really well and we took our chances, that was the difference as the Germans didn't use theirs.
"It means a lot for the country, but we need to stay focused and concentrate on our next game on Tuesday."
The Germans had their impressive record of 33 qualifiers without defeat, dating back to a 3-0 defeat to the Czech Republic in October 2007, ended in abrupt fashion.
Their their last away loss in a qualifying match was a 3-0 defeat against Portugal in June 2000.
They stumbled again at Warsaw's National Stadium, the same venue where they lost the Euro 2012 semi-final to Italy.
Germany have struggled since winning the World Cup in Brazil in July.
They were humbled 4-2 by Argentina in an August friendly, then laboured to a 2-1 win over Scotland last month in their opening Euro 2016 qualifier.
"The defeat was avoidable, but we didn't play badly," said Germany's stand-in captain and goalkeeper Manuel Neuer.
"I came out late for the first goal, it was my mistake."
With several stars injured including captain Bastian Schweinsteiger, Sami Khedira and Mesut Ozil, Loew handed Bayer Leverkusen's Karim Bellarabi a start on his debut in a new-look midfield.
Even with retirements and injuries, Loew could still call on six of the side which started the World Cup final.
In Schweinsteiger's absence, Christoph Kramer partnered Toni Kroos in the defensive midfield with Mario Götze played in the central attacking midfielder role with Thomas Mueller as a lone striker.
Having routed Gibraltar 7-0 in their previous qualifier, Poland made a confident start and Germany only enjoyed clear chances to test Poland's Arsenal goalkeeper Wojciech Szczesny towards the end of the first-half.
Bellarabi fired wide when he connected with Götze's cross at the far post on 38 minutes, just before Mueller failed to finish his chance after a great run through the middle of the Polish defence.
The hosts took the lead when Borussia Dortmund's Lukasz Piszczek finished off a flowing attack and floated an inch-perfect pass over the German defence for Milik to put his header past Manuel Neuer.
The world champions raised the tempo in a bid to get back on level terms, but Poland also had their chances as Robert Lewandowski and Milik found space to attack as the Germans pressed forward.
With time running out, Loew swapped Kramer for the more attack-minded Julian Draxler while Lukas Podolski came on for Chelsea star Andre Schürrle on Germany's left wing.
But Mila settled the matter when he converted Lewandowski's pass just before the final whistle to the delight of the home fans.