Japan captain Homare Sawa, who won the tournament's golden boot, kept her side in the game with a header in the 117th minute after US striker Abby Wambach's had put her side ahead early in extra-time after the two sides were locked at 1-1 over 90 minutes.
"I cannot believe this," said 32-year-old Sawa, who was playing at her fifth World Cup.
"We never gave up and played until the last minute, I just kept running until the last moment. We have given all we had."
In the dramatic shoot-out, Japan goalkeeper Ayumi Kaihori saved two penalties before defender Saki Kumagai hit the winning spot-kick.
The US had taken the lead in the second-half of normal time thanks to striker Alex Morgan's sublime strike, but Japan's Aya Miyama poached the equaliser after a mix-up in the American defence in the 81st minute.
This was Japan's first win over the Americans at the 26th time of trying and was heartbreak for the Stars and Stripes as they missed out on a third World Cup title despite dominating Japan for large periods of the game.
"We made just a couple of mistakes. We let them back in the game but I still think we did well," said USA coach Pia Sundhage.
"We had a lot of possession and played some good soccer. It's hard to lose on penalties, some go in and some go out. It's a small difference between winning and losing. We just didn't take our chances."
German Chancellor Angela Merkel celebrated her 57th birthday by joining the 48,817 sell-out crowd in Frankfurt, while Dr Jill Biden, wife of US Vice President Joe Biden, and Chelsea Clinton, daughter of US Secretary of State Hillary, were also in the stands.
Japan made it to final after their shock 1-0 defeat of Germany in the quarter-finals before they dominated Sweden 3-1 in the last four.
The USA followed up a dramatic 5-3 win on penalties over Brazil in the last eight before they beat France 3-1 in the semi-finals.
The USA controlled the opening 20 minutes of the final as left-wing Megan Rapinhoe and forward Lauren Cheney, who replaced striker Amy Rodriguez in the starting team, caused havoc, while Wambach hit the crossbar on 28 minutes.
Japan's first real chance came with 30 minutes gone but midfielder Kozue Ando fired her shot straight at US goalkeeper Hope Solo.
The first-half finished goalless, but Cheney, with a heavily strapped ankle, was replaced at half-time by goal-scorer Morgan.
The USA kept up the pressure and Kaihori palmed Wambach's header over the bar on 64 minutes.
The Americans' pressure finally told when Rapinhoe's long-range pass - which was half the length of the pitch - found Morgan who blasted her shot across Kaihori on 69 minutes.
Japan were gifted the equaliser on 81 minutes when US defenders Rachel Buehler and Krieger scrambled to clear the ball in their own goalmouth, but left-wing Miyama reacted fastest and stabbed the ball home.
In extra-time, Wambach headed the US back into the lead in the 104th minute.
But Sawa equalised from a corner in the 117th minute with her fifth goal of the tournament to take the game to penalties.
The Japanese had defender Azusa Iwashimizu sent off in the 121st minute for a foul on US striker Alex Morgan.
After the devastating earthquake and tsunami which killed tens of thousands in northeast Japan on March 11, coach Norio Sasaki said messages of support from back home had played their part in victory.
"We had lots of support and messages from Japan and that was one of the keys for success," he said.