The map, thought to be worth several hundred thousands euros, was lost after apparently being misfiled in Munich's Ludwig Maximilian University in the 19th century.
"We've made a sensational find," commented library director Klaus-Rainer Brintzinger.
Two researchers found the map between two 16th century prints on geometry "in an otherwise unremarkable volume that had been rebound in the 19th century," a statement said.
It is a slightly younger and smaller map than Waldseemüller's famous three-square-metre world map of 1507, sometimes referred to as “America's birth certificate”, as it was the first map to give the continent that name.
Chancellor Angela Merkel presented that map to the US on an official visit to Washington in 2007, when the map was exactly 500 years old. It has been registered as a world heritage document by UNESCO.
Until the new find in Munich, only four examples of the smaller version were known, one of which sold at auction for $1 million in 2005.
The newly unearthed map, one of the so-called globe segments, is believed to have been produced by Waldseemüller himself, who died in 1522.
The smaller versions were "at least as important for the dissemination of geographical knowledge in his time," as the three-square-metre map, the university said.
The new find shows the world divided into twelve segments which taper to a point at each end and are printed on a single sheet, which would form a small globe if cut out and assembled, with the three rightmost segments depicting a boomerang-shaped territory named America.
"Even in our digital age the originals have lost none of their significance and unique fascination," said Brintzinger in a statement. "We intend to make the map accessible to the public in digital form in time for the Fourth of July, Independence Day in the USA."
The map survived World War II despite air raids devastating the library - many of the older books were moved out in 1942.
Waldseemüller probably used information gleaned from the transatlantic voyages of Christopher Columbus, Amerigo Vespucci - after whom America is named - and unknown Portuguese and Spanish sources, to draw the 1507 map.
Historians still wonder why he named the new continent after Vespucci and not Columbus, who “discovered” America in 1492.