"The inspector general of the Bundeswehr (Germany's armed forces) has ordered an inspection of all of its properties in order to see if any of them contain memorabilia of the Wehrmacht and if so, to remove it," a defence ministry spokesman told AFP.
The Wehrmacht was the army which served the Nazi regime.
The move follows a growing scandal over far-right leanings of some within the German military which has shaken the army and the defence ministry over the past two weeks.
Details emerged in late April following the arrest of a 28-year-old soldier stationed at a Franco-German base near Strasbourg who had expressed far-right extremist views and was plotting an attack disguised as a Syrian refugee.
Following his arrest, Defence Minister Ursula von der Leyen scrapped a trip to the United States and went to his base in Illkirch in northeastern France.
There, officials found Wehrmacht memorabilia openly displayed in the common room without any apparent effort to remove it.
The ministry, which has banned such symbols, then discovered other Wehrmacht items at another base in southwestern Germany.
Press reports also referred to another incident at the Illkirch base in 2012 when German soldiers painted a huge swastika on the floor of the base to provoke their French counterparts ahead of a football match between Bayern Munich and Lille.
A battalion of German combat troops has been stationed there since 2010 as part of the joint Franco-German Brigade.
As the scandal widened, Von der Leyen called her generals to order, demanding they show zero tolerance with any extremist tendencies within the ranks.
The minister, who is close to German Chancellor Angela Merkel, has sharply rapped the armed forces for leadership failures, criticising "a misunderstood esprit de corps" that led superior officers to "look the other way".
"This process of clarification demands courage and tenacity," she told the Bild newspaper on Sunday.
"We must all support it, from the general down to the new recruits because it concerns the reputation of the Bundeswehr."