The 28-year-old suspect, who was not identified, was thought to have a "xenophobic background", they said.
The Die Welt daily reported he may have plotted to pin the blame for an attack on foreigners.
The case involved a joint police operation across Germany, France and Austria with raids on 16 locations, prosecutors in Frankfurt said in a statement.
Police arrested the soldier - a lieutenant usually stationed on a Franco-German military base near Strasbourg - in the southern German city of Hammelburg on Wednesday.
The same day they also arrested a second German man, a 24-year-old student and alleged co-conspirator in possession of flares and other objects that breach weapons and explosives laws.
The lieutenant had been temporarily detained by Austrian police in February at Vienna airport when he tried to retrieve a loaded, unregistered handgun he had hidden in a toilet there a few days earlier.
This sparked an investigation that threw up an even bigger surprise: the suspect had in December 2015 created a false identity as a Syrian refugee.
He led "a double life", said a prosecution spokeswoman about what she called an unprecedented and "extraordinary" case.
He had registered himself at a refugee shelter in the central German state of Hesse and later even launched a request for political asylum in Bavaria state, said the prosecution statement.
The request was accepted, even though the man speaks no Arabic.
He was alloted a place in the refugee home and has since January 2016 received monthly financial payments under this false identity, the prosecutors said.
"These findings, and indications of a xenophobic background of the Bundeswehr soldier, suggest that the accused was planning a serious crime endangering state security with the weapon that was earlier deposited at Vienna airport," said the statement.
Security services in the sweeping cross-border raids targeting contacts of the two men confiscated mobile phones, laptops and written materials, said the statement.
There was evidence the 24-year-old student shared anti-foreigner views, including text messages between the two suspects, who both hailed from Offenbach near Frankfurt.
Green party lawmaker Irene Mihalic called for an investigation into whether "in far-right circles, attacks are being planned specifically in order to blame them on refugees".
The case recalled the bizarre suspected plot behind an April 11 explosives attack against the bus of the Borussia Dortmund football team, which wounded one player and a police officer.
A German-Russian suspect allegedly staged the attack, with a fake Islamist claim of responsibility, in a bid to profit from a resulting plunge in the club's stock value.
Germany has taken in more than one million asylum seekers since 2015, many from Syria, Iraq and Afghanistan, sparking an anti-foreigner backlash and a spate of racist hate crimes.