The incident happened on a train travelling west from Vienna when the 65-year-old German man began dishing out racial insults to a woman travelling with her young son after the child's foot had touched the man's bag.
After a man sitting nearby intervened in the incident, the German shouted at him: “I gassed hundreds of tattooed pigs like you in Auschwitz.”
The tattooed man then pulled out his ID showing he is a member of Austria's special operations unit EKO-Cobra, which happens to be responsible for carrying out counter-terrorism operations in the country.
The off-duty Cobra officer seized the man, who according to witnesses continued to shout insults at him before being taken off the train at Linz station and handed over to police to the cheers of the other passengers.
Under Austria's Prohibition Act 1947, it is forbidden - as it is in Germany - to broadcast Nazi sentiments or deny or try to justify Nazi crimes.
Outrage at 'Heil Hitler' sign
In late June, residents in Graz were outraged after a photo was published online of a ‘Heil Hitler' sign pinned to the back of a van driving in the city.
A month earlier, police in Upper Austria arrested a suspected neo-Nazi who "repeatedly told friends that he wanted to 'shoot dead all asylum-seekers with his shotgun' at a refugee centre".
The man used neo-Nazi language and sent a friend two text messages that included "National Socialist ideology," police said.
Latest figures show the country has seen a rise in far-right crime in recent years, with authorities pressing charges in about 1,690 cases related to right-wing extremism in 2015, the highest yearly figure to date.