The grave of Reinhard Heydrich was “dug up in the night between Wednesday and Thursday” and an investigation has been opened on charges of disturbing a burial site, a police spokeswoman told AFP.
German media said it appeared nothing was removed.
Heydrich was the powerful head of Hitler's Reich Security Office, which included the Gestapo.
Less well known than other Nazi leaders, he was nevertheless highly influential and was marked out for his cruelty even within the Third Reich elite.
Adolf Hitler admiringly used to refer to him as “the man with the iron heart”, according to the biography “Heydrich: The Face of Evil” by Mario Dederichs.
Heydrich hosted the Wannsee Conference on January 20th, 1942 when leading Nazis discussed the extermination of the Jews in German-occupied Europe.
During the Nazi occupation of what is now the Czech capital, he became known as “the Butcher of Prague”.
Reinhard Heydrich. Photo: DPA
His car was attacked with an anti-tank mine in the city on May 27, 1942 by Czechoslovak agents trained by Britain's secret Special Operations Executive.
Heydrich died of his injuries a few days later.
His body was brought back to Berlin and buried in the city's Invalidenfriedhof, a military cemetery.
During the Cold War, the cemetery became a no-man's land along the Berlin Wall and his grave – along with the ones of other top Nazis – was dismantled.
But Heydrich's remains were never disinterred and the location of the grave was an open secret.
In 2000, a group of anti-fascists said they had opened up the grave of Nazi stormtrooper Horst Wessel in Berlin, taken his skull and thrown it into the Spree River, according to the Tagesspiegel newspaper.
Police at the time said no remains were stolen.