Klaus-Dieter Weiß, a well-known amateur palaeontologist in the central German region, was taking part in a regular public dig at the famous Messel Pit near Darmstadt when he found the remains.
"We pushed a layer aside and saw it lying there," Weiß told the Frankfurter Rundschau newspaper.
"You can dig around for a year and find nothing, but we had luck again here," he told local radio station FFH.
The pit of oil shale - a former lake which formed in a volcanic crater - has been designated a Unesco world cultural heritage site for the quality of fossils found there from between 57 million and 36 million years ago - the era when mammals were developing rapidly.
The fossil found by Weiß was a Hessian pholidocercus, an insectivorous mammal similar to and said to be related to hedgehogs but with a long bare tail like those found on present-day rats.
This specimen is thought to be only the seventh ever found, and one of the best preserved.
Dr Norbert Micklich, from the Hessen state museum in Darmstadt which has a huge collection of fossils from the Messel Pit, told The Local the find was extraordinary.
"You can see the shapes of the animal's hairs - by the fossilised bacteria left there," he said. "If we can take samples from what looks like the remains of the gut and get them under an electron microscope, we could even be able to work out what it had been eating."
The museum has the holotype of the species - the one from which the scientific classifying description was taken. In some ways this new one is in even better condition. The holotype only has half of its tail, said Micklich. Although the new one has all its tail, a section of its spine was missing and has been carefully reconstructed.
Although it was discovered in 2009, the fossil has only now been presented to the public. Much work was needed to expose as much of the fossil as possible and set it in preserving plastic. But the current reorganisation and renovation of the museum also contributed to the delay.
It is hoped the fossil will go on public display some time this year - but in 2014 at the latest, said Micklich.