"We often have the impression that these people were not just in Syria as sentries or to be trained in the use of weapons, but that they took part in maimings, killings and bomb attacks," German Public Prosecutor General Peter Frank told the Süddeutsche Zeitung.
"We assume that these perpetrators have blood on their hands, but we often can't prove it," Frank added.
It's extremely difficult to obtain evidence from the conflict zones in Syria and Iraq, he added, with no co-operation between the countries' justice systems.
While Germany has strong laws against terrorism, these "investigative problems" prevent them from being applied with full force against suspects, Frank said.
Frank added that prosecutors are currently bringing around 130 terrorism-related cases against 200 individual defendants, most of them alleged members or supporters of foreign terrorist groups.
Police call for more European co-operation
Meanwhile, the German Police Trade Union (DPolG) on Tuesday called for improved co-operation between European countries in fighting terrorism.
"We need to work much, much more closely together – not just police, but also intelligence services," DPolG chairman Ernst Walter told broadcaster ZDF.
Walter said that Germany's Joint Terror Defence Centre in Berlin, which brings together representatives of 40 security agencies from the federal government and Germany's 16 states, could be a model for European co-operation.
"We can't have information being withheld when it comes to organized crime and terrorism," Walter added.
German authorities and police have already called for European nations to share more data about terrorism in the wake of the deadly attacks in Brussels one week ago that killed 35 people and injured around 340 more.