The three were named by the Attorney General's office as Steven N., 26, Abdullah W., 28, and Abdulsalam W., 23.
The office said in a statement on Monday morning that the men were arrested because of “a strong suspicion of membership of foreign terrorist organization al-Shabaab and the preparation of serious subversive violence”.
The apartment of another accused was also searched.
The three travelled to Somalia in 2012 and 2013. Prosecutors believe they were trained in an al-Shabaab camp and “subsequently participated in the armed struggle” against the Somalian transitional government and African Union forces.
But they added there was no evidence that they were plotting an attack on German soil.
The three men were arrested at around 6pm after arriving on a plane from the Kenyan city of Mombasa.
Broadcaster SWR reported that the men had returned to Germany from Kenya as they no longer wanted to take part in the bloody campaign being waged by al-Shabaab in east Africa.
Al-Shabaab is aligned with al-Qaida and has stepped up attacks against government and African Union forces in Somalia.
The terror group's leader was killed by a US airstrike last Monday.
The group carried out the terror attack on the Westgate shopping mall in Nairobi last September in which 67 people were killed.
SEE ALSO: Isis 'releases' kidnapped German