The probe came as a German state minister accused Turkey of the "unacceptable" espionage against supporters of Gülen, blamed by President Recep Tayyip Erdogan for a failed coup attempt last year.
The claims open a new front in the diplomatic row between NATO allies Germany and Turkey, whose relationship has been strained by a series of disputes centred on human rights issues.
"It is clear that the Turkish secret service MIT is investigating people living in Germany," said Boris Pistorius, interior minister of the northern German state of Lower Saxony, deploring the "intensity and ruthlessness" of Turkey's pursuit of people living abroad.
"It's intolerable and unacceptable," he said at a press conference.
Erdogan's government had asked Berlin to help spy on about 300 alleged Gülen supporters, Pistorius said, adding that the list was handed to Germany's BND spy service, which turned it over to state governments.
But Pistorius's state decided to inform the more than ten targets in Lower Saxony, including a school and at least two companies, fearing people could suffer "retaliation" if they travelled to Turkey while unaware they were on a watch list.
Turkish authorities were acting with "something close to paranoia," he said, adding that "all Gülen supporters are assumed to be terrorists and enemies of the state even though there is not the tiniest scrap of evidence."
"As of today, we have no evidence whatsoever that Gülen supporters have violated any rules in any way."
According to German media, Turkish officials handed the target list including names, addresses, telephone numbers and photographs to their German counterparts during the Munich Security Conference in February.
Federal prosecutors will now examine how Turkey compiled such detailed information on their targets.
"The success of our investigation will depend largely on the information shared with us by German counter-espionage agencies," spokeswoman Frauke Koehler said about the probe into "persons unknown".
Interior Minister Thomas de Maiziere meanwhile warned Turkey against spying in Germany, saying "espionage activities on German soil is punishable by law and will not be tolerated by us".
Although Gülen, a 75-year-old cleric living the United States, has denied charges that he was involved in the failed coup last July to overthrow Erdogan, Ankara has cracked down hard on the preacher's followers.
More than 41,000 people in Turkey have been arrested over suspected links to Gülen's movement, and 100,000 fired or suspended from their jobs. Many of them are teachers, police, magistrates and journalists.
In February, German police raided the homes of four Turkish Muslim preachers suspected of spying on alleged Gülen supporters for Erdogan's government.
Erdogan has in turn accused Germany of harbouring Kurdish and other "terrorists", claiming that Berlin is refusing to hand over alleged suspects.
Separately, the foreign ministries in Sweden and Denmark have called in Turkey's envoys over claims of Turkish spying on opposition figures living in the Scandinavian countries.
Germany's foreign intelligence chief Bruno Kahl drew Ankara's ire last week when he said he did not believe that Gülen was behind the failed coup.
Turkey had tried to convince Berlin that Gülen, who lives in a secluded compound in Pennsylvania, was behind the coup, "but they have not succeeded", Kahl told Der Spiegel magazine.
Kahl said the putsch was launched by a "part of the military" that expected to be targeted in an ongoing government purge.
Berlin has emerged as a strident critic of Ankara's post-coup crackdown, and is also urging Turkey to release a correspondent for the German daily Die Welt who is jailed on terror charges.
Ankara has been riled by German authorities' refusal to allow some Turkish ministers to campaign in the country's Turkish communities for a "yes" vote ahead of the April 16th referendum on giving Erdogan the powers of an executive presidency.