Tunisian authorities issued an arrest warrant for the 36-year-old last year on suspicion of "participating in planning and carrying out" the attack, as well as a deadly jihadist assault on the border town of Ben Guerdane last March, the prosecutor's office in the western state of Hesse said in a statement.
Twenty-one people, mostly European tourists, were killed at the scene of the attack of the Bardo National Museum in March 2015, which was claimed by Isis. A further victim died ten days later.
Tunisian police killed two of the gunmen during the assault, but the third attacker managed to escape.
Police in Hesse raided over 50 properties early on Wednesday morning and arrested the Tuinisian suspect over an alleged plan to carry out a terror attack in Germany.
As well as searching addresses in Frankfurt, police carried out dawn raids in Offenbach, Darmstadt and the state capital, Wiesbaden. The raids were conducted on apartments, commercial properties and mosques.
"With this operation, we are sending a clear message to radical Islamists in Hesse: We have the scene firmly within our sights," said Hesse interior minister Peter Beuth.
A total of 16 males between the ages of 16 and 46 are suspects in the investigation and searches involving more than 1,100 officers.
The 36-year-old Tunisian is accused of recruiting for Isis in Germany and of planning to carry out a terror attack. Police say he was in the process of building up a terror network with the aim of carrying out an attack. The 36-year-old was the only person taken into custody.
A spokesperson for police in Hesse said that the group had not yet chosen a target for their attack and that the plans were in the early stages.
The suspect arrived in Germany as an asylum seeker in August 2015, the prosecutors said on Wednesday, after already living in the country for a decade some years earlier.
He was arrested the following August on an outstanding 2008 conviction for causing bodily harm.
After serving a 43-day sentence, he was kept in detention awaiting deportation to Tunisia before the authorities were forced to release him again.
"As the Tunisian authorities, despite repeated reminders from the German authorities, failed to supply the necessary deportation documents within the 40-day period, the suspect was released on November 4th, 2016," the statement said.
He was kept under surveillance from the day of his release until his arrest on Wednesday, it added.
Berlin police arrest three suspected jihadists
Police in Berlin said they arrested three people on Tuesday suspected of trying to reach foreign "war zones" in order to train to carry out an attack.
The suspects may be linked to Isis and had "planned to travel to war zones," likely Syria or Iraq, a Berlin police spokesman told AFP.
Police also carried out searches, but there has been "no indication of concrete plans to carry out an attack in Germany", according to the Bild daily's website.
The suspects, whose gender was not immediately given, were aged 21, 31 and 45, the spokesman said. He added that investigators suspect they were trying to reach a camp to train to launch an attack.
At least two of the people arrested lived in Berlin, Bild reported.
Those arrested frequented the same mosque attended by a Tunisian man suspected of ploughing a hijacked lorry into a Berlin Christmas market, killing 12. The December 19th attack was claimed by Isis.
The mosque, which is being investigated by authorities, was also searched.