The Italian embassy announced it was doing the same.
The moves follow similar decisions earlier this week by Britain, France and the United States regarding their embassies in Sanaa, the Yemeni capital.
"The situation is anything but stable," said a German foreign ministry spokeswoman, calling the ouster of the government by Shiite militiamen known as Huthis "unacceptable, dangerous and with consequences for the region".
"We decided yesterday to temporarily close the embassy in Sanaa and the personnel departed the country early this morning," she said.
On Wednesday, officials from the Foreign Ministry encouraged any German nationals residing in Yemen to leave.
The Italian foreign ministry said in a statement that "we hope mediation efforts led by the UN envoy Jamal Benomar will create security conditions allowing a return of diplomatic personnel to Yemen as soon as possible".
UN Secretary General Ban Ki-moon said Thursday that Yemen was "collapsing before our eyes" towards civil war.