Gunmen in a vehicle opened fire at the bodyguard as he was leaving a store in the southern Hada district of the capital Sanaa, where foreign embassies are located, a security official and witnesses said.
An earlier report said the ambassador herself had been the target of a failed kidnap attempt. But a foreign ministry spokesman told AFP that Ambassador Carola Müller-Holtkemper "is currently outside the country".
In Berlin, the German foreign ministry confirmed on Monday that a security official at its embassy in Yemen has been killed but denied reports of an attempted kidnapping of its ambassador.
"We must unfortunately confirm that a German security official who was employed at the embassy in Sanaa was killed in Yemen," a German foreign ministry spokeswoman said in a statement about Sunday's incident.
"Reports about an attempted abduction of the German ambassador are unfounded," she added.
Medics in Sanaa said the corpse of the bodyguard was taken to the Saudi-German hospital in the capital. Security forces sealed off the area after the shooting.
In the north of city, gunmen kidnapped a Sierra Leonean citizen who works for UNICEF, diplomatic sources told AFP.
They said the man was seized at gunpoint from a UN vehicle, adding that his Yemeni driver was not abducted.
Foreigners are frequently attacked or kidnapped in Yemen, home to Al-Qaeda in the Arabian Peninsula (AQAP) which is viewed by the United States as the network's deadliest franchise.
President Abdrabuh Mansur Hadi on Sunday urged security forces to "boost alert levels to restore security and stability" in the face of "terrorist threats", the Saba state news agency reported.
On Wednesday, the pan-Arab Al-Hayat daily quoted Foreign Minister Abu Bakr al Qirbi as saying that ransoms paid to AQAP kidnappers have helped strengthen the network.
"Unfortunately, the ransoms that were paid to release hostages seized by the network have provided it with the money needed to renew its activity," he said.
Kidnappings are often carried out by tribesmen who use their hostages as bargaining chips in disputes with the central government and usually free them unharmed after tribal mediation.
But recently, AQAP militants operating from hideouts in the impoverished country's south and east have threatened foreigners.
Germany was among several Western countries, including the United States, which closed their embassies in August after US warnings of an Al-Qaeda attack.
The alert was reportedly linked to the interception of communications between Al-Qaeda leader Ayman al-Zawahiri and the leader of the group's Yemen-based affiliate, Nasser al-Wuhayshi.
Yemeni authorities have since claimed to have thwarted the alleged plot, and there have been several reports of US drone strikes killing suspected militants.