Heinrich Boere, who confessed to the killings as part of an SS hit squad in 1944, was brought to a prison hospital, the public prosecutor's office in the city of Aachen said, 21 months after being sentenced.
An expert had declared Boere – who suffers from heart problems and is wheelchair-bound – fit enough to begin his term provided certain medical care was provided, a spokesman said.
He had been staying at a nursing home until space became available at the prison hospital in the western state of North Rhine-Westphalia.
On several occasions, Boere admitted to shooting in cold blood pharmacist Fritz Bicknese, bicycle shop owner Teunis de Groot and Frans-Willem Kusters.
But he argued that as a member of an SS commando unit tasked with killing suspected resistance members or supporters, he risked being sent to a concentration camp if he refused.
He spent six decades one step ahead of the law after escaping from a prisoner-of war camp in 1947 and returning to his birthplace in Germany.
Boere, whose father was Dutch and who grew up in the Netherlands, was sentenced to death in Amsterdam – in absentia – in 1949. The sentence was later commuted to life imprisonment.
Afterwards, he remained a free man – working as a coal miner in Germany until 1976 – as Germany refused to extradite him in the 1980s, saying it was unable to determine if he was German or stateless.
Germany as a rule does not extradite its citizens to stand trial in other countries.
The Nazi-hunting Simon Wiesenthal Center launched a new drive in Germany Wednesday to catch the last perpetrators of the Holocaust still at large based on a groundbreaking precedent set by the May conviction of former camp guard John Demjanjuk, 91.