The improvement was in large part the result of a change in the way unemployment is calculated though, and one analyst warned that the jobless rate in Europe's biggest economy would worsen in coming months.
The number of unemployed in Germany rose on a seasonally adjusted basis meanwhile by a meagre 1,000 in May from April, the agency said.
In April, the number of jobless had jumped by 58,000 as Germany's worst recession since World War II slammed the labour market.
Figures in May were distorted downwards by around 20,000 owing to a new rule that instructed the labour agency not to count people undergoing training within a company as unemployed, it said.
And Capital Economics economist Jennifer McKeown noted that "survey measures of employment intentions point to an underlying picture of heavy job cuts.”
"As the severity of the recession has become clear, government incentives to maintain headcounts and cut hours have become increasingly ineffective," she warned.
A boost to household incomes from inflation that has now dipped into negative territory "will be at least partly offset be a further sharp rise in true unemployment in the coming months," McKeown said.