Four youths, thought to be of "Arab descent" confronted the 53-year-old in the Schöneberg district of Berlin in the early evening of Tuesday, asking him "Are you a Jew?" according to a police statement.
One of the men blocked his path, while three others stood behind the rabbi and his daughter. The man blocking his path suddenly hit the rabbi repeatedly, injuring his head.
"There were then insults directed towards the man, his beliefs and his mother, as well as a death threat in the direction of his daughter," the police said in a statement.
The youths fled after the attack and the rabbi was taken to hospital. Police said he was wearing "a traditional Jewish head covering."
Prosecutors have launched an investigation.
Berlin's mayor Klaus Wowereit condemned the attack "in the strongest possible terms," saying it was "an attack on the peaceful co-habitation of all people who live in our city."
The Jewish forum for democracy and against anti-Semitism said it was "very shocked by the act of violence."
Group spokesman Levi Salomon said the victim was "one of the first rabbis to be ordained in Germany after the Holocaust" when six million Jews were murdered by Adolf Hitler's Nazis.
He teaches religious studies at a Jewish school and has pushed for years for dialogue with Christians and Muslims, Salomon said.