The Russian bomb was discovered during initial building work on an apartment block in the Zehlendorf neighbourhood on Thursday morning, the Berliner Morgenpost reports.
Police immediately cordoned off the surrounding streets while a bomb disposal unit arrived on the scene.
An appraisal of the situation then led police to evacuate an area within a 250-metre circumference of the find, with officers going house-to-house asking people to leave their homes.
As the find was in a wealthy neighbourhood of detached houses, only 500 residents are affected by the evacuation which started at around 2pm.
But among the buildings being evacuated are a kindergarten, a care home and a hospital. A total of 120 patients are being relocated from the Helios Clinic, a private hospital which is situated in the evacuation zone.
Police aim to have the evacuation finished by 4.30pm so that the bomb can be diffused an hour later.
Many bombs dropped during the Second World War still lie unexploded under German cities and are discovered during excavations for construction work.
In September several thousand people were evacuated in Cologne when a bomb was discovered in the Riehl neighbourhood.