Explosives experts from Cologne police were searching the neighbourhood of Riehl, on the west bank of the Rhine on Wednesday when they the discovered the American-made explosive, the Rheinische Post reports.
A total of 3,500 people who live within a 700-metre radius of the discovery are set to be affected by the evacuation.
Experts plan to defuse the bomb on Thursday afternoon.
— ksta.de/koeln (@ksta_koeln) September 29, 2016
Drivers have also been advised to avoid the area around the Mülheim Bridge on both banks of the river.
Over 70 years after the end of the Second World War, it is still common for unexploded bombs dropped by the Allies to be found in German urban centres.
This operation is comparatively small compared to evacuations that have taken place in the past.
In Hanover in 2013, 25,000 people had to leave their homes after several bombs were discovered in the northeast of the city.
Meanwhile city authorities in Berlin announced last year that they believe there are still 3,000 unexploded bombs lying under the city.