The approximately 80 centimeter crocodile was swimming in a backwater of the Innerste river, police told German daily Hildesheimer Zeitung on Tuesday.
Officials believe the worker's statements to be reliable. City spokesman Horst Richter told news agency AP that one of the men is a reptile enthusiast. But city water service workers have been unable to locate the animal since the sighting.
The men saw the animal for about five minutes, Richter said. After the men observed the crocodile, they said it disappeared under a low hanging branch.
The city has recruited an animal control expert to conduct a search for the animal, which could be a cayman, crocodile or cayman, Richter told the AP.
An animal expert told news agency DPA the animal could have been released into the river by a frustrated pet owner.