The Karl Jaspers Clinic in the town of Wehnen has provided counseling for sex offenders for the past eight years.
"He is the first person who has ever managed to escape from here," the clinic's head doctor, Joachim Dedden, told news agency DPA. A police spokesman said the man was able to pry apart the bars on the window using a strap and a broom.
Clinic personnel noticed the man was missing during a 6 am inspection round. Three hours earlier, during the previous inspection, the 40-year-old had still been in his cell.
The man was tried and sentenced for rape in January 2010. He was also tried before an Aurich court six years earlier for attempted rape, taking of a hostage and aggravated battery.
According to a description provided by police, the man is approximately 170 centimetres in height, with thinning hair, glasses and gapped teeth. He also has a tattoo of a cross on his forearm.
Investigators have launched a manhunt, and a police spokesman told news agency DAPD that they had received a number of tips but no concrete leads. It could not be confirmed whether the man was still in the area or had already fled.
Dedden said the man had arrived at the clinic in August and was still in the early stages of treatment.
State authorities have sent a representative to be briefed on details of the escape.
Spokeswoman Heinke Träger said the representative would examine security and conduct talks with employees who were working at the time.