Two of the fire's victims are in critical condition with serious burns and may not survive, a police spokeswoman told German news agency DDP.
A third seriously injured person was transported to the University Clinic Ulm with multiple broken bones.
Seventeen other adults were sent to surrounding hospitals with minor injuries and smoke inhalation suffered in the blaze that was reported at about 2:30 am.
Several residents of the house jumped from the windows as firefighters struggled for an hour and a half to contain the blaze, DDP reported. Forty-nine people were evacuated from neighboring buildings.
Officials believe the fire may have started in the building's rear courtyard. Neu-Ulm police investigators had to wait until late Monday morning to enter the building because of fears it would collapse.