"Everyone is just in shock here," said Claudia Bienek, a spokeswoman for the zoo in the German capital.
Knut, who shot to fame as a highly photogenic, snow-white cub in 2007, was aged just four years and three months, well below the average life expectancy for polar bears of around 35.
He was found floating dead on Saturday afternoon in the enclosure he shared with three other bears including his mother Tosca.
The cause of death was not immediately known, said Heiner Kloes, in charge of bears at Berlin Zoo, said. Vets were due to conduct an autopsy on Monday.
The BZ daily quoted witnesses as saying that Knut was sitting on rocks in his enclosure when his left leg began to shake. The bear then started walking around in circles before falling into the water, lifeless. Zoo workers then erected a screen around the enclosure. Soon afterwards the zoo closed, BZ said.
"This is awful," said Berlin Mayor Klaus Wowereit. "We had all taken him to our hearts. He was the star of the Berlin Zoo."
Knut weighed nine kilogrammes (19 pounds) at birth and triggered a wave of media coverage after an animal activist said he should have been put down after he and his brother, who later died, were rejected by their mother.
There was then a storm of attention as Knut was reared by hand, with his keeper Thomas Dörflein bottle-feeding him and strumming Elvis Presley songs for him on his guitar.
The first public appearance of "Cute Knut" in March 2007 attracted 100 camera crews from around the world.
He generated millions of euros for the Berlin Zoo in lucrative merchandising and extra entrance fees.
Twice-daily appearances allowed the public to watch Knut splash in a pond, jump on rocks and play with his handler, who had carefully removed the newborn from his aggressive mother's cage.
The cuddly white cub sent Berlin Zoo's shares soaring to an all-time high on the Frankfurt Stock Exchange as visitors queued in their thousands to watch him play.
Stuffed toy "Knut" bears sold out several times over, the cub made it onto the cover of glossy magazine Vanity Fair and Hollywood superstar Tom Cruise and his daughter Suri came to visit him, as did Leonardo DiCaprio.
But Dörflein, his keeper, died at 44 of a heart attack in 2008 and once Knut grew into a strapping adolescent and then adult, experts began to worry that he was displaying abnormal behaviour because of all the attention.
He would sway to and fro and even imitate people taking photos by lifting a paw to his face. Hopes that Knut might form a romantic attachment with one of the females were dashed as stories emerged of him being violently mauled by them.
"Knut's short and distressful life shows us again that polar bears do not belong in zoos, even if they are called Knut," said Wolfgang Apel, head of the German animal protection association.