The building was evacuated quickly, and no injuries have been reported. The blaze is thought to have been caused by renovation work on the roof that sparked insulation underneath the structure's metal hull.
The city's fire department rushed to the scene with more than 20 trucks and some 170 firefighters. They also had to remove some of the metal sheeting with blowtorches so water did not simply run down the sides of the structure.
"We can rule out a total loss of the Philharmonic," a spokesman for the fire brigade told news channel N-TV. "It's a steel and concrete structure that can withstand even larger fires."
A small concert had taken place in the building's foyer shortly before the fire. The visitors were able to leave without incident, and the musicians carried their priceless instruments to safety with the help of firemen. Although there was little danger immediately posed by flames, there were concerns the instruments could be damaged by water used to extinguish the blaze.
"We are doing all we can to save this Berlin cultural treasure," the spokesman said.
The iconic concert hall shaped like a giant bronze tent was build by architect Hans Scharoun in the 1960s. It is home to the city's world-renowned orchestra led by Sir Simon Rattle and is considered to have some of the best acoustics anywhere.
The orchestra had been due to hold a major rehearsal an hour after the fire started for two Beethoven and Berlioz concerts with renowned Italian conductor Claudio Abbado at the weekend, in what was intended as a highlight of the season.
"Thank God the fire broke out before," said Pamela Rosenberg, general manager of the orchestra.
Ralf Bauer Diefenbach, in charge of video transmission of the Abbado concert, called the fire "a catastrophe."
Solene Kermarret, a French violinist, said she was in another rehearsal when the smoke alarm sounded: "We all rushed out. This is strange, it is almost as though my house were on fire. It is awful to see it covered in smoke."