A core group of around 40 protesters were occupying the Gerhart-Hauptmann-Schule in Berlin’s Kreuzberg on Wednesday afternoon.
The remaining demonstrators were demanding a personal meeting with Berlin Interior Minister Frank Henkel to discuss their right of abode, reported RBB broadcaster.
"Nobody knows what will happen next," Councillor for Construction Hans Pandorf told the radio station on Wednesday lunchtime, with authorities still attempting to find a peaceful solution.
Around 900 police officers entered the area around the occupied school on Tuesday morning.
A spokesman for the local council said they had tried to persuade the refugees to leave voluntarily and had found alternative accommodation for them.
The site of the former school has constantly made the headlines as a trouble spot. One person was killed and several injured during clashes at the school over the last 18 months. According to Bild newspaper, police have been recorded more than 100 violence incidents at the site.
Police closed off Ohlauer Straße around the school on Tuesday morning and buses took the 140 refugees to new accommodation in Spandau and Charlottenburg.
But around 20 refused to leave while several hundred demonstrators also gathered to try to stop the clearance. Tagesspiegel newspaper reported on Wednesday morning that there was still a handful of refugees left in the building.
Refugee rights group, Flüchtlingsrat Berlin, called for the police operation to end on Tuesday, arguing that “many” refugees living in the building rejected the request to leave because it was unclear what would happen to them. They said the refugees should get a right to live and work in Berlin.