The far-right National Democratic Party of Germany (NPD) tried to drum up anti-foreigner feelings at a rally near the asylum seekers' centre, a former school set amid drab tower blocks in the city's east, earlier this week.
Outside the refugee centre, meanwhile, anti-fascist protesters set up a vigil in a tent, playing music and putting up signs that said "Nobody is Illegal" and "Refugees welcome".
By Thursday, the remaining Antifa protesters were helping to keep an eye on things from the tent. They decide when television crews interview the refugees, and make sure any remaining NPDers stay at bay, Tagesspiegel Berlin newspaper reported.
But neo-Nazis have threatened to protest in other areas of Berlin where refugees are also being housed ahead of Germany's election on September 22nd.
Politicians have also condemn the protests which erupted on Tuesday night. Berlin MP Frank Steffel called for a national summit to discuss the problems faced by the refugees. He told Berlin newspaper BZ. “It cannot be the case that people who fear for their lives in other countries, also live in fear in Germany.”
On Tuesday night more than 500 anti-fascist protesters confronted around 40 anti-immigration activists, who eventually left in a street tram under police protection.
Hundreds of riot police separated the groups. One officer was injured by a bottle thrown in his face and 25 people were arrested, one for making an illegal straight-armed Hitler salute.
Stuck in the middle have been about 80 political asylum seekers, among them seven children, mostly from war-torn Afghanistan and Syria as well as Serbia – the first of about 200 people expected at the centre.
Around their temporary new home, where they arrived Monday, they have faced signs saying "No to the home" and "Have a nice flight home". Reports said some refugees had already left in fear.
"It is unbearable how right-wing demagogues are trying to sow fear," said Berlin's Mayor Klaus Wowereit. "Berlin is a city that is open to the world, and that's why we must allow no space for xenophobia."
Immigration has so far not been a major theme in the election campaign ahead of the September 22nd vote, and no far-right party has ever crossed the five-percent hurdle for entry into the national parliament.
But Interior Minister Hans-Peter Friedrich warned that refugees “must not be exploited by right-wing extremists for their propaganda of hatred."
Amid the Syrian war and other conflicts, the number of asylum seekers in Germany has risen to 52,754 people so far this year – 90 percent more than in the same period last year.
During the Berlin protests most of the refugees hunkered down inside the former Max Reinhardt high school, named after an Austrian-born Jewish theatre director who fled Nazi Germany to the United States.
One man who briefly looked out the door Wednesday was Winnie, 20, a former shopkeeper from Afghanistan who fled with his wife and child.
He said that as a Hindu he faced persecution by the Islamist Taliban and that he had paid people-smugglers $15,000 to organise a 25-day escape journey to Germany.
Baffled by the confrontations outside, he said "We don't know who is friendly to us and who is not". He added that nonetheless, "I feel safe. More than in Afghanistan."
A few blocks away, the Islamophobic group Pro Deutschland took the opportunity to launch their election campaign.
About a dozen party members were shielded by police as 100 counter-demonstrators sought to drown them out with boos and whistles.
The party's leader Manfred Rouhs, 47, claimed that 90 percent of political asylum seekers in Germany turn out to be "economic refugees", and that political refugees should first seek help closer to home.
One activist, Florian Klein, a 25-year-old student, said that neo-Nazis had in recent weeks sought to stoke neighbourhood sentiment by spreading "fear, lies and prejudice".
"With the vigil, we want to tell the asylum-seekers, 'you are not alone, you are welcome'," he said.
"To the racists and neo-Nazis ... we want to show the red card, we want to push back."
After the dark era of Nazi Germany, he said: "We know what persecution means. If, 80 years later, there are conflicts in other countries, and other people need refuge, we must help them."
Some neighbours made a point of welcoming the refugees, among them a young family who dropped off a bag of toys from their two-year-old daughter Jasmin.
"These people really have absolutely nothing," said Cindy Laqua, 29. "And the children have the very least to do with the whole situation."
Her partner, Omar El Aoud, said the events of the past few days were "disappointing", adding: "How can you blame the people who are inside here?"