The town in North Rhine-Westphalia now faces heated criticism after the city council said that the 21 refugees could live in what used to be the quarters of SS overseers during World War II the Westdeutsche Allgemeine (WAZ) newspaper reported on Tuesday.
Schwerte was home to a forced-labour camp, a sub-camp of nearby Buchenwald, where Polish prisoners repaired train locomotives. None of the prisoners' accommodation is left standing, but the guards' barracks remains.
Director of the North Rhine-Westphalian Refugee Council, Birgit Naujocks, told radio broadcaster Mitteldeutschen Rundfunk that the decision was extremely "questionable".
"The plans recall bad memories and sinister pictures", said Naujocks, who also questioned why Schwerte could not have used shipping containers, as others have, to house the refugees.
Spokesman for Schwerte Carsten Morgenthal told WAZ that using containers was not an option because of cost reasons.
He also said the barracks were more suitable to make into a refugee home because they have previously been used to house refugees on one occasion 20 years ago. They have also been used to house a kindergarten, a warehouse and most recently a studio.
Up to 700 Polish workers were forced into labour there by the Nazis while it was used as a work camp between 1937 and 1945.
About 56,000 people died there, including nearly 12,000 Jewish people, according to Stern.