On the other hand, 45.2 percent of respondents said the Chancellor's welcoming stance to refugees was not a reason for her to leave office, the poll carried out by Insa for Focus magazine found.
The survey published on Friday shows fierce opposition to the Chancellor on the left and right of German politics.
Two out of every three voters for the right-wing Alternative for Germany (AfD) wanted Merkel to go.
The AfD has risen to become the third most popular political party in recent polls, with experts attributing this to its consistent rejection of Merkel's refugee policy.
At the same time, 45.4 percent of voters for the left-wing Die Linke party saw Merkel's refugee stance as cause for her to resign.
Die Linke are strong supporters of refugee rights and, speaking to the Local on Thursday, criticized the government for its policy of confiscating valuables from asylum seekers when they enter the country.
More surprisingly, 41.3 percent of voters for the Social Democrats (SPD) said they wanted the Chancellor to go.
The SPD leadership, who are in coalition with Merkel's Christian Democrats (CDU), have largely stayed loyal to the Chancellor on the refugee issue.
But ex-Chancellor and leader of the SPD Gerhard Schröder recently accused Merkel of "not having a plan" in the refugee crisis.
Most loyal were voters of her own party. Only 26.6 percent of CDU voters wanted her to call it quits.