The xenophobic Alternative for Germany (AfD), which was running for the first time for seats in the regional parliament of Mecklenburg-Western Pomerania, is projected to possibly unseat Merkel's Christian Democratic Union from second place.
Although the former Communist state is Germany's poorest and least populous, it carries some symbolic meaning as it is home to Merkel's constituency Stralsund.
The polls are also held exactly a year after the German leader made the momentous decision to let in tens of thousands of Syrian and other migrants marooned in eastern European countries.
Although she won praise at first, the optimism has given way to fears over how Europe's biggest economy will manage to integrate the million people who arrived last year alone.
Her decision has left her increasingly isolated in Europe, and exposed her to heavy criticism at home, including from her own conservative allies.
In the sprawling farming and coastal state of Mecklenburg-Western Pomerania, where economic regeneration and jobs used to top residents' concerns, the issue of refugees and integration is now the deciding factor for one in three voters.
“I am voting AfD. The main reason is the question over asylum-seekers,” said a pensioner and former teacher who declined to be named.
“A million refugees have come here. There is money for them, but no money to bring pensions in the east to the same levels as those of the west,” he said, referring to the lower retirement payments that residents of former Communist states receive compared to those in the west.
AfD leader Frauke Petry released a video Friday urging voters to “make history not only in the state-region, but the whole of Germany” by backing the party massively in the polls.
By 1200 GMT, voter turnout had reached 32.8 percent, above the 29.8 percent recorded at the same time in the last elections in 2011.
Illustrating the political damage to Merkel over her policy, a survey published Thursday show her Christian Democratic Union (CDU) expected to garner 22 percent at the polls — only as much as the AfD.
The Social Democrats (SDP) — which had topped the vote in the last polls in 2011 — meanwhile were predicted to win around 28 percent at the elections, two weeks before capital Berlin holds its state polls.
Compared to other parts of Germany, the northeastern Mecklenburg-Western Pomerania hosts just a small proportion of migrants under a quota system based on states' income and population — having taken in 25,000 asylum seekers last year.
Most of them have already decided to abandon the state, preferring to head “where there are jobs, people and shops,” said Frieder Weinhold, CDU candidate. But he acknowledged that the “migration policy has sparked a feeling of insecurity among the people.”
After a series of attacks by asylum-seekers in July — including two claimed by the Islamic State organisation — the mood has also darkened.
Political analyst Hajo Funke estimates that the AfD will win 25 percent of the vote and become the second biggest party in the state.
The neo-Nazi NPD could garner five percent, he said, even if he did not expect Merkel to change course over the vote given the relatively small weight of the state. Yet this would mean that the AfD, which was founded in 2013, would enter yet another regional parliament. The party is now represented on the opposition benches of half of Germany's 16 regional parliaments.
Leading members of the party have sparked outrage over insulting remarks, including one disparaging footballer Jerome Boateng, of mixed German and Ghanian descent, as the neighbour no German wants.
Days ahead of Sunday's vote, Merkel urged the population to reject AfD.
“The more the people who go to vote, the less the percentage won by some parties that, in my view, have no solution for problems and which are built mainly around a protest — often with hate,” she said.
Polls will close at 1600 GMT and first result estimates will be published shortly after. The chancellor, who is attending the G20 summit in the Chinese city of Hangzhou to attend the G20 summit, is not voting in the polls as her main residence is in Berlin.