Exit polls credited the AfD with around 13 percent of the vote, making it the third biggest political force in Germany -- a stunning result for a party that was founded just four years ago.
"We will change this country," vowed Alexander Gauland, one of the party's top candidates, pledging to "go after" Chancellor Angela Merkel's government.
The results mean at least 60 AfD lawmakers will sit on the opposition benches and be able to spread their views, including challenging Germany's culture of atonement over World War II and the massacre of six million Jews and others in the Holocaust.
The International Auschwitz Committee warned that the "conglomerate of anti-Semites, enemies of democracy and nationalistic agitators" will bring "an inhuman coldness" to the glass-domed chamber of the Reichstag building.
The Sueddeutsche Zeitung daily called the election "a watershed in the federal republic's history, comparable to the first vote of a unified Germany in 1990, although that case was a welcome event".
"For the first time in more than 50 years, a nationalist, extreme-right, broadly racist party will sit in the Bundestag ... That is sad, shameful and will change the climate in the country."
'Bikinis, not burkas'
The AfD began life as an anti-euro protest party but then shifted focus to capitalise on misgivings over the record one million asylum seekers who arrived in Germany since 2015.
Its tone turned increasingly extreme in the last stretch of election campaigning, with one of its two leading candidates saying Germany should be proud of its war veterans and claiming that terror was grounded in Islam.
Provocative posters declared "Burkas? We prefer bikinis" and "New Germans? Let's make them ourselves".
Its supporters heckled Merkel's rallies across the country, jeering, whistling and chanting "get lost" in attempts to drown her out.
Foreign Minister Sigmar Gabriel, a Social Democrat, has warned that "for the first time since the end of the second World War, real Nazis will sit in the German parliament". The Greens party has used the same term.
Merkel, more cautiously, urged voters to choose "the parties that are 100 percent loyal to our constitution".
The AfD will be a pariah in the Bundestag as all mainstream parties have said they won't work with it, but the party could still be vocally disruptive from the opposition benches.
'Le Pen pales in comparison'
A commentator in Spiegel weekly blamed Merkel for the rise of the AfD, saying she should be voted out simply for failing to stop "the Nazis from entering the Bundestag" on her watch.
Critics say widening social inequality, despite record employment, is also playing into the hands of AfD populists, who are most popular in the deindustrialised heartlands of the former communist east.
Thorsten Benner, director of the Global Public Policy Institute in Berlin, said the AfD's rise shows that "Germany is not so different from other European countries, where there are strong right-wing populist forces."
"Our population is no more virtuous than the French population," he noted, saying that even France's far-right leader Marine Le Pen "pales in comparison".
The party "will challenge key themes" in parliament, he said, pointing to Germany's culture of wartime remembrance and debate on cultural identity.
Gauland recently called for Germans to stop atoning for the past.
He also said integration commissioner Aydan Ozoguz should be "disposed of in Anatolia", suggesting she will never be German because of her Turkish origin.
The presence of the AfD "will very much change the tone of debate in parliament," Benner warned.
But Manfred Guellner, who heads the opinion poll institute Forsa, predicted that the AfD will self-destruct in the Bundestag.
"The AfD will disintegrate because that's what happens to sectarian groups from the right," he told Neue Osnabruecker Zeitung.