Petry, who heads a national conservative faction within the fledgling AfD, won a clear majority at a two-day extraordinary party congress in the western city of Essen, which gathered about 3,500 members.
The party had been riven by a months-long power struggle between Lucke, 52, its economic neo-liberal leader and founder, and 40-year-old Petry, with analysts even warning it could implode.
Some 60 percent of delegates voted for Petry, who is head of the AfD in the eastern state of Saxony which late last year saw the rise of the anti-Islam, anti-immigration PEGIDA movement.
Petry, a businesswoman and mother of four who supported talks with PEGIDA, had accused Lucke of focusing too much on criticising Germany's policy on Europe.
She wants to address all issues "without taboos" that are of concern to citizens, including immigration, she told the congress.
She drew particular applause with remarks about "massive integration problems linked to the fact that a religion like Islam conveys a vision of the state that is totally foreign to that which we know in Europe".
Ahead of the congress, her ally, Alexander Gauland, who heads the AfD in eastern Brandenburg state, said in Spiegel news weekly that Lucke had wanted to "almost exclusively" concentrate on criticism of the euro. "But that is not enough to survive politically," he said.
"He wants to exclude all the topics that have led us to success in the regional elections -- many people's fears faced with a growing number of refugees, border crime, multicultural society," he added.
After it was founded in early 2013, the AfD's main battle cry was for an orderly dissolution of the euro and a return to the deutschmark after the financial turbulence that nearly brought the eurozone to its knees.
It came close to entering the German parliament in 2013.
Last year it won seats in the European Parliament, followed by representation in five German state assemblies, after it sought to widen its appeal by incorporating populist positions on law and order, immigration and traditional social values.
A poll carried out during the congress showed that “uncontrolled immigration" was the main concern of supporters, while the euro crisis was second, the day before a crunch bailout referendum in Greece.
Petry had told delegates before her election that she wanted to create a team ready to work together and to "politically provoke".
"We must end the confrontations," she later told delegates after her win.
"It's not a victory of conservatives over liberals. It's only together that we are strong and it's together that we can change politics," she added.
Lucke, on leaving the podium, said he was "obviously disappointed but also relieved" to be free of the weight of the party leadership. He had garnered just over 38 percent of the vote.
But he declined immediate comment on his future within the group which has grown to 23,000 members.
Asked if he had lost the AfD, he responded "yes" and said he was disappointed with the atmosphere among delegates.
He recently created an offshoot group within the AfD entitled "Wake Up Call 2015" to gather supporters.