"This is the right time for me to call for a process of reconciliation," he said at the conference in southern Germany.
"We will invite all those Taliban who are not part of al Qaeda, who are not part of terrorist networks, who want to return to their country, who want to live by the constitution of Afghanistan and who want to have peace in their country and live a normal life, to participate, to come back to their country."
Karzai is set to stand again in presidential elections due on August 20, but his popularity has waned amid allegations of government corruption, growing opium production and an ever-more tenacious Taliban-led insurgency.
NATO nations and their partners fighting the Taliban in Afghanistan have given mixed reactions to Karzai's proposals to try to talk to the insurgents, with many rejecting talks with militants who have blood on their hands.
"I would request the international community to back us in this, fully, and be of one view on this, not of divided views on this," he said.
"There is no way that we can succeed in the way we want to, in the right time, without some form of reconciliation," he said.
Earlier at the conference, NATO Secretary General Jaap de Hoop Scheffer had called on some European allies to boost their contributions to the Afghanistan mission, especially regarding participation in combat operations against Taliban militants.
While the NATO chief did not mention any countries by name, it is thought Germany was one of the targets of the criticism.
But on Sunday, German Defence Minister Franz Josef Jung defended the German mission there, saying the discussion was too often focused merely on the military aspect. It is also important, he said, to consider civilian reconstruction efforts in which Germany is playing a leading role.
At the same time, Jung said he welcomed the speech given by US Vice President Joe Biden in which he called for closer cooperation between the US and Europe.