Gauck, 72, claimed 991 votes out of 1,232 from a special assembly of MPs and other dignitaries who voted on Sunday - against prominent Nazi hunter Beate Klarsfeld, 73, who was nominated as a protest candidate by the socialist Left party.
"What a beautiful Sunday," Gauck said to enthusiastic applause from the chamber of the glass-domed Reichstag parliament building in central Berlin after the vote.
It was the third presidential election in three years for Germany after the abrupt resignations of Gauck's two predecessors.
Gauck helped drive the peaceful revolution that brought down communist East Germany and later fought to ensure that the public would be granted access to the vast stash of files left behind by the despised Stasi secret police after reunification in 1990. He oversaw the archive for the next decade.
In a short acceptance speech, he noted that his election fell on the 22nd anniversary of the first free elections in East Germany after the fall of the Berlin Wall the previous November.
Chancellor Angela Merkel, who also grew up under communism, gave her backing to the plain-spoken Lutheran pastor in February after then president Christian Wulff stepped down amid a flurry of corruption allegations dating from his time as a state premier.
Wulff only served 20 months of his five-year term in office.
He had replaced Horst Köhler, a former head of the International Monetary Fund who bowed out after an uproar over comments he made appearing to justify using the military to serve Germany's economic interests.
Claudia Roth, co-leader of the opposition Greens party, which supported Gauck's candidacy along with the rest of Germany's mainstream parties, said the country was looking to Gauck to "give this badly damaged office dignity and respect again."
A poll for ARD public television released Saturday indicated that 80 percent of respondents consider him to be trustworthy.
The media and the public cheered his candidacy as an opportunity to remove some of the tarnish from the largely ceremonial office which serves as a kind of moral compass for the nation.
Expectations are outsized for Gauck, who has won a reputation across the country as an inspiring public speaker, albeit with a touch of vanity.
But as a staunch Protestant like Merkel, he is also keen to remind Germans that their hard-won freedoms carry weighty responsibilities with them - a lifelong theme he has said he will take to the presidential palace.
"From Gauck we can learn that democracy means thinking and acting for one's self rather than waiting for political redeemers," the influential news weekly Die Zeit said in its current issue.
Gauck himself warned scandal-weary Germans against seeing him as a redeemer, telling reporters the night he was nominated that they should not expect "Superman."
Story continues below…
He said he would seek to relieve Germans of some of their angst as Europe grapples with its sovereign debt crisis and to reach out to immigrant groups to foster integration.
Gauck looked set to buck tradition from the start, as his longtime girlfriend Daniela Schadt said he had no plans to divorce his estranged wife of more than 50 years Gerhild and wed her.
"I see no reason to marry for reasons of protocol," the 52-year-old journalist who is now Germany's First Lady told the Bild am Sonntag newspaper.
"Considering the fact that not only Jochen and I but the whole family accepts our arrangement, I think the rest of society can live with it," she said, using Gauck's nickname.