He died in the early hours of Tuesday in a hospital near his hometown in the southwestern state of Baden-Württemberg, a spokeswoman said.
Herzog served as head of state - a largely ceremonial but influential post - under Chancellor Helmut Kohl and later under Gerhard Schröder between 1994 and 1999, previously holding a position as a judge on the Constitutional Court.
Herzog, who grew up under the Nazi regime, is remembered for travelling to Poland as soon as he assumed his post to ask for forgiveness on the 50th anniversary of the 1944 Warsaw uprising.
The following year, half a century after the liberation of the Nazi death camp Auschwitz, he attended a Jewish ceremony at the site.
In 1996, Herzog declared January 27th as the first official day of remembrance for victims of the Nazi regime, marking the 1945 liberation from Auschwitz. He also made a powerful statement in 1997, recognizing the murder and persecution of the Roma and Sinti peoples under Adolf Hitler.
The politician from the conservative Christian Democratic Union (CDU) - Angela Merkel's party - was also an outspoken critic of politicians who were resistant to reforms. Herzog made it his task to tackle blockages in politics and society, and is often remembered for saying in 1997 “Germany needs to jolt itself into action”.
"Everyone is addressed, everyone has to make sacrifices, everyone has to participate," he implored the nation, a year before centre-left chancellor Schröder would take power and implement sweeping social welfare and labour reforms.
Chancellor Merkel praised Herzog as a "patriot" who had served the country in multiple posts and was "plain-spoken, unpretentious, humorous, and quite self-ironic".
The World Jewish Congress (WJC) on Tuesday honoured Herzog as a "great fighter for the rule of law and for a free and tolerant society".
Herzog had "distinguished himself with his great openness and friendship toward the Jewish community and for promoting its role in German civil society," said Maram Stern, the WJC's deputy CEO, in a statement.
Current President Joachim Gauck remembered his predecessor as a "distinctive character" with "progressive courage".
"He shaped and influenced Germany's self-image and the togetherness of our society," wrote Gauck in a letter to Herzog's wife, published online.
"Your husband called for a willingness to reform and at the same time stood for the preservation of what is tried and true... With these qualities, he contributed greatly to understanding between citizens and politics, and gained the respect and great sympathy of many people."
Justice Minister Heiko Maas took to social media to express his condolences.
"Sadness at the passing of former President Herzog: a president who was argumentative in a positive sense, who shaped our country and strengthened our togetherness," Maas wrote.
German EU Parliament President Martin Schulz also mourned Herzog on Twitter.
"With the passing of Roman Herzog, Germany and Europe have lost an influential personality and a great thinker and constitutional expert," Schulz wrote.
"Our condolences are with his relatives and friends."
Mit Roman #Herzog verlieren Deutschland und Europa eine prägende Persönlichkeit und einen großen Denker und Verfassungsrechtler 1/2- EP President (@EP_President) January 10, 2017
Born in Landshut, Bavaria in 1934, Herzog began a career in law and later became a professor at the Free University of Berlin.
Herzog joined the CDU in 1970 and became a minister in the southern state of Baden-Württemberg in 1978. By 1987, he became president of the Constitutional Court, where he served until becoming German President in 1994.
At the time of his death, Herzog was living in Jagsthausen, Baden-Württemberg with his second wife Alexandra Freifrau.