The proposal was presented by Justice Minister Heiko Maas, and would offer gay men convicted under the law a lump sum of €3,000 as well as an additional €1,500 for each year they spent in prison and thus "suffered a deprivation of liberty".
The measure marks a triumph for activists after a decades-long struggle to clear the names of gay men who lived with a criminal record under Article 175 of the penal code.
An estimated 5,000 of those found guilty are still alive.
The legislation must still be approved by the Bundestag (German parliament), where Merkel's ruling right-left coalition enjoys a large majority.
"We can never completely erase the travesty of justice, but we want to rehabilitate the victims," Maas said in a statement.
The measure follows Britain's so-called "Turing Law" approved in October, which offered pardons to thousands of men convicted of homosexuality before its decriminalization in 1967.
The legislation was named for World War II hero Alan Turing who was prosecuted under the law in 1952 and forced to undergo chemical castration treatment. He committed suicide two years later at the age of 41.
However the British measure drew criticism from campaigners for only automatically pardoning dead people while the living must still make an individual application to have their names cleared. It also failed to provide compensation.
'Contrary to nature'
Germany's Article 175 outlawed "sexual acts contrary to nature... be it between people of the male gender or between people and animals".
Sex between women was not explicitly illegal.
Although it dated from 1871, it was rarely enforced until the Nazis came to power, and in 1935 they toughened the law to carry a sentence of 10 years of forced labour.
More than 42,000 men were convicted during the Third Reich, and sent to prison or concentration camps.
In 2002, the government introduced a new law which overturned their convictions, and also applied to those convicted of desertion during Nazi rule.
But that move didn't include those convicted after the war when article 175 was still in force.
The article was finally dropped from the penal code in East Germany in 1968.
In West Germany, it reverted to the pre-Nazi era version in 1969 and was only fully repealed in 1994.
Convicted under the law as a teenager in 1957, Fritz Schmehling, 74, said time was running out for victims to see justice.
"I don't want to die with a criminal record," he told AFP in a recent interview at his Berlin apartment.
"I've had cancer twice and was operated on but maybe I will still get to enjoy the moment my name is cleared. As sad as it is, in the time it takes, many of the older ones among us are going to die."
'Guardians of virtue'
Noted historian of the Nazi period Wolfgang Benz said in a 2016 book that although it was less brutal, the subjugation of gays continued seamlessly after the war.
"The authorities and other guardians of virtue in the post-Hitler era differed from the Nazis in that they tended to believe people were seduced (into homosexuality) rather than being born that way," he wrote.
"But their rage against the unwanted minority was no smaller."
Gay rights groups and the opposition Greens party have long pushed for the post-war convictions to be annulled.
But until now, their demand had been refused on grounds that the sentences were handed down by a court in a democracy and confirmed by a federal tribunal on appeal.