Members of Tel Aviv's gay community turned out to see the stone monument, modelled on the pink triangle Nazis made homosexuals wear in concentration camps during World War II, which features inscriptions in German, Hebrew and English.
"In addition to the extermination of Europe's Jews, the Nazis committed many atrocities, in an attempt to destroy anyone who was considered different," Tel Aviv Mayor Ron Huldai said at the unveiling.
"This monument reminds us all how important it is for us to respect every human being," he said.
The heart of Israel's cultural life and a bastion of secularism, Tel Aviv hosts an annual gay pride parade with relatively few objections from the country's religious community, unlike similar events in Jerusalem that have seen violence and even one stabbing.
Israel is widely seen as having liberal gay rights policies, despite the hostility shown towards homosexuals, particularly men, from the ultra-orthodox Jewish community.
Former Tel Aviv council member Eran Lev, who initiated the monument, said the idea behind it was "to commemorate those victims of the Nazi regime in a universal way; namely, not only Jews but all individuals."
German Ambassador Andreas Michaelis said that "In Israel, we have to focus, first and foremost, on the Shoah," using the Hebrew word for Holocaust.
"But in doing so, we must not forget other groups of people who fell victim to the murderous machine operated by Germany in those fatal years. Today we do so for those who suffered and lost their lives just because of their sexual orientation."
Adolf Hitler's Nazi Germany sent thousands of homosexuals to concentration camps in the 1930s and 1940s, and the Gestapo secret police arrested an estimated 100,000 men for being gay.