About 5,000 attended the event, held under the banner "Stand Up: Jew Hatred - Never Again!", which coincided with the first ever annual meeting of the World Jewish Congress (WJC) in the German capital.
"The fight against anti-Semitism is our state and civic duty," said Merkel in an address at Berlin's iconic Brandenburg Gate, close to the city's Holocaust Memorial.
"I will not accept it, and none of us here will accept it."
In July, at the height of the Israeli bombardment of Gaza, Germany's Jewish community condemned an "explosion of evil and violent hatred of Jews" at pro-Palestinian rallies where some demonstrators chanted that Jews should be "gassed", "burnt" and "slaughtered".
More seriously, four people were shot dead in May at the Jewish museum in Brussels, Belgium. The museum only reopened on Sunday.
The spate of ugly incidents that deeply unsettled Germany's resurgent 200,000-strong Jewish community also saw a petrol bomb hurled at the facade of a synagogue in the western city of Wuppertal.
The attacks came 75 years after the outbreak of World War II, during which Nazi Germany murdered six million Jews, a crime that remains a source of enduring shame in the country.
Merkel said the fact that today Jews are living again in Germany "is almost a miracle".
"Jewish life belongs here and is part of our identity.... There must be no room for discrimination and marginalization," she said.
Merkel - who has won Israel's highest civilian honour for her efforts against anti-Semitism and Holocaust denial - said that "Germany is aware of its eternal responsibility for the break of civilization called the Shoah".
"Those who hurl abuse at people who wear a kippah or a Star of David on a chain, attack them or assault them, also hit out at and injure all of us," she said.
"Those who deface gravestones in a Jewish cemetery also deface our culture. Those who make synagogues targets of hatred and violence also shake the foundations of our free society."
Germany, which was home to some half a million Jews before WWII, saw that number plummet to only around 15,000 after the war. The Jewish community has grown again, in part as Germany took in Jews from the former Soviet Union, and now numbers around 200,000.
'Medieval stain of anti-Semitism'
WJC president Ronald S. Lauder praised post-war Germany for being "one of the most responsible countries on earth" and "Israel's ally and friend," but added that "something has changed".
"This summer, all of the progress of the last 70 years has been darkened by a rising tide of anti-Semitism," he said. "There are some places I might expect to see this - but not here in Germany."
He cited several reasons for the recent resurgence of the "medieval stain of anti-Semitism".
"When the economy declines, people become fearful and often they look for a scapegoat. Throughout history, that scapegoat has been the Jews," he said.
He also pointed at "some of the vilest anti-Jewish propaganda coming out of the Middle East" and spreading across the Internet.
He warned that history has shown that "a group that instills fear and hatred may start small but can grow into a large and dangerous tidal wave".
The head of the Central Council of Jews in Germany, Dieter Graumann, said recent months saw "horrible shock waves of hatred against Jews, across Europe".
"Synagogues were attacked, Jewish people have been threatened," he told the rally, attended by many German political and religious leaders.
"In social networks tons of buckets of hatred, of malice and agitation were poured over us."
Graumann demanded that "Muslim organisations here must do a lot more than in the past to consistently combat the catastrophic hatred of Jews in their ranks".
Many Israeli flags and placards were raised at the rally, one of which read "Against Violence and Hatred - Security for Jews".
Eduart Schechter, 69 and originally from the former Soviet Union, said: "We moved to Germany with great hopes and dreams. But the reality is different to what we imagined. Now we are scared."
Alexander Schramm, a 29-year-old PR professional, said: "I think it's regrettable that it took so long to organize this rally. I am here to show that we'll stay alert."