Netanyahu's meeting with Sigmar Gabriel was cancelled after the German diplomat decided to go ahead with talks with Israeli rights groups Breaking The Silence and B'Tselem.
Gabriel held a joint meeting with the rights groups in the evening, a spokesman for B'Tselem told AFP, without giving further details.
Breaking The Silence seeks to document alleged Israeli military abuses in the Palestinian territories, while B'Tselem has worked on a range of issues and has strongly opposed Israeli settlement building.
Cancelling the meeting was a rare step, but in line with the current right-wing government's stance against groups it accuses of having political agendas and of unfairly tarnishing Israel.
Netanyahu's office issued a statement saying, in a reference to Breaking The Silence, "imagine if foreign diplomats visiting the United States or Britain met with NGOs that call American or British soldiers war criminals".
"Diplomats are welcome to meet with representatives of civil society but Prime Minister Netanyahu will not meet with those who lend legitimacy to organisations that call for the criminalization of Israeli soldiers."
It added, however, that "our relations with Germany are very important and they will not be affected by this".
Gabriel told journalists in Jerusalem that he regretted Netanyahu's decision, but also said that he did not think it would badly impact relations between the two countries.
He and Netanyahu, who also serves as Israel's foreign minister, "will find an opportunity to speak with each other on the phone or to meet in an upcoming visit", he said.
"This does not mean the breaking of diplomatic relations. We need to keep this a little bit in perspective."
Gabriel earlier told German public television station ZDF that a decision to cancel the meeting would be "extremely regrettable".
"It is completely normal that we speak with civil society representatives during a visit abroad," he said.
Gabriel added that it would be "unthinkable" to cancel a meeting with Netanyahu if he met critics of the German government during a visit to Germany.
Such disputes have arisen in the past between visiting foreign officials and Israel's government.
In February, Israel reprimanded the Belgian ambassador after his country's premier, Charles Michel, met both B'Tselem and Breaking The Silence during a visit to Israel.
However, there was no public rebuke when British Foreign Secretary Boris Johnson met with anti-settlement NGO Peace Now during a visit in March.
Israel has occupied the West Bank for 50 years, and Jewish settlement building in the Palestinian territory has drawn intense international criticism.
Israeli settlements are seen as illegal under international law and major stumbling blocks to peace efforts as they are built on land the Palestinians see as part of their future state.
Germany has been among critics of Israeli settlement policy.
A German government spokesman said in February that a summit with Israel planned for May had been delayed, with Israeli media reporting it was because of the Jewish state's controversial new settlements law.
Israel passed a law in February that legalizes thousands of settler homes built on private Palestinian land in the West Bank.
The current government, seen as the most right-wing in Israeli history, has frequently criticised NGOs it accuses of unfairly tarnishing the country's image.
'A red line'
Last year, Israel's parliament adopted a law seen as targeting left-wing groups critical of the government by forcing NGOs that receive most of their funding from foreign states to declare it.
The law was criticized by the European Union, which said it risked "undermining" values that the EU and Israel shared.
Both B'Tselem and Breaking The Silence are among groups that have received European support.
Israeli deputy foreign minister Tzipi Hotovely said Tuesday that her ministry for the past couple years was waging a battle against "anti-Israel organisations that are massively funded by European states".
"We've decided to draw a red line," she told army radio.
Gabriel, however, still met President Reuven Rivlin, whose role is mainly ceremonial, on Tuesday afternoon.
After the two men gave speeches, they held a heated meeting lasting more than an hour, a source in the president's office told AFP.
Rivlin repeatedly called Israel's army the "most moral in the world", the source said.
By Laurent Lozano