News weekly Der Spiegel writes in its latest edition that Merkel told Cameron on the sidelines of an EU summit last month that his moves to limit an influx of EU migrant workers undermined core principles of the bloc.
If he continues, Germany would no longer fight to keep Britain in the EU.
"For the first time, Cameron is pushing his country toward a 'point of no return' in terms of its EU membership, the chancellery and the foreign ministry believe," Spiegel said, citing unnamed German government officials.
If Cameron imposes a quota on EU citizens moving to Britain, "then that would be it", the publication quoted a government source as saying.
Spiegel said that Merkel's private discussion with Cameron barely touched on the more high-profile spat of the summit, a €2.1 billion ($2.6 billion) surcharge that Britain has been asked to pay for the EU budget by December 1st.
Cameron told parliament last week that the amount requested and the short deadline were "unacceptable", setting up a showdown with Brussels.
His Conservative Party is aiming to curb immigration, as it faces increased electoral pressure from the anti-EU UK Independence Party.
Under EU rules, citizens of most EU countries have the right to live and work in any other member state.
Merkel had previous warned in a newspaper interview last month that any attempts to "tamper with the fundamental principles of free movement in the EU" would not be tolerated by Britain's European partners.
Cameron has agreed to hold an in-or-out referendum on Britain's EU membership in 2017 if he wins the general election in May.
Up until now, Merkel has defended Britain from its critics in the EU.
SEE ALSO: Merkel blasts UK critics