The two leaders also agreed in their exchange on Tuesday that Ukraine's territorial integrity had suffered "unacceptable blows", according to a a German government spokesman.
Obama and Merkel spoke after President Vladimir Putin signed a treaty claiming the Black Sea region of Crimea as Russian territory, and as Ukraine warned the showdown had entered a "military stage" after soldiers were killed on both sides.
Russia's move, less than three weeks after pro-Moscow troops first seized control of the strategic peninsula, was slammed by Western leaders.
"The one-sided declaration of Crimea's independence and the absorption into the Russian Federation that started today are unacceptable blows against the territorial integrity of Ukraine," German spokesman Steffen Seibert said in a statement.
Merkel and Obama agreed that the "so-called" referendum on secession held in Crimea on Sunday breached Ukraine's constitution and international law, Seibert added.
In a statement issued by the White House later on Tuesday. the leaders warned "there would be costs" for Russia's moves.
"They agreed it was vital to send international monitors from the Organization for Security Cooperation in Europe and the United Nations to southern and eastern Ukraine immediately," the statement said.
However, both Berlin and Washington said there was room for negotiations to de-escalate the situation.
"Both leaders agreed to continue to underscore to Russian President Putin that there remains a clear path for resolving this crisis diplomatically, in a way that addresses the interests of both Russia and the people of Ukraine," the White House statement said.
And the German statement emphasized "their willingness to pursue the path of political dialogue and communication," adding that the two countries had agreed on a "closely coordinated approach."