A day before a Paris summit on the migrant crisis, Merkel said the Libyan coastguard had to become capable of patrolling its waters and should be given "the necessary equipment to do its job".
"At the same time, of course we also consider it of the utmost importance that the Libyan coastguard adheres to international law, both in its dealings with refugees and migrants as well as non-governmental groups," she told the Welt am Sonntag newspaper.
"Should any doubts be raised about this, then we will investigate the allegations," she vowed.
The Libyan government sparked controversy this month when it barred foreign vessels from a stretch of water off its coast, claiming that charity boats active in the search and rescue zone were facilitating illegal migration.
The move was welcomed by Italy, the main port of arrival for migrants from north Africa, but several NGOs have since suspended their sea missions, accusing the Libyan coastguard of making threats and creating a hostile environment in the Mediterranean.
So far this year more than 100,000 people have made the perilous journey from Libya, according to the International Organization for Migration, and the European Union is eager for Libya to clamp down on the influx.
Over 2,300 have died attempting the crossing.
"We cannot allow the business of people smugglers who have the deaths of so many on their conscience," said Merkel.
French President Emmanuel Macron is hosting talks on the crisis at the Elysee Palace on Monday, with Merkel and her Spanish and Italian counterparts in attendance.
Libya's unity government chief Fayez al-Sarraj and the leaders of Chad and Niger will also attend the gathering.
Merkel, who is campaigning for a fourth term in next month's general election, told Die Welt am Sonntag that she had no regrets about her bold move to open Germany's borders to refugees in 2015.
"I would make all the important decisions of 2015 again in the same way," she said.
The arrival of hundreds of thousands of asylum seekers, mainly from Iraq, Syria and Afghanistan, deeply divided Germany and hurt Merkel's approval ratings.
But as the influx has slowed in recent months, Merkel's popularity has rebounded and her conservatives are comfortably leading in the polls ahead of the September 24th vote.