The ship went down close to St. Goarshausen not far from Mainz early Thursday morning for unknown reasons, the shipping authorities in Bingen reported. However, the spot is infamous for its treacherous currents.
Helicopters equipped with infrared cameras circled the area in what authorities described as a "frantic" search for the missing men.
"The chances of finding both of them alive are unfortunately very slight," said a police spokesman.
The water police were able to rescue two other members of the crew, who have been taken to the hospital and are in good condition.
The ship, the Waldhof, has not yet been secured, but a police spokesman said the two-hull vessel was “aligned with the current” and there was no reason to believe any sulphuric acid, which is highly corrosive, had leaked into the river.
"There does not seem to have been any acid leakage into the water, according to an initial, provisional investigation," said spokesman Ralf Schomisch.
The ship measures roughly 110 metres (360 feet) long and was turned on its side, with its keel above the waterline.
The Waldhof's dangerous cargo was loaded at Ludwigshafen from BASF, the world's biggest chemical group, a spokesman for the company told news agency AFP. It was steaming north towards the sea reportedly heading for the Belgian port of Antwerp.
The authorities have closed the Rhine to all shipping traffic while the search and rescue continues.
"We are not able to say at the current time how long shipping traffic will be blocked," Martin Mauermann, a spokesman for Germany's Federal Shipping Administration, said. "We'll be occupied for several days or weeks."
The narrows near the Loreley have been the bane of sailors for centuries. According to legend, a beautiful siren at the spot calls to those sailing the Rhine, causing them to crash their boats on the rocks.
The scenic gorge is a major tourist draw, with holidaymakers from around the world taking cruises up and down the busy river.
Melting snow and rains have swollen waterways in the region in recent days, but it was not immediately known if that was a factor in the accident.