The balloon was caught by a gust of wind as it came into land at about 8pm on Sunday near Urbar, south of Koblenz in western Germany, according to an initial investigation by local police.
The balloon hit the ground several times, throwing four passengers out of the basket, before being dragged down a hill and getting caught in a tree, police said.
The pilot, who was stuck in the basket, died from his injuries.
The two injured are in a serious condition but out of danger, according to police, with all the passengers flown to local hospitals.
More than 100 people assisted in the rescue operation.
The crash site is across the river from the Lorelei, a steep slate rock and one of the Rhine gorge's most photographed natural phenomena.
The blue envelope of a hot-air balloon hung in the hillside above the Rhine valley near St. Goar after the accident on Monday morning, caught in trees. Photo: DPA