"RWE and Gazprom today decided by mutual agreement to terminate at the end of the year their exclusive talks on a potential joint venture for the construction and operation of power stations" in the three countries, the German company said in a statement.
The two companies had announced the planned tie-up in July and agreed to explore potential cooperation possibilities by the end of the year.
"Although our discussions were conducted in a very constructive manner we were unfortunately not able to agree on a framework for cooperation which would be sustainable for both parties," said RWE chief executive, Jürgen Grossmann.
Separately, RWE said it had completed a planned capital increase that would bring some €2.1 billion ($2.7 billion) into its coffers.
RWE said in a statement it had succeeded in placing 80.4 million new shares at a price of €26 per share.
The power supplier is in need of fresh cash because earnings are being hit by the decision by Germany to phase out nuclear power.
RWE plunged into the red in the third quarter, running up a net loss of €174 million. It intends to divest 11 billion euros' worth of assets before the end of 2013, scale back investment and slash costs.