Both institutions, state-backed banks known as Landesbanken, were slammed by the global financial crisis and had sought a merger since late September.
But BayernLB said in a statement that "the economic benefits of a tie up compared to the profitability of an independent BayernLB do not sufficiently outweigh the typical challenges of implementing a merger of this size.
"The sole criterion for a potential merger was whether the merged bank could earn a sufficiently higher return on equity than each bank on its own," it added.
A WestLB statement said it "notes with regret the decision to suspend the talks with immediate effect before the key issues involved have been clarified."
WestLB sales agent Friedrich Merz said the bank "will continue to explore the possible options for a merger" within the context of a consolidation of the Landebanken sector.
WestLB agreed in May 2009 to cut its assets in half after a public bailout, and regional officials have mooted a tie-up with Hessian counterpart Helaba and a third, publicly-owned bank.