The Bundestag, the lower house of parliament, rejected the motion, which called for a 40-percent binding quota for women in supervisory and management boards of large companies within a decade.
After an emotional debate, 320 lawmakers voted 'no', while 277 voted in favour, with one abstention.
Five months ahead of elections, opposition deputies had forced the vote which earlier in the week backed Merkel into a corner after some of her allies threatened to break ranks and help pass the motion.
Urging parliament's support, Frank-Walter Steinmeier, a leading member of the centre-left Social Democrats, said if progress continued at the current rate "it will take until the middle of the century until we have 40 percent of supervisory boards occupied by women".
"That is decidedly too late," he said.
Merkel, 58, the first female leader of Europe's top economy and often cited as the world's most powerful woman, has opposed the introduction of compulsory quotas for women in the boardroom. But her Labour Minister Ursula von der Leyen has long been a staunch backer of the move and initially had not made it clear whether she would toe the party's line in Thursday's ballot.
At what was described by a senior party member as an "intensive discussion", leaders of Merkel's Christian Democratic Union (CDU) agreed a compromise Monday to include in its manifesto from 2020 a 30-percent female quota in supervisory boards of large companies.
The move marked a change of course just four months after CDU members backed a flexible approach for companies to voluntarily decide on quotas for women in the boardroom at their party congress.
Merkel appeared to play down her party's internal squabbling.
"Issues of equal treatment, family policy, also the childcare benefit, such things are always discussed by us in the CDU with a great deal of passion," Merkel told Thursday's Bild mass circulation daily.
"And one learns, not all women think alike," she added.
Her spokesman told reporters Wednesday that Merkel's stance in the debate remained unchanged - there must be more women in top business posts.
The compromise succeeded in uniting the CDU, including von der Leyen who is also a deputy leader of the centre-right party, to reject the opposition's draft legislation.
However von der Leyen has faced stiff criticism from within her own party, with Christean Wagner telling the daily Frankfurter Allgemeine Zeitung that the minister could not afford another such incident.
"Otherwise she'll endanger the election victory of the Union," he was quoted as saying, referring to Merkel's conservatives.
Media commentators viewed as a setback the change of policy by Merkel, who hopes to clinch a third term in September 22nd general elections, and said it had created bad blood within the conservative bloc.
"Instead of a compromise which everyone can take into the election campaign, there are bloody noses and distrust in the party," the Tagesspiegel commented.
"Because the chief (Merkel) once again didn't commit, everyone involved is in the end standing about embarrassed," it added.
Bild called it a "defeat" for the CDU leadership and Merkel but questioned whether any good could come out of it in the winning of more female votes.
"Whether this turnaround on the quota issue will now send female voters in droves to the CDU may be doubted," it said, suggesting it was about von der Leyen positioning herself for the post-Merkel era.
Merkel's junior coalition partners, the pro-business Free Democratic Party, opposes fixed quotas. Before the vote FDP parliamentary group chief Rainer Brüderle expressed relief the opposition's attempt to drive a "wedge" into the coalition had failed.
Women made up 4 percent of management board membership among the 200 biggest German companies in 2012, according to a study by the DIW economic research institute.