A steering committee in Berlin "rejected the request by Arcandor for aid from the economic fund" established to help firms hit by Germany's worst post-war slump, a government spokesman said.
Arcandor, which owns the Karstadt chain of department stores, had asked for €650 million ($900 million) in guarantees to help refinance loans that come due on Friday.
But Chancellor Angela Merkel has made it clear that state aid is "absolutely not imaginable" without a credible business plan.
The group, which owns 52 percent of Thomas Cook, might have to file for insolvency and Finance Minister Peer Steinbrück told ARD television Monday: "A bankruptcy is not totally excluded."
Arcandor had also requested €437 million in six-month credits from state-owned development bank KfW, and warned it could go under if it does not get help from Berlin, which has backed a multi-billion euro rescue plan for the carmaker Opel. But the German government also rejected the request for emergency funding later Monday, according to government sources.
Business publication Manager Magazin said in its online edition that Arcandor directors would present a new rescue plan to officials in Berlin on Monday.
A spokesman for the retailer told AFP the results of its requests would "probably be known on Wednesday."
Arcandor employs nearly 50,000 staff in Germany but its call for aid has left many observers cold because it was in trouble before the global economic crisis slammed Europe's biggest economy in mid-2008.
Steinbrck told ARD that "shareholders must assume their responsibilities" for the group's long-running problems.
"Suppliers and property owners should also be solicited," he added after press reports said Arcandor was paying excessive rents on some stores.
Shares in the retailer plunged in afternoon trading on the Frankfurt stock exchange, losing nearly one third of their value while the MDAX index on which they are traded was off by 1.56 percent overall.
Arcandor has two dominant shareholders, the Oppenheim and Schickedanz families, each with around 30 percent of the group, which also owns mail-order firms Quelle and Primondo.
The group is also mulling a possible merger with the biggest German retailer Metro, owner of the Kaufhof chain, with talks scheduled on Monday and Tuesday.
The government has stressed that its efforts on behalf of Opel were exceptional and warned that Arcandor must bear the consequences of poor management.
The European Commission has also expressed doubts regarding state aid for the retailer.
Former boss Thomas Middelhoff and his wife are suspected of involvement in a real-estate scheme that saw Arcandor pay top prices to rent stores, and prosecutor Angelika Matthiesen told AFP her office in the western city of Essen is mulling an investigation.
The company, which was founded in 1881, has also failed to grasp the mood of German consumers, more inclined these days to shop in malls and at discount retailers than in department stores.