Klose, four shy of Ronaldo's World Cup record of 15 goals, picked two of several controversial bookings by Spanish referee Alberto Undiano and was sent off in the 37th minute for fouling Serbia captain Dejan Stankevic.
“Overall it was a fair match. It wasn't a nasty foul,” said Germany coach Joachim Löw, pointing out that ref doled out nine yellow cards. “That many yellows and a red were simply too many.”
Less than a minute later Serbia went ahead after a three-man move.
Milos Krasic sent a high ball in from the right post for Nikola Zigic, the tallest player at 2m 2cm high, to head down for Milan Jovanovic to flick in past keeper Manuel Neuer.
A goal and a man down, Germany attempted to get back into the match and they almost grabbed a first half stoppage time equaliser when Sami Khedira's shot hit the crossbar.
“We had difficulty finding our rhythm,” said Löw. “We were down a man for a long time and still managed to get our chances.”
Germany came on strong early in the second half and the ref awarded Löw's side a penalty after a deliberate hand ball by Serbian defender Nemanja Vidic.
But Lukas Podolski was unable to convert it and the striker failed to make use of several other good chances to equalise.
Subs Marko Marin, Cacau and Mario Gomez couldn't turn the tide and Serbia held on for their first victory of the tournament. The result marked a stunning turnaround in fortunes after Germany crushed Australia 4-0 in Durban in the team's first match of the tournament.
"I have to take responsibility," he said, referring to his flubbed penalty. "You can't blame the ref. But maybe the red card was an overreaction."
However, midfielder Bastian Schweinsteiger was visibly angry at the Spanish official after the match.
"I'm at a loss for words," he said. "For me that was ridiculous. If there's a red card in every match you have to ask yourself what's the point. It's no longer any fun."