Goals in each half from Marco Reus and Mesut Özil ensured Germany stayed ahead of Ireland and Sweden in Group C, after a 3-0 victory against the Faroe Islands in its first qualifier on Friday.
But a goal by Werder Bremen midfielder Zlatko Junuzovic in the 57th minute, as well as narrowly missed chances by a combative Austrian team in the last few minutes before the final whistle, kept the Germans on their toes until the end.
"I'm very happy with the result because we got three points away in Austria against a strong team," Germany coach Joachim Löw said after the match.
"With the game itself, I'm only marginally happy... all the Austrian chances came from errors on our part. After the 2-1, we should have controlled the game.
"We got ourselves into difficulty and we were lucky (Marko) Arnautovic missed that chance," he added after the Werder Bremen midfielder missed a close range shot on goal in the dying minutes of the game.
Real Madrid midfielder Sami Khedira agreed. "We can be happy with our three points but not with the way we won them," he said.
"We were up 2-0 and then we let Austria back into the game with tactical errors. That should not happen."
Germany seemed to have the game in the bag when Junuzovic hit the back of the net, spurring the Austrians back into action, encouraged by some 47,000 spectators in Vienna's packed Ernst Happel Stadium.
"We made mistakes and literally handed our opponents two or three invitations (to score)," said Bayern Munich's Thomas Müller, who helped Germany to its second goal when he was brought down by Austria's Veli Kavlak, leading to a penalty that was expertly converted by Özil.
"We did not deliver a perfect game... we were just lucky in the end."
The encounter had been built up on the Austrian side as their best chance in years to beat traditional rivals Germany, with confidence high after several international victories, and players benefiting from experience playing abroad.
The Germans are now focussed on their next opponents Ireland and Sweden – ranked 26th and 18th in the world by FIFA, compared to Austria's 49th place – which are the greater adversaries in Group C.
Still, skipper Philipp Lahm acknowledged that Austria had posed serious problems for the German team in the first and last 25 minutes of the game.
"Austria confirmed it had a good team today," he noted. Austria's Swiss coach Koller meanwhile vowed to build on Tuesday's experiences.
"We showed we could play well but unfortunately the points didn't show that."
"When you lose, you're not happy. But how it happened, how they played, that was good. We need to keep that up," he said, as Austria prepare to face Kazakhstan twice next month.
VfB Stuttgart's Martin Harnik, who had a number of chances in the first half, agreed.
"We created our own failure... we had the Germans under control. We can lose against them; the important thing is now to get the points against the other teams."