A first-half goal by West Ham's Matthew Upson was cancelled out by Bayer Leverkusen Patrick Helmes' second-half equaliser following a mix-up in the England defence before Terry headed the winner in the closing stages.
Chelsea star Terry had been doubtful coming into the game after injuring his foot last weekend against West Brom, but the 27-year-old led from the front and scored the vital goal for his country.
"It was a good performance and the youngsters in the team have really given the manager some problems," said Terry whose goal inflicted Germany's first defeat in Berlin since 1973.
The skipper also took the blame for Germany's equaliser which came after an embarrassing mix-up between him and substitute goalkeeper Scott Carson. "I hold my hands up. I should have cleared it up. It wasn't Scotty's fault."
Capello admitted he was delighted with the result. "It was a very good performance. We made lots of chances. I liked the attitude, confidence and personality," said the Italian.
England were missing a host of stars through injury including Wayne Rooney, Rio Ferdinand, Ashley Cole, Frank Lampard and Steven Gerrard while veterans Michael Owen and David Beckham were left out through a lack of match fitness. Arsenal teenager Theo Walcott was ruled out on the eve of the match after dislocating his shoulder in training.
Germany, who lost the Euro 2008 final to Spain in June, were also without their own key men in Michael Ballack, Torsten Frings and Philipp Lahm. But with one eye on the future, Germany coach Joachim Löw started uncapped Hoffenheim defender Marvin Compper while Capello gave Aston Villa's striker Gabriel Agbonlahor his first cap and start.
It was England who scored first when they were rewarded for an early piece of pressure. Their second corner in quick succession was swung in and Germany goalkeeper Rene Adler looked to have been hindered by Jermain Defoe as he went to punch the ball clear.
In the resulting scramble, Upson was the sharpest to react and stabbed the ball home on 23 minutes to put the visitors ahead as England controlled the first half with Germany's attack rarely threatening.
There was plenty of changes at the break with Capello swapping goalkeeper David James for Scott Carson, while striker Darren Bent took over from Defoe. Löw switched his 'keeper Adler for Werder Bremen's Tim Wiese to make his debut and teenage midfielder Marko Marin for Schalke 04's Jermaine Jones, while Helmes took over from captain Miroslav Klose.
The changes worked in Germany's favour as a horrible mix-up in the England defence between Terry and West Bromwich Albion shot-stopper Carson saw the ball bobble to Helmes who fired into an empty net on 63 minutes to equalise.
But Terry rectified the situation with time running out when he rose higher than his marker Heiko Westermann to header past Wiese to give England the winner.
"The Englishmen were better over the entire game," said Löw. "We had a bad day at the office and earned the defeat.You could see England getting stronger and we didn't take our chances when we had them."
This was the 31st meeting between the old rivals with England pulling further ahead in the series with 15 wins now compared to Germany's 11 victories.