The last time Germany beat Italy was a 2-0 win in Zurich in June 1995 to celebrate the 100th anniversary of the Swiss Football Federation.
With the Azzurri winning all four games between the two sides since, Italy's 24-year-old Rossi cancelled out Miroslav Klose's first-half goal for the hosts.
The Bayern Munich forward's strike was his 59th on his 106th appearance as he draws closer to Gerd Müller's all-time Germany record of 68 goals.
Germany hosted Italy at Dortmund's same Signal Iduna Park stadium where the Italians knocked the hosts out of the 2006 World Cup on their way to winning the title.
The home side have a poor record in matches between the two nations with Italy enjoying 14 wins compared to Germany's seven victories with now nine games drawn.
"It was a pity that we couldn't get another goal," said Germany's vice-captain Bastian Schweinsteiger. "It was very difficult on the pitch, we are a team who try and dominate the game through our technical ability."
"It didn't work out, but I trust in the fact that we will be able to beat Italy once again in the future."
With the Azzurri rebuilding after their early exit from last year's World Cup, Italy coach Cesare Prandelli had included veteran goalkeeper Gianluigi Buffon less than a month after the veteran made his injury comeback.
Germany made a bright start and took the lead after just 16 minutes. Real Madrid winger Mesut Özil's long ball found Thomas Müller on the right side of the penalty area and he squared the ball for Klose whose shot clipped Buffon before hitting the back of the net.
Goalkeeper Manuel Neuer translated his top-class recent form for Schalke onto the international stage as he kept his side in the game early on with several excellent save.
Italy's best first-half chance came in the 37th-minute came when veteran midfielder Daniele de Rossi blasted his shot straight at Neuer who parried the danger away.
Germany coach Joachim Löw brought on Dortmund's 18-year-old wunderkind
Mario Götze for Müller at the start of the second half to win his second cap.
The son of a university professor in Dortmund, Götze still lives at home and is rated as a star of the future for the national side.
Having come on just after the break, Rossi made sure Italy left the Ruhr Valley with their 15-year-old unbeaten record intact when he pounced on a mistake in the Germany defence to equalise on 81 minutes.