Germany winger Marco Reus scored both Borussia's goals to spark hopes of the hosts reaching the semi-finals.
Real held on but clearly missed their star performer Cristiano Ronaldo, who was on the bench because he had not recovered fully from the knock to the knee he took in the first leg.
Dortmund coach Jurgen Klopp delivered on his pre-match promise that his side would bow out in style.
"This game is one for the archives, a video must be made and all teams who lose the first leg 3-0 must be shown it to say, 'it can still happen'," said Klopp.
"I am proud of the image the boys have given of Borussia Dortmund."
Dortmund could easily have finished with the 4-0 result they needed to take them through as midfielder Henrikh Mkhitaryan hit the post in the second-half with the goal at his mercy, then fired a shot straight at goalkeeper Ilker Casillas.
"Neither of his chances were easy," said Klopp.
"It happens in football, we missed other chances, Kevin (Grosskreutz) had one and we missed five or six chances in Madrid.
"With a goal at Real and one extra tonight, we'd have been through, but it would be great to judge the match as a whole and not only on the two missed chances."
Captain Casillas said Real Madrid were given a Champions League wake-up call as they lost to last season's beaten finalists.
"Maybe it's good for us to receive a wake-up call like that from time to time," said Casillas.
"We committed errors at the back that cost us and we cannot commit in the semi-finals."