Both Mainz striker André Schürrle and St Pauli midfielder Matthias Lehmann scored two goals, but a deft first-half chip by Mainz's Hungary striker Adam Szalai kept the guests ahead on a bitterly cold night in Hamburg.
Mainz midfielder Marco Caligiuri settled the matter in the 83rd minute when he took a clever pass from substitute striker Sami Allagui and drilled his shot past the goalkeeper as Mainz went second.
"The way we started is what we had worked towards all week," said Mainz coach Thomas Tuchel after his side raced into a 2-0 lead after 30 minutes. "If we keep producing performances like that, the luck will come back."
Dortmund's lead at the top has been cut to ten points and Leverkusen can bridge the gap further if they can reclaim second by beating Freiburg on Sunday.
Earlier leaders Dortmund were stunned when the league's top-scorer Theofanis Gekas netted his 14th league goal of the season on 86 minutes with some superb finishing. It was only the second defeat of the season for the league's top side.
"They deserved to win," admitted Dortmund coach Jurgen Klopp with the Bundesliga set to take its customary winter break on Sunday. "They were more passionate than we were in the one-on-one challenges. After 17 matches, it doesn't feel so good because we lost, but things are better than they feel right now."
Dortmund would have claimed the German league record for the best start to the season if they had drawn or won.
Defending champions Bayern Munich are at Stuttgart on Sunday and will be 14 points behind the leaders if they claim a victory.
Fourth-placed Hanover were stunned 3-1 at mid-table Nuremberg, while Champions League last 16-qualifiers Schalke picked up their third-straight league win with a 3-0 win over Cologne as Spain striker Raul hit a hat-trick.
It was his second in the German league since joining Schalke from Real Madrid at the start of the season. The hosts took the lead when midfielder Ivan Rakitic pounced on a mistake in the Cologne defence and squared the ball for the ex-Real forward to head home on 30 minutes.
The Royal Blues made sure of the result which leaves them 10th in the table when Raul completed his hat-trick with strikes on 50 and 87 minutes to take him on to nine league goals this season.
Hoffenheim allowed a 2-0 lead to slip to a 2-2 draw at Wolfsburg as ex-England manager Steve McClaren's side fought back. The result leaves Wolfsburg 13th in the league and just four points from the relegation zone, while Hoffenheim are eighth.
Werder Bremen slipped further towards the relegation zone when they were beaten 2-1 by Kaiserslautern to suffer their second straight defeat. Both sides had players sent off in a bad-tempered match.
It was Bremen's eighth reversal of the season in 17 games and they are 14th in the league, four points above third from bottom Cologne.