Adolf Sauerland has consistently refused to resign and denied any responsibility for the 2010 disaster despite accusations of having ignored repeated warnings about the safety of the plans.
He was so unpopular in Duisburg afterwards, that relatives of those who died said he was not welcome at memorial services, and he has been pelted with food and ketchup when out in public.
Those wanting to see him kicked out of office before he was due to leave in 2015 made an enormous effort to gather 80,000 signatures to force the referendum which was held on Sunday.
Figures released in the evening showed 129,833 people voted for the controversial conservative mayor to step down, while 21,557 wanted to keep him. The law says 25 percent of those entitled to vote would be needed to dismiss him – in the event 35.5 percent of those entitled to, voted him out.
Sauerland, a member of the Christian Democratic Union, said he was surprised by the result, and said he would clear his desk by Wednesday.
“This is a good day for democracy,” said North Rhine-Westphalia state interior minister Ralf Jäger of the Social Democrats.
A new election in the city must be held within the next six months.
The Love Parade held in Duisburg in July 2010 had been promoted as a sign of the region's economic recovery and increasing cultural significance within Germany. But it ended in disaster after people trying to enter the main party area collided with those trying to leave and a crush led to panic in which 21 people were killed and more than 500 injured.
Investigations are still ongoing into the cause, and although city employees are being examined, Sauerland is not. He is under investigation for alleged corruption in another matter.