Hamburg is now out of the race, leaving Budapest, Paris, Los Angeles and Rome as the candidate cities bidding to stage the 2024 Summer Games, with the International Olympic Committee (IOC) due to announce their decision on September 13, 2017.
Local residents in Hamburg and Kiel, where the sailing events would have been held, vetoed the proposed bid with 51.6 percent of the 651,589 votes against the Olympic bid, according to official figures.
Hamburg's mayor Olaf Scholz conceded defeat shortly after 9pm.
"Hamburg will not bid to host the Olympic and Paralympic Games," he said glumly.
"I would have liked a different decision, but the result is clear and has to be accepted."
This is the second time in two years the German public has turned down the chance to host an Olympics after residents scuppered Munich's bid to host the 2022 Winter Games by voting 'No' in the 2013 referendum in southern Germany.
The Hamburg result is another body blow for the German Olympic Sports Confederation (DOSB), who had opted to back the Hanseatic city over the capital Berlin.
The recent terror attacks in Paris and concerns about the massive costs of hosting an Olympics had played on voters' minds as 64 percent of Hamburg residents had said they were in favour of an Olympic bid in a poll in September.
"This is a body blow and a set back for sport in Germany," said DOSB president Alfons Hoermann.
"We had set out with Hamburg to give sport in Germany a new perspective, but the next generation will now not get this chance.
"Our goal now is to develop sport in Germany without the vision of an Olympic Games, but on this path so far, we have only tended to meet an opposing head wind."
Michael Vesper, the DSOB's chief executive officer, said Germany may not be able to bid for an Olympics for decades.
He blamed the 2006 World Cup scandal, Paris' terror attacks and uncertainty caused by the refugee crisis for the defeat.
"We live in a democracy, there is no easy solution and we have to accept that," said Vesper.
"We have to take our time analyse the 'No' answer and maybe conduct a survey.
"The first response is to say that Germany won't be able to bid for an Olympics for decades, but we'll have to think about it.
"The German FA (DFB) and FIFA scandals, doping in athletics, refugees and the terror (in Paris) were all a factor, the voters' attention was drawn elsewhere.
"I don't want to make any excuses at the moment, but all these things certainly hampered the decision.
"The decision has been made and the project is now done.
"I am very sad, I think the decision is the wrong one, but I have been in politics a long time and I can only say, 'life goes on'."
"Having followed the discussions in Germany over the last weeks, this result does not come as a complete surprise," said a spokesman for the International Olympic Committee (IOC).
"With this decision a great opportunity for the city, the country and the sport in Germany is lost."
The spokesman said Hamburg would miss out on about €1.7 billion in IOC investment while the city had estimated it would need €1.2 billion euros.
"It is understandable that the citizens of Hamburg are very sensitive with regard to the not yet finalised financing plan of the candidature. Even more so in a situation in which Germany has to manage a historic challenge with a high number of refugees coming into the country."