Initial projections from the ARD television station showed around 58 percent of those who voted wanted to see Baden-Württemberg state continue to participate in the project. Voter participation on Sunday was at around 48 percent the station said - considered low in Germany.
The opponents needed to get a clear majority to force the state government to pull out of the multi-billion-euro project.
Winfried Kretschmann, the Green party state premier who was elected on a wave of frustration at the previous government's support of the project, conceded defeat, having himself voted against.
"The people have spoken," he said on SWR radio. "We will accept this vote. The whole state government will do so."
Yet he also called for Deutsche Bahn, which is leading the project, to be clear about costs.
"I do not want to allow myself to be put in the position where the building work has started, then the costs explode and I have to contribute more whether I want to or not," he said on ZDF television on Sunday evening.
He said the Bahn should commit itself to taking on any costs over €4.5 billion. "We will not pay more than our share of the €4.5 billion," he said.
The project involves rebuilding the city's main train station underground and turning it 90 degrees, as well as laying 57 kilometres of new tracks. The aim is to make the city a major European rail hub.
But opponents mounted massive protests against the project last year, calling it too expensive and unnecessary. In October 2010, more than 100 demonstrators were injured in a violent clash with police. This summer nine police officers were hurt in further clashes.
Lengthy talks between state officials, national rail provider Deutsche Bahn and Stuttgart 21 opponents have taken place. But officials ultimately said they were making a few adjustments while still going ahead with the project, after a “stress test” in July showed the project should continue.