Bremen had only narrowly avoided relegation this season.
Having joined Werder in 1972 as a youth player, Schaaf had spent more than 40 years at Bremen as a defender, then youth team, reserve team and assistant trainer before taking over as head coach in 1999.
In a statement, the club said the 52-year-old has left "by mutual agreement" with immediate effect and while no replacement has been named, Schaaf will not be on the bench for Saturday's final league match of the season against Nuremberg.
Bremen only secured their status in Germany's top-flight last Saturday with a 1-1 draw at home to Eintracht Frankfurt.
"We have looked at things in the last few days and we've come to the conclusion that we want a fresh start," said Bremen's director of sport Thomas Eichin in a statement.
"We want to thank Thomas for everything he has brought to the club after more than 40 years as a player and coach for Werder Bremen."
Under Schaaf, Werder lifted the 2004 Bundesliga title, when his team won the double, and he guided them to two more German Cup triumphs in 1999 and 2009.
Schaaf was in charge for 479 Bundesliga matches, with 223 victories, 106 draws and 150 defeats.
After 5,119 days in charge, Schaaf is the third-longest serving coach in Bundesliga history behind ex-Freiburg boss Volker Finke, 5,843 days from 1991-2007, and former Bremen coach Otto Rehhagel, who spent 5,202 days in office between 1981-1995.