In the ruling, the judge said that Friedhelm Adolfs' eviction was not carried out through due process, and cigarette smoke was not a reason to force someone out of an apartment without notice.
The case now has to be re-opened and the reasons for his eviction investigated to see if they were warranted, but for now, Adolfs can stay in his home.
In 2013, Adolfs was ejected from his Düsseldorf apartment because neighbours complained of the smell of cigarettes that penetrated the hallway of their building.
Adolfs' landlord said that the long-time smoker didn't properly ventilate his apartment - a particularly German clause in many rental agreements - and was thus in violation of his contract, warranting the immediate eviction.
The judge in a lower Düsseldorf court ruled in the landlord's favour, saying that Adolfs should take every measure to reduce the smell of cigarettes, including regularly emptying out his ash trays.
The 76-year-old smokes arond 15 cigarettes a day, he said. He called the apartment in Düsseldorf home for 40 years and even served as the buidling's caretaker for 30 of those.
He hoped to continue living there because its where he lived with his wife - who also smoked - until her recent death of cancer.
Through his plight, Adolfs has become a hero for cigarette smokers in Germany who say they are being forced outside more often.
The passionate smoker said that this court case wasn't just about him, "but for the community at large".
In January, the BGH ruled that municipalities could have the right to limit the times smokers can light up on their balcony after a couple sued their cigarette-smoking neighbours for stinking up their afternoon tea time.
In that decision, the judge declared smoking to be a "substantial nuisance" that could be regulated.