Volker Sieradzki was not there in the morning, as the diggers moved in to destroy the castle he spent eight years - and around €25,000 - building out of clay, wood and stone.
"No-one used to come and complain before," the stung 45-year-old told the Badische Zeitung newspaper. "Everyone knew about it."
He said he invested all his profits from his vineyard in the area into the castle, which he built to attract school children to his herb garden.
He had been intending to build a second and a third castle, plus a solar power unit, and turn the whole area into a nature adventure park.
But all those plans came to nothing on Wednesday. "It's bitter," said local district councillor Matthias Schimpf, but added that the castle had been built illegally and represented a safety risk for anyone who went inside.
"You have to respect his achievement," commented Rainer Burelbach, mayor of the nearby town of Heppenheim. "But it's still illegal."