The 35-tonne stone is currently part of the “Global Stone” circle in central Berlin's Tiergarten park, created by 79-year-old sculptor Wolfgang von Schwarzenfeld.
Despite von Schwarzenfeld wanting to bring conflicting nations together through what he calls a "world peace project", the sculptor is now in the middle of a squabble over what happens to the boulder, the Berliner Zeitung newspaper reported on Monday.
The first rock he installed there was the Venezualen one back in 1999. It comes from the Gran Sabana planes in the south east of the country, home to the indigenous Pemón people. They have now decided they want the whale-shaped boulder they call Kueka (meaning grandmother) back, saying it is sacred and should never have been removed.
“If the stone goes back to Venezuela then my whole idea would be destroyed. That troubles me immensely,” Schwarzenfeld told the paper.
While the argument has been rumbling on for some time, the Pemón have stepped up their protest over the past couple of weeks - a 50-strong group even gathered outside of the German embassy in Caracas, to demonstrate.
The Berliner Zeitung said Venezuelan government was starting moves to claim it back and had told German officials a delegation would head for Berlin soon to visit the rock.
The foreign ministry "is trying in talks with all parties to mediate an amicable solution", spokesman Andreas Peschke told a government news briefing.
"In order to facilitate a possible handing back of the stone and at the same time protect the interests of the artist, the foreign ministry has made relevant proposals for an amicable agreement," he said.
But Schwarzenfeld said the Pemón were being exploited by politicians. “If the stone had have had particular meaning to the Pemón then I absolutely would have had chosen another,” he told the paper.
Research by Bruno Illius from Berlin's Free University, suggests no evidence exists to link the giant boulder to the Pemón tribe's mythology the Berliner Zeitung said.
The stone was flown to Germany only after prolonged talks with heads of the Pemón tribe and local conservation officials. Schwarzenfeld told the paper he had a sheaf of documents proving its authorised departure.