Several dozen leftist protesters shouted "Nazis out" while counter-demonstrators responded with "Everyone hates Antifa" (the anti-fascist movement), DPA news agency reported.
The protests were sparked by leading AfD member Bjoern Hoecke's visit to the annual fair to attend the presentation of a book by the hard-right publishing house Antaios.
Elsewhere at the trade show, a reading by two right-wing authors had to be cancelled because it was drowned out by protesters, DPA said.
The tumult comes just three weeks after the Alternative for Germany party (AfD) won its first seats in the federal parliament after taking nearly 13 percent of the votes in a national election.
Hoecke is one of the AfD's most divisive figures. He made headlines earlier this year when he said Germany should stop atoning for its guilt over World War II.
Organisers of the book fair, the world's largest publishing event, issued a statement condemning "the targeted provocations" between "left- and right-wing groupings".
"The Frankfurt book fair thrives on diversity of opinion and is a space for free dialogue," it said.
The presence of far-right publishers at this year's fair has been controversial, with critics accusing organisers of giving a platform to rabble-rousers.
The small publisher Antaios, which describes itself as "new right", earlier this week complained that some of its books were stolen and others smeared with toothpaste and coffee in apparent protest actions.
It is the publishing house behind the bestselling book "Finis Germania", Latin for "The End of Germany", which news weekly Der Spiegel described as "extremely right-wing, anti-Semitic and historically revisionist".