"Extremist groups, whatever their orientation, are gaining ground in Germany," said Interior Minister Thomas de Maiziere, presenting the 2015 report.
The security agency had "observed not just a rise in membership but also an increase in violence and brutality," he said in a statement.
Some 1,408 acts of far-right violence were recorded last year against 990 the previous year, said the service called the Federal Office for the Protection of the Constitution.
The sharp rise in racist hate crimes came as Germany took in a record number of more than one million refugees and migrants asking for political asylum, and as jihadist attacks in Paris and Brussels stoke terrorism fears in Europe.
"The intensity of right-wing extremist militancy started in early 2015 and increased steadily - from threats against politicians and journalists to arson attacks on asylum seeker shelters and attempted killings," said the report.
There were 75 arson attacks against refugee shelters in Germany, five times more than 2014.
The report said that online "social networks play an important role in agitation and radicalisation", as uninhibited hate speech dehumanises minorities and fuels real-world violent crime.
Far-left acts of violence - often targeting far-right activists or police - also rose sharply, to 1,608 violent offences from 995 the previous year, said the report.
The worst spate of attacks came during mass protests in March against the European Central Bank in Frankfurt, when rioters rampaged through streets and set ablaze police cars.
The service also pointed to the rising threat posed by Islamists, estimating their number at about 10,000.
The report said it assumed that some jihadists and war criminals had entered the country with the massive refugee influx.
Other potential threats were posed by "self-radicalised" individuals, jihadist fighters returned from Syria and Iraq, and possible sleeper cells from militant groups, it said.
By late last year, more than 780 people from Germany had travelled to join jihadists in Syria and Iraq, although the number of departures was thought to have gradually declined.