The police had been called to a family home, where the caller had explained that an argument was taking place between a married couple.
According to the police report, the man opened the door for the police and immediately tried to attack the officers with a kitchen knife.
When several requests for him to put the knife down and the use of pepper spray had failed to stop him, one of the officers shot him twice.
The man was hit in the stomach and died later in the night in hospital.
The special police commission for Westpfalz has taken over the investigation.
It's extremely rare for police to kill someone using their sidearms in Germany, with just nine cases in 2013 according to Dr. Clemens Lorei of the Hesse Police and Administrative University.
In 2012, police reported firing just 36 shots at people.