German engineer Edgar Raupach was kidnapped on the outskirts of Kano in January. AQIM said last week it was holding the German and that it wanted to swap him for a jailed Muslim woman, a private news agency in Mauritania said.
A video obtained by the ANI agency and seen by AFP showed Raupach, his hands tied behind his back, surrounded by masked gunmen.
Germany has confirmed one of its nationals was kidnapped in northern Nigeria, and the German construction company Bilfinger Berger has said he is one of their employees.
Four of the suspects were arrested in a raid on a supermarket in the northern city of Kano owned by the Mauritanian on Thursday, while the fifth was held in a separate raid, the sources said.
"Guns and a laptop were recovered in the store and the documents found in the computer, including an AQIM operation manual, showed that the suspects are linked to AQIM and were involved in the kidnap of the German engineer in January," one of the sources said in describing the supermarket raid.
AQIM is the abbreviation for Al-Qaeda in the Islamic Maghreb, the extremist group's north African arm.
It has not been known to operate directly in Nigeria, though some have suggested links between AQIM and Nigerian Islamist group Boko Haram.
"The arrests are an important lead that could help in resolving the kidnapping of the German," one of the security sources said.
The initial raid on the store was carried out on suspicion that the suspects were tied to Boko Haram, but investigations after the arrests led authorities to suspect they were linked to AQIM and the kidnap, the sources said.
The second source, speaking of the Mauritanian, said "he turned the upper floor of his store into a hideout where he housed his three Nigerian accomplices."
Nigerian authorities have come under intense pressure over the kidnapping as well as violence blamed on Boko Haram. They also faced criticism after a failed bid to rescue an Italian and a British hostage earlier this month.
The British and Italian hostages were killed by their captors before they could be rescued in a joint operation with British security forces, authorities said.
Nigerian authorities blamed the kidnap of the British and the Italian on a faction of Boko Haram, which had not been previously known to carry out abductions. A purported Boko Haram spokesman denied any involvement.
A security source however had offered an explanation that implicated the group by association, alleging that the mastermind was a man named Abu Muhammad who was affiliated with AQIM and Boko Haram.
The source said the kidnappings were aimed at collecting ransoms which could be used to finance Boko Haram activities, and that in return Abu Muhammad would be given security cover by the group to carry out further abductions.
Abu Muhammad died in custody after being wounded in a raid which led to his arrest, Nigerian authorities have said.
There has been speculation that criminal groups may be seeking to profit from the security situation in northern Nigeria, where authorities have been unable to stop scores of bombings and shootings blamed on Boko Haram.
Nigeria, Africa's most populous nation with 160 million people, is roughly divided between a mainly Muslim north and predominately Christian south.