The pair, Ignace Murwanashyaka, 46, and Straton Musoni, 48, are the leader and deputy leader respectively of the Democratic Liberation Forces of Rwanda (FDLR), German federal prosecutors said in a statement.
Murwanashyaka, who lived in Mannheim and directed his forces from there, was arrested in Karlsruhe. The authorities picked up Musoni near Stuttgart.
The FDLR is estimated to have 5,000 to 6,000 fighters, many of whom took part in the 1994 genocide in Rwanda before crossing into the neighbouring Democratic Republic of Congo.
"The accused are strongly suspected, as members of the foreign terrorist organisation FDLR, of committing crimes against humanity and war crimes," said federal prosecutors in a statement.
Murwanashyaka is additionally accused of being the ringleader of a terrorist organisation.
According to the statement, the paramilitary FDLR are suspected of "killing several hundred citizens, raping several women and plundering and burning several villages" between January 2008 and July 2009.
Around 800,000 people, mostly Tutsis and moderate Hutus, were butchered in the space of 100 days in Rwanda in 1994.
Prosecutors said that after a year-long investigation, they had amassed enough evidence to arrest the men.
Rwanda has voiced its concern over the FDLR's continued use of Germany as a safe haven, despite a UN Security Council resolution imposing travel and financial restrictions on the group's leaders.
The FDLR is almost entirely composed of ethnic Hutus opposed to the government of President Paul Kagame.