German conservative Mueller, 69, who served a five-year posting as head of the powerful department responsible for church doctrine, the Congregation for the Doctrine of the Faith (CDF), had clashed with the pope over key reform issues.
READ ALSO: How rich is the Catholic Church in Germany?
He was one of several cardinals who questioned Francis's determination for the Catholic Church to take a softer line on people traditionally seen as "sinners", including remarried divorced people who want to take communion.
Mueller had also been caught up in the controversy surrounding the church's response to the clerical sex abuse scandal after his department was accused of obstructing Francis's efforts to stop internal cover-ups of abuse.
"In space of three days, two leading Vatican cardinals out of their posts," said Vatican watcher Christopher Lamb, after Vatican finance chief George Pell was charged with historical sexual assault this week.
The Vatican said Mueller's five-year term would not be renewed and he would be replaced by CDF secretary Archbishop Luis Francisco Ladaria Ferrer, a 73-year-old Spaniard.
Ladaria was appointed to the CDF by former Pope Benedict in 2008, and was asked last year by Francis to head up a new papal commission studying the possibility of having women deacons in the Church.
'Neither angel, nor pope'
Francis may not have liked Mueller's "excessive media exposure" and "interventions... that almost always sounded like he was distancing himself from the pope", Vatican expert Andrea Tornielli wrote in the Vatican Insider.
The German was dragged into the row over Francis's attempt to shift Church attitudes after the pope intimated last year that some believers who have remarried should be able to take communion.
Traditionalists were horrified; Roman Catholic marriage is for life, so divorcing your first partner and marrying someone else is considered adultery.
Four conservative cardinals accused the pope of sowing confusion and publicly demanded an answer to "doubts" about family guidelines Francis published in April. The pontiff has yet to respond.
Mueller said the cardinals were within their rights to challenge the guidelines and in February said marriage was a "sacrament, and no power in heaven or on earth, neither an angel, nor the pope... has the faculty to change it".
In March a prominent church reform group called for Mueller's resignation after accusations that senior officials had wilfully ignored Francis's decision to create a new tribunal to judge bishops who cover up sexual abuse.
Irish survivor of abuse Marie Collins, who quit the pope's commission on the protection of minors in disgust, singled out Mueller's ministry, which is in charge of the clerical abuse dossier.
The German cardinal retorted in an open letter that the tribunal had only been a "project" which Vatican departments felt would needlessly duplicate initiatives already in place to deal with wayward bishops.
Mueller is six years short of the traditional retiring age. The Vatican did not say what his next appointment might be.
His dismissal comes at the end of a turbulent week in the heartland of the Roman Catholic faith, following the charges of sexual offences brought against the Vatican finance chief on Thursday.
The Australian cardinal, 76, was granted a leave of absence by the pope to defend his name after he became the most senior Catholic cleric to be charged with criminal offences linked to the Church's long-running sexual abuse scandal.