Under the new legislation, the field for gender can be left blank on birth certificates, effectively creating a category for indeterminate sex in the public register.
Activists promoting the rights of so-called intersex people said they hoped the creation of a third gender option would open the door to broader changes that would limit genital surgery on newborns with both male and female characteristics.
"It's a first, important step in the right direction," Lucie Veith, an intersex person from the northern German city of Hamburg, told AFP.
But Veith said leaving the gender undefined on birth certificates was never the main lobbying point for the Association of Intersexed People in Germany,
or others in the intersex community.
"That we forbid cosmetic genital surgeries for newborns, that is our first demand," said Veith, who leads the organisation.
The organisation calls for a ban on medically unnecessary surgeries until the child turns 16, so the intersex individual can decide for themselves whether to live as a man, woman, or neither.
The new German law is intended to remove pressure on parents to quickly make a decision about controversial sex assignment surgeries for newborns, but advocates it does not go far enough.
"The surgeries are likely to continue in Germany," said Silvan Agius, policy director at ILGA Europe, a lesbian, gay, bisexual, trans and intersex rights group.
Agius noted even prior to having the choice of not recording a baby's gender, already had the option of saying "'No, thank you very much, I don't want any surgery until my child can choose his or her gender.'"
Agius and other advocates fear that the new law would do nothing to change a society that operates largely on a gender binary with facilities such as separate male and female public toilets.
"There could be many other laws that could follow it and make it implementable and good," Agius said.
"My point is that if it remains as is…then it's greatly deficient."
Experts estimate one in 1,500 to 2,000 births result in a baby of indeterminate gender or both male and female gender features.
Surgeries such as clitoral reductions, removal of undescended testicles, and vaginoplasty, or the surgical creation of a vagina, can be performed to physically assign a gender.
According to a 2012 European Commission report on the topic, surgeries are performed on intersex babies and infants in many European countries without adequate informed consent by the patients.
The report also found that many intersex-born people are angry these surgeries were performed without their consent.
In 2009, a regional court in Cologne in western Germany awarded more than €100,000 ($135,130) in damages to an intersex person who was raised as a boy and whose uterus and womb were removed as a teenager.
The court found that the doctor had "violated (the person's) health and self-determination," according to the EU Commission report.
"Efforts to produce an unambiguous body impose circumstances upon the child that it may not want," Hertha Richter-Appelt of the University Medical Center Hamburg-Eppendorf told the online edition of German news magazine Der Spiegel.
But not all people who undergo these procedures as children are unhappy as adults, she told the magazine.
The Council of Europe for the first time last month addressed the issue of genital surgery for intersex individuals.
Its parliamentary assembly adopted a resolution calling on member states to study the prevalence of "non-medically justified operations" that may harm children and take steps to "ensure that no-one is subjected to unnecessary medical or surgical treatment that is cosmetic rather than vital for health during infancy or childhood."