Merkel's Christian Democratic Union and its Bavarian allies fell to 38.4 percent from 44.5 percent at the last European election in 2004, ARD public television reported.
The second-placed Social Democrats (SPD) garnered only on 21.1 percent - a historic low for the centre-left party. The SPD, junior partners in Merkel's uneasy grand coalition, had expected to improve on 2004's 21.5 percent.
"This is disappointing, without question," Franz Müntefering, SPD chairman, said in Berlin.
With 99 seats out of a total of 736, Germany has the largest contingent of MEPs in the European Parliament. The vote in Germany was seen by some observers as a warm-up for general elections in September, when Merkel is running for a second term.
"Tonight we are 17 points ahead of the SPD. The CDU alone is stronger than the SPD and the Greens combined," CDU general secretary Ronald Pofalla said in Berlin.
Frank-Walter Steinmeier, the SPD's chancellor candidate, called the result "disappointing" but predicted that with turnout low, it would be a happier outcome for his party when Germany goes to the polls in less than four months.
"Around 42 percent of people voted. At the general election it will be twice that and I will definitely be commenting on quite a different election result," said Steinmeier, the current vice-chancellor and foreign minister.
The pro-business Free Democrats (FDP), Merkel's preferred coalition partners after September, were the biggest winner of Sunday's vote, with their score soaring to 10.6 percent from 6.1 percent.
The Greens were the third strongest party with 11.6 percent of the vote, compared to 11.9 percent in 2004. The hard-line socialist party The Left improved its score to 7.5 percent from 6.1 percent.
Story continues below…
Turnout in Germany was down slightly at 42.5 percent from the last vote in 2004, when 43 percent of voters bothered to head to the polling stations.
Merkel, who is a strong supporter of the EU's reforming Lisbon Treaty, also told Bild am Sonntag newspaper that it was up to politicians to get voters more interested in European elections.
"In a world with six and a half billion people, 80 million Germans have a better chance of being heard if they can speak as part of a community of 500 million Europeans. Perhaps we have to say this more clearly," she said.