Merz, who has no previous experience in office, is set to become chancellor as Europe's largest economy is ailing, its society is split over migration and its security is caught between a confrontational US and an assertive Russia and China.
After the collapse of incumbent Olaf Scholz's coalition, Merz, 69, promised his government meant making Germany "present in Europe again, so that the world notices that Germany is being reliably governed again".
"Tonight we will celebrate, and from tomorrow we start working. ... The world out there is not waiting for us," he said on Sunday.
US President Donald Trump, whose ally Elon Musk had repeatedly endorsed the AfD during the campaign, cheered the conservative victory on Truth Social.
"Much like the USA, the people of Germany got tired of the no common sense agenda, especially on energy and immigration, that has prevailed for so many years," Trump wrote.
Following a campaign roiled by violent attacks for which people of migrant background were arrested, the conservative CDU/CSU bloc won 28.4 per cent of the vote, followed by the AfD with 20.4 per cent, said a projection published by ZDF broadcaster.
All of the mainstream parties have ruled out working with the AfD, which looks set to double its score from the previous vote and saw the result as just the beginning.
 "Our hand remains outstretched to form a government," leader Alice Weidel told supporters, adding "next time we'll come first".
Merz is heading into what are likely to be lengthy coalition talks without a strong negotiating hand. While his CDU/CSU emerged as the largest bloc, it scored its second worst post-war result.
It remains unclear whether Merz will need one or two partners to form a majority, with the fate of smaller parties unclear in a way that could jumble parliamentary arithmetic.
A three-way coalition would likely be much more unwieldy, hampering Germany's ability to show clear leadership.
Chancellor Scholz's Social Democrats (SPD) tumbled to their worst result since World War II, with 16.4 per cent of the vote share, and Scholz conceding a "bitter" result, according to the ZDF projection, while the Greens were on 12.2 per cent.
Strong support particularly from younger voters pushed the far-left Die Linke party to 8.9 per cent of the vote.
The pro-market Free Democrats (FDP) and newcomer Sahra Wagenknecht Alliance (BSW) party hovered around the five per cent threshold to enter parliament.
Voter turnout at 83 per cent was the highest since before reunification in 1990, according to exit polls. Male voters tended more towards the right, while female voters showed stronger support for leftist parties.
"A three-party coalition runs the risk of more muddling through and more stagnation unless all parties involved realise that this is the last chance to bring change and to prevent the AfD from getting stronger," Carsten Brzeski, global head of macro at ING.
A brash economic liberal who has shifted the conservatives to the right, Merz is considered the antithesis of former conservative chancellor Angela Merkel, who led Germany for 16 years.
Merz conditionally supports equipping Ukraine with longer-range Taurus missiles, a step Scholz's government shied away from, and sees Europe as firmly anchored in NATO.
The election came after the collapse last November of Scholz's coalition of his SPD, the Greens and pro-market FDP in a row over budget spending.
Lengthy coalition talks could leave Scholz in a caretaker role for months, delaying urgently needed policies to revive the German economy after two consecutive years of contraction and as companies struggle against global rivals.
It would also create a leadership vacuum in the heart of Europe even as it deals with a host of challenges such as Trump threatening a trade war and attempting to fast-track a ceasefire deal for Ukraine without European involvement.
Germans are more pessimistic about their living standards now than at any time since the financial crisis in 2008.
Attitudes towards migration have also hardened, a profound shift in German public sentiment since its "Refugees Welcome" culture during Europe's migrant crisis in 2015, that the AfD has both driven and harnessed.
