With 25 per cent of the vote counted, the coalition was on 35 per cent of the national primary vote to Labor's 31 per cent.
The Greens were sitting on 12.6 per cent of the primary vote, while independents held just over six per cent.
The most likely outcome on current trends is a Labor minority government.
As vote counting continued, the House of Representatives crossbench could be as many as 15-strong including four new Greens MPs in addition to Melbourne MP Adam Bandt.
Officials figures showed Labor incumbents trailing in Gilmore and Fowler, while Liberals were behind in Wentworth, North Sydney, Chisholm, Goldstein, Brisbane, Mackellar, Higgins, Reid, Kooyong, Robertson, Ryan, Boothby, Sturt and Deakin.
Labor could lose the Brisbane seat of Griffith to the Greens.
Labor frontbencher Jim Chalmers said a majority Labor government was "still possible", but he did not think the coalition could retain power.
"There are a lot of results where the second place getter and the third place getter is not absolutely determined," he told the ABC.
Labor colleague Tanya Plibersek said it appeared many people who had voted Liberal in the past had opted for independent or even Greens candidates in Sydney, Brisbane and Melbourne seats.
"They are people who have always voted Liberal in the past," she said.
"It is a big jump for someone who has always voted Liberal to make the jump to Labor.
"Those voters are trying to send a message that climate change is important to them, a national integrity commission with teeth is important to them, and equality with women."
Liberal and Labor strategists admitted well-funded independents and minor parties stretched the resources of the major parties during the campaign.
Finance Minister Simon Birmingham said it was a "difficult pathway" to holding on to power.
"We have had to fight a campaign across a broader range of electorates than customarily the case," Liberal campaign spokesman Simon Birmingham told the ABC.
"Not just the 'teal' challengers, probably a broader range of offensive measures in terms of seats that we could see potentially in the Labor Party range."
Labor deputy leader Richard Marles said his party did not "escape judgment" at the election.
Independent MP for Warringah, Zali Steggall, who appeared on track to hold her Sydney seat, said she expected more community independents would be elected.
"People are really frustrated," she said.
"Communities are turning to alternatives to the major parties."
Labor starts with 68 seats, plus notionally the new Victorian seat of Hawke, and required 76 seats for a majority in the 151-seat House of Representatives.
Scott Morrison's government went into the election holding 75 seats, having lost Stirling in WA in a redistribution.
Former Liberal minister Christopher Pyne predicted a Labor win, expecting the party will gain seats numbering in the "high 70s".
"If we win today, it will be very surprising," Mr Pyne told the Seven Network.
"Labor will win but, I think things very much tightened up in the last six weeks and what looked like a blowout … is obviously much closer."
Both major party leaders will be attending functions in Sydney on Saturday night.