McCarthy received 216 votes, one shy of the number needed for a victory, as a small faction of right-wing hardliners held out despite his promise of spending cuts and other concessions that had won over many of their colleagues.
McCarthy had predicted victory earlier on Friday. "It's going to happen," he said after picking up support in two afternoon votes.
In hopes of securing enough votes to win, he had called an hours-long recess for the chamber, with lawmakers returning at 10pm ET (2pm AEDT Saturday) for another round of voting.
The result left the House rudderless after a week of repeated votes that highlighted Republicans' divisions and raised questions about their ability to govern. The chamber has not seen this level of congressional dysfunction since 1859.
It was not immediately clear what Republicans' next step would be, but they will need to call additional votes through the night or try again in the coming days.
The caucus could turn to another potential leader, such as McCarthy's No. 2 Steve Scalise or popular conservative Jim Jordan, who was repeatedly nominated as a candidate by hardliners this week.
Republicans' weaker-than-expected performance in November's midterm elections left them with a narrow 222-212 majority, which has given outsized power to the right-wing hardliners who have opposed McCarthy's leadership.
They accuse him of being too open to compromise with President Joe Biden and his Democrats, who also control the US Senate.
Some say they want a leader who will be ready to force government shutdowns to cut spending.
That raises the possibility the two parties would fail to reach a deal when the federal government comes up against its $US31.4 ($A45.8) trillion debt limit this year.
A lack of agreement or even a long stand-off risks a default that would shake the global economy.
