Holmes has been battling a hamstring injury, suffered during the preliminary final, and was deemed too big a risk for Saturday's season decider against Sydney.
O'Connor was the Cats' medi-sub in their two previous finals victories over Collingwood and Brisbane.
Brandan Parfitt has been named as the substitute, having last featured in round 23.
It is heartbreaking for Holmes, who has enjoyed a breakout season.
The 20-year-old was one of the Cats' best players in the preliminary final before hobbling off early in the second half.
He moved well during a fitness test on Friday morning but sat out the team's light training run on the MCG that afternoon.
"Mark O'Connor comes into the line-up and Brandan Parfitt will be our sub," coach Chris Scott told the Seven Network.
"It's desperately unlucky for Max because he was so close.
"This was one of those calls where in my mind he's probably 98 per cent there but we just needed certainty for the team
"It's really hard on him. I think he probably could have done it but this was the right decision we think under the circumstances and Brandan and Mark O'Connor have been so good for us all year.
"We're just going to go with the fit guys."
Sydney have also had injury concerns during the week, but coach John Longmire insists Sam Reid (adductor) and Justin McInerney (ankle) will play.
Reid was substituted out of the preliminary-final win over Collingwood but McInerney managed to finish the game.
"(Reid) got some scans on Monday and that was really encouraging, so we were confident from that point," Longmire told reporters on Friday.
"He did some training on Thursday and did exactly what he needed to do.
"Our medical staff have been terrific, we were always confident that (McInerney) would be right.
"He finished off the game strongly last week so he's right to go."
Braeden Campbell, Harry Cunningham, Will Gould and dropped tall forward Logan McDonald were named as the Swans' emergencies.
McDonald was replaced by ruck-forward Hayden McLean, who will play his first game since round eight.