Goalkeeper Raya made errors that led to two goals for Luton, leaving Arsenal on the cusp of dropping two points at Kenilworth Road.
However, with the clock having run past the six allotted minutes of stoppage time, Rice got on the end of Martin Odegaard's cross and glanced a header into the bottom corner for a victory that lifted Arsenal five points clear at the top.
"Those are the moments you look back on," Rice said.
"What a game. To score the winner is an honour."
Gabriel Martinelli and Gabriel Jesus scored either side of a Gabriel Osho strike for Luton as Arsenal led 2-1 at halftime.
Then came Raya's mistakes.
First, the goalkeeper was beaten in the air at a corner by Elijah Adebayo, who powered in a header to equalise for the hosts shortly after the break.
Eight minutes later, Raya allowed a shot by Ross Barkley to go under his body as he dived to his left, giving promoted Luton an unexpected 3-2 advantage.
Kai Havertz levelled in the 60th minute before Rice - who joined Arsenal from West Ham for more than $190 million over the off season - completed the comeback after relentless late pressure from the visitors.
Luton manager Rob Edwards said he was "immeasurably proud" of his players but "really flat".
"I don't think that has ever happened to me, conceding that late to lose a game," he said.
"I believe in this group and with the spirit and fight we are showing, we hope we can achieve something special."
In Tuesday's other EPL game, Hwang Hee-chan continued his fine scoring form to fire Wolves to a 1-0 win over second-from-bottom Burnley at Molineux.
The visitors had on Saturday ended a seven-game losing run with a thumping 5-0 win over fellow strugglers Sheffield United, but they were undone by defensive frailties against Wolves.
Hwang took advantage after sloppy play from Burnley's backline shortly before halftime, keeping his composure to roll the ball past keeper James Trafford for his ninth goal of a prolific season.
"The numbers are incredibly impressive," Wolves boss Gary O'Neil said of his in-form striker.
"Everything about him is impressive. Will he continue at this rate for the rest of the season? I hope so. If the team can keep performing.
"It's no fluke he has as many goals as he has, he's bought in fully to what we're trying to do."
- with PA
