O'Connell, four years younger than the 34-year-old veteran Serb, and at 87 ranked 22 places higher, took one hour 41 minutes to claim the win.
After both players dropped their opening games the first set went with serve until O'Connell snagged a break in the tenth to take the set. He quickly went 4-1 up in the second, saving four break points in the fifth game, but was broken in the seventh to have his lead cut to 4-3.
However, the Sydneysider broke again immediately and saved another break point to seal victory.
The Australian will now meet fourth-seeded former top-10 player Nicolas Jarry. The Chilean, now world No.58, is on a poor run of form having won one of his last six matches.
O'Connell won their only previous meeting, in the opening round of last year's US Open.
There were also early victories on Tuesday (local time) for Marton Fucsovics, Damir Dzumhur, eighth seed Camilo Ugo Carabelli and qualifier Filip Misolic.
12°C