The central banker announced his departure on Wednesday after seven years in the top role.
"I leave the role with consumer price inflation at target, and an economy in a cyclical recovery following the long period of COVID-related disruption," he said.
"I'm incredibly proud of the RBNZ's people, our work and the impact of our mahi (work) on all New Zealanders."
He is yet to explain his decision to walk away.
Mr Orr is a hugely divisive figure in New Zealand, blamed by many for exacerbating the boom-and-bust cycle of recent years.
The RBNZ slashed the official cash rate (OCR) from one to 0.25 per cent in March 2020 as the first lockdowns took hold, issuing guidance it would remain at that emergency low level for at least 12 months.
"We are doing nothing that any other central bank is not doing, we're very much strictly ballroom," he said in late 2020.
Combined with a large-scale asset buying of government bonds and wage guarantees from the government, the stability of the low-rate environment saw a mass plunge on housing, with average prices jumping 45 per cent in two years.
Further down the line, New Zealand slumped into a major recession last year, with a contraction of two per cent between March and September.
In November 2022, with headline inflation at 7.3 per cent and an election due five months later, he was reappointed for a second five-year term to the opposition's fury.
Nicola Willis, then the National party's finance spokeswoman, and now the finance minister, called his reappointment "a serious mistake".
"(Mr Orr) signed off on an extraordinary programme of money printing and cheap lending that pumped tens of billions of dollars into the economy," she said at the time.
"That program directly contributed to house prices rising 28 per cent in one year, inflation rising to a 32-year high, and record bank profits. New Zealanders now suffering through a cost of living crisis are owed some answers."
On Wednesday, she offered just a brief statement on his departure.
"I wish him well for the future," she said.
Wastewatch group the Taxpayers Union also cheered his departure, which was "not before time".
"Orr's work caused the single worst economic downturn in New Zealand in over three decades. He was far too slow to react as inflation increased, and far too slow to lower interest rates as inflation fell," spokesman James Ross said.
Deputy Governor Christian Hawkesby is acting as governor until March 31, when Ms Willis can - on recommendation from the RBNZ board - appoint a temporary governor for up to six months.
In an intriguing piece of timing, Mr Orr leaves a day before Wellington hosts a major international conference to mark "35 years of inflation targeting", with former US reserve chair Ben Bernanke giving the keynote address.