Ms O'Connor, one of two Greens MPs in Tasmania's lower house, was first elected in 2008 via a countback and became leader in 2015.
She held a press conference on Thursday in Hobart, saying she was quitting as leader and stepping down as member for Clark.
"This hasn't been an easy decision," she told reporters.
"I love my job. I love working in parliament and especially in an exciting, dynamic, balance-of-power parliament. But it is time to go.
"It is the right time for the Greens. We need renewal in leadership and the House of Assembly."
Her departure comes after the Liberal government was plunged into minority in May when two MPs quit to the crossbench over concerns with the state's deal for an AFL club and stadium.
There is speculation a state election will be held well before its June 2025 due date.
Ms O'Connor said she intended to seek pre-selection for the upper house seat of Hobart.
A recount will be held to replace Ms O'Connor in Clark, with her seat expected to go to Greens candidate Vica Bayley.
Ms O'Connor was appointed to two ministerial portfolios in 2010 under a Labor-Greens state government.
She said her proudest moment was voting for the Tasmanian Forest Agreement in 2013.
"That vote ... protected more than half a million hectares of some of our most beautiful forests," she said.
Former federal Greens leader Bob Brown described Ms O'Connor as a strong and wise defender of Tasmania's future, particularly its wild and scenic beauty, wildlife, Aboriginal heritage and visitor hospitality.
"She has been a champion for social justice and was the most innovative minister for housing Tasmania has seen," he said in a statement.
"Cassy was the real opposition leader in Tasmania, on a whole range of social justice and environmental issues in which Labor lines up with the Liberals."Â
Rosalie Woodruff will be acting state Greens leader.