Mr Shaw, who is also climate change minister in Jacinda Ardern's Labour-led government, failed to meet an 75 per cent satisfaction threshold among party delegates.
The party's co-leader since 2015 is a former consultant and is considered too moderate by a section of the party's grassroots.
He polled 70 per cent support of delegates, indicating broad support, but not enough to avoid triggering a spill of his position.
Mr Shaw admitted his shock in the aftermath of the vote, pledging to consult his partyroom and take soundings before deciding on his political future.
On Monday morning, he told Radio NZ he had would seek a return "because I'm not done".
"The climate crisis is unabated and we have a lot more work to do as a country there," he said.
Under party rules, a ballot will be held at week's end.
The Greens have a co-leadership model, with Mr Shaw's co-leader Marama Davidson going unchallenged at the AGM and continuing on.
It remains to be seen whether any others from the 10-strong partyroom, or outside parliament, will challenge Mr Shaw.
"I honestly don't know," Mr Shaw said.
Chloe Swarbrick, the 28-year-old Auckland Central MP, is considered most likely to beat Mr Shaw - if she chooses to run.
The next-gen political star toppled former minister Nikki Kaye and defied Labour in its 2020 landslide to win her electorate in a stunning upset last election.
"Chloe is an astonishing political talent. Anyone with any political nous can see that, but it's up to her to make that announcement if she wants to," he said.
The Greens enjoy two ministerial positions in Ms Ardern's government despite Labour's majority in parliament.
Ms Ardern granted the minority party's co-leaders the ministries after the 2020 election as a nod to their supportive role during her first coalition government, from 2017-2020.
New Zealand's electoral system makes coalitions the norm, and polls suggest Labour will need the Greens to have any shot of returning to office after the 2023 election.
Ms Ardern said Mr Shaw "does a fantastic job" and would stay in the job whether a leader or not.
"James Shaw as the climate minister within our government has introduced policy changes that are first in the world," she told TVNZ, citing climate reporting and work to price emissions from agriculture.
"Do we have a lot of work to do as a nation, as a world? Yes, of course.
"Does he often want to go further? Yes. He's a member of the Green party. I would expect that. But that hasn't ... diminished what he has done."
Mr Shaw said he intended to stay on regardless of the leadership role.
"My primary concern is to stave off the climate crisis and to stop it from getting any worse. I will find any route to achieve that outcome," he said.