The report - Fault lines: An independent review into Australia's response to COVID-19 - led by former public servant Peter Shergold, also found some lockdowns and border closures weren't necessary and schools should have remained open.
It warns governments against the perils of overreach when dealing with future health crises.
"For many of us, the story of COVID-19 will be one of inconvenience," the private sector-funded report released on Thursday said.
"For others, COVID-19 will be a story of trauma, isolation and terrifying uncertainty."
Politically driven health orders and excessive lockdowns failed to protect the old, ignored the young and abandoned disadvantaged communities, the report said.
But Victorian Premier Daniel Andrews, who oversaw one of the longest lockdowns globally - with Melbourne clocking 262 days under stay-home orders - pushed back against the report's findings, which he admitted he hadn't read yet.
"There was nothing academic about COVID-19," he told reporters.
"There's nothing academic about the fact that we didn't have any vaccines".
Meanwhile, Queensland Premier Annastacia Palaszczuk also defended her policies of effectively isolating the state for months.
"We made the decisions in the best interests of Queenslanders and it kept Queenslanders safe," she said.
"I stand by our world leading result when it comes to the number of lives that were lost compared to other jurisdictions and what happened around the world".
South Australian Premier Peter Malinauskas - who was opposition leader when the pandemic began - was more understanding, saying government departments should examine the review's recommendations.
"To the extent that this review could help to inform governments about future responses, that is useful," he said.
"Every government around the country, Liberal and Labor, were trying to deal with very difficult circumstances".
On lockdowns, Prof Shergold said they were a lost opportunity authorities to get on with the job of logistically managing a pandemic from dispatching personal protective equipment to keeping the economy afloat.
"Lockdowns are a part but they can't be the key way of dealing with a health crisis," he said.
While school closures were probably the right decision when the virus was little understood, "it was wrong to close entire school systems, particularly once new information indicated that schools were not high-transmission environments", the report said.
"For children and parents (particularly women), we failed to get the balance right between protecting health and imposing long-term costs on education, mental health, the economy and workforce outcomes."
As well, COVID-19 rules were often enforced in ways that lacked fairness and compassion.
"Business people were often allowed to travel across borders whilst those wanting to visit dying loved ones or newborn family members were not afforded a similar opportunity," the report said.
One of those consulted in the report was western Sydney GP Jamal Rifi, who became a prominent advocate for the vaccination of migrant communities during the uneven rollout of vaccines last year.
He spoke of the punitive aspects of curfews across 13 LGAs in southwest and western Sydney, enforced by the NSW government.
"The curfews were very damaging because it facilitated the spread of the virus rather than limiting it," Dr Rifi told AAP.
"In Sydney, it was a tale of two cities. There were different rules for western and eastern suburbs ... which accentuated societal fault lines."
Prof Shergold advocated for a more localised approach that would have drawn on the skills of local governments.
"You're dealing with a national pandemic but you're interfacing more directly with local governments, community and business leaders, not just in making decisions but in communicating them," he said.
"The danger in doing that is that it can undermine public trust with people in Bankstown feeling unfairly treated compared to people in Bondi.
"The single biggest failure was not sufficiently planning for the fault lines in society."
The report also recommended the establishment of an Australian centre for disease control, more transparency around decision making and better collaboration across state lines.