This comes after a spike to 5.2 per cent in October resulting from half the population having been in lockdown battling the coronavirus.
The Australian Bureau of Statistics also said the number of people employed surged by 366,100 in the month.
"Todays jobs numbers are the clearest indication of what happens when you ease restrictions," Mr Frydenberg told reporters in Canberra as he handed down his mid-year budget review.
The review shows the unemployment rate is now expected to fall to 4.5 per cent in the June quarter next year and 4.25 per cent 12 months after that.
It also forecast economic growth of 4.5 per cent over 2021 and 4.25 per cent in 2022.
While Mr Frydenberg conceded uncertainties remain regarding the pandemic, he said Australia must learn to live with the virus.
"We're not going to see Omicron derail the recovery," he said.
Commentating on the latest jobs figures, ABS head of labour statistics Bjorn Jarvis said most people remained attached to their job through the Delta lockdowns and to a greater extent than was seen earlier in the pandemic.
"This job attachment meant, as restrictions were eased, many people were able to quickly return to work," he said.
The easing of COVID-19 restrictions in NSW and Victoria had a large influence on the national figures, with employment in both states increasing by 180,000 people and 141,000 people between October and November.
Speaking earlier on Thursday, Reserve Bank of Australia governor Philip Lowe said he expected the jobless rate to hit four per cent by the end of 2023.
Dr Lowe does believe that with COVID-19 lockdowns lifted, spending is bouncing back quickly, although he admits the outbreak of the Omicron variant presents a downside risk.
Even so he expects the positive momentum in the economy to be maintained through the summer, underpinned by the opening up of Australia, high rates of vaccination, significant fiscal and monetary support, and the strengthening of household and business balance sheets.
But he reiterated the RBA board would not increase the cash rate until actual inflation was sustainably in the two to three per cent target range, something he had previously not expected to occur before 2024..
"We are still a fair way from that point. In our central scenario, the condition for an increase in the cash rate will not be met next year," he told the CPA Australia Riverina Forum in Wagga Wagga NSW.
"It is likely to take time for that condition to be met and the board is prepared to be patient."
National Australia Bank economist Tapas Strickland noted Governor Lowe was again pushing back against financial market expectations of an interest rate rise next year.
"But interestingly (he) made little mention of 2024, which to us suggests the RBA is now seeing 2023 as being more probable than previously," he said.
