Protesters chanted anti-Israel and anti-Jewish rhetoric at a rally which started at Town Hall and ended on the steps of the Opera House on Monday night as the venue's sails were lit up in the colours of the Israeli flag.
NSW Premier Chris Minns approved the lighting of the sails to show solidarity with the Jewish community after the Islamist group Hamas invaded Israeli towns and massacred residents and festival attendees on Saturday.
Flares were lit outside the Sydney Opera House during a pro-Palestine rally.
The surprise attack and subsequent retaliatory shelling of Hamas-controlled Gaza have killed at least 1500 people.
Mr Minns led a host of prominent figures on Tuesday condemning the protest and the "shocking and abusive" comments shouted there, which included a chant of "gas the Jews".
But organisers Palestine Action Group defended their right to protest and said media coverage had focused on a tiny fringe of "vile anti-Semitic attendees"
Mostly young teens shouting "f** the Jews" were quickly condemned and asked to leave, they said in a lengthy statement.
"Long-standing Palestinian organisers and activists, Palestinian, Arab and Muslim elders attending the protest were disgusted and deplored by the action," it said.
The group, which promises a second rally on Sunday, drew a distinction between Judaism and the state of Israel, which they described as "a genocidal apartheid regime".
More than 1000 people attended a pro-Palestine rally outside the Victorian State Library on Tuesday evening.
Initial reports from Victoria Police indicated there were no major incidents or arrests despite a strong police presence.
Federal Opposition Leader Peter Dutton expressed concern for the safety of Australian Jewish communities, adding that the prime minister should call a national security meeting.
"There are stories coming out of Jewish communities where they're telling (kids) not to wear their school uniforms in public," he said.
NSW Police confirmed they were investigating the behaviour of those at the snap Opera House protest while defending their decision to not break it up.
No formal application to protest on the streets, which can take a week to process, was lodged.
"The police activities last night were not about condoning, supporting or facilitating - this was about providing a safe environment," Assistant Police Commissioner Tony Cooke said.
More than 1000 people attended a peaceful rally in Melbourne.
A Jewish man was arrested at Town Hall and had his Israeli flag confiscated to keep him safe, he added.
Mr Cooke said police spoke with members of the Jewish community on Monday about the risk of conflict if they went to the Opera House in the evening.
Their call to not attend showed good sense in the circumstances, he said.
Police and the government have pledged to consult with Jewish leaders to ensure communities could safely mourn those affected by the Israel attacks.
The premier meanwhile stood by his $10,000 decision to light the Opera House sails with the Israeli flag, comparing it to illuminating buildings in the colours of the Ukrainian flag after the Russian invasion.
"The absence of the flag on that national monument, given the entire world showed solidarity and sympathy with the Israeli community after that barbaric attack, would have been conspicuous," Mr Minns said.
He also defended the NSW government's support of Israel in the wake of terrible atrocities and said rallies at the Opera House and a night earlier in Lakemba were "largely in support of Hamas activity."
"It's not demonstrative of all members of the Muslim, Islamic or Palestinian communities," he said.
NSW Opposition Leader Mark Speakman said the anti-Israel scenes were simply unacceptable.
"Israel is not perfect, it is far from perfect, but it is a real example of a liberal democracy in the Middle East," he said.