Liverpool Mayor Ned Mannoun says the resettlement of the families in western Sydney could involve re-traumatising recently arrived refugees such as Yazidis and Assyrians fleeing IS violence from Iraq and Syria.
"You would not put a victim of a crime next to the perpetrator or the person who is an accessory to the perpetrator," Cr Mannoun told AAP.
"If the government has an obligation to bring them into the country then that's fine but surely common sense says you do not put these people in these neighbourhoods.
"I have consulted with members of my community and they're saying it is insane."
Four Australian women who have been in the al-Roj camp in Syria since the fall of ISIS arrived in Sydney along with 13 children last month following a secret operation.
Cr Mannoun, Fairfield Mayor Frank Carbone and Campbelltown Mayor George Greiss are expected to receive a security briefing from Ms O'Neil on Friday.
The meeting comes after the three wrote to Prime Minister Anthony Albanese outlining their concerns.
"We don't want to politicise the matter but this affects the social cohesion and beautiful tapestry of western Sydney," Cr Mannoun said.
He said hundreds of ISIS family members could be resettled in the three culturally diverse local government areas in question and he is worried whether refugees living within them would feel safe.
"Wherever the refugees who fled ISIS from Iraq and Syria settled in, just don't put them near there. Put them away from them at a distance.
"Let's minimise the interaction with the people who fled. Let's have some humanity for them and the trauma that these people have gone through."
Federal Labor minister Jason Clare noted that the former coalition government also repatriated people in a similar situation back to Australia, and to his electorate of Blaxland in Sydney's west.
"(Former home affairs minister) Peter Dutton didn't come to Bankstown, he didn't tell me anything about it and left my community totally in the dark," he told Nine's Today Show on Friday.
Mr Clare said he could only assume the women who were recently returned had renounced ISIS.
"ISIS was a rubbish terrorist organisation which went about killing innocent people," he said.
"Nobody ... in Australia would have a good word to say about them and I would assume these people would not either."
Opposition deputy leader Sussan Ley said the people of western Sydney had been disrespected but it was good Ms O'Neil was finally meeting with community leaders.
"Please listen to the people of western Sydney," she told Nine.
ISIS grew out of the political vacuum left in Iraq and the civil war in Syria, declaring a "caliphate" in 2014 and ruling with an iron fist over swathes of territory between both countries.
ISIS attracted thousands of foreign fighters, including dozens of Australians, who committed atrocities and genocidal acts in some instances, according to the United Nations, towards the Gnostic-worshipping Yazidis and Christian Assyrians.
Federal police commissioner Reece Kershaw confirmed on Tuesday the force was investigating whether the repatriated Australians had broken any laws prohibiting travel to select war zones, including Syria.
He said the 17 returned women and children, who have settled in Victoria and NSW, had co-operated with police.
The government says surveillance and restrictions will help mitigate any potential threat of extremism from the returned women and children.
