An exclusion zone remains in place at the Surry Hills building, with the unstable walls posing a threat to public safety, as excavators clear rubble at street level after Thursday's devastating blaze.
Heavy demolition machinery was expected to arrive later on Monday evening before taking apart what remains of the seven-storey, heritage-listed brick building on Tuesday.
Fire and Rescue NSW Superintendent Adam Dewberry says the demolition will start as soon as the light allows.
"The whole thing is about getting in as early as possible," he told AAP.
Safety remained a priority after concerns more parts of the structure would collapse over the weekend, he said.
One of the factory's walls rained bricks on Randle Street as the inferno raged on Thursday evening, one of the largest seen in central Sydney for about a decade.
Firefighters then warned another wall moved slightly inwards on Friday night, highlighting the importance of the exclusion zone affecting neighbouring apartments.
At least 100 people living in the area have registered as being displaced since the fire lit up the sky, closing streets and diverting public transport.
Police are appealing for anyone who witnessed suspicious activity in the vicinity in the lead-up to the fire to make contact.
Teenagers spotted running from the building shortly after the blaze started handed themselves in on Thursday, with officers hoping three or four more teens present to police.