The seven-storey, 111-year-old building was destroyed by fire last week in inner-city Surry Hills.
The building's roof and two of its street-facing walls partially collapsed during the blaze, which lit up the city skyline on Thursday night.
A 60-tonne long-armed excavator hammered at the remaining unsupported walls of the taller building on Tuesday, sending bricks crashing into the street as crowds watched from outside an exclusion zone.
A neighbouring three-storey building was also destroyed and fire crews will begin safety operations on its remains after demolition works are complete.
NSW Fire and Rescue Superintendent Adam Dewberry said the timeline of the demolition works would be reworked if crews encountered problems during the precarious operation.
The site sits above gas, electricity and water mains, which all needed to be considered as the building is pulled apart.
The derelict factory was known to regularly house rough sleepers and up to 15 people were in the building the night before the fire, with two still unaccounted for.
Once the area is safe, police will take possession of the site and cadaver-detection dogs will comb the area for evidence.
Arson squad detectives investigating the cause of the fire and spoke with a fourth teenage boy over the incident on Tuesday.
Three other teenage boys handed themselves in to police over the blaze last week.
No charges have been laid.
14°C