A fire in the structure has been smouldering for the past two weeks due to 800 tonnes of grain inside fermenting in the hot weather.
The government said the fire expanded after flames reached nearby electrical cables.
The fire and the dramatic sight of the smouldering, partially blackened silo is reviving memories and in some cases trauma for survivors of the gigantic explosion that tore through the port two years ago.
Experts say part of the structure is leaning and in danger of tipping over.
Prime Minister Najib Mikati, who oversees a caretaker government, ordered firefighters and civil defence volunteers to step back on Friday for their safety.
Civil Defence volunteer Youssef Mallah told the Associated Press that they are still at the port but have been ordered to stay far from the silo.
The August 2020 blast was caused by hundreds of tonnes of ammonium nitrate, a highly explosive material used in fertilisers, that had been improperly stored for years at the port's warehouse.
The tall structure withstood the force, effectively shielding the western part of Beirut from the explosion that killed more than 200 people, wounded more than 6000 and badly damaged entire neighbourhoods.
The government said experts had warned that trying to put out the fire with water could worsen it due to humidity but the Interior Minister on Thursday ordered firefighters to try to contain the fire with water anyway.
Over the weekend, the Lebanese army sent a helicopter to try to douse the fire with water as well.
Another minister warned last week that the situation at the port was "tricky and complicated" and warned of a collapse.
Emmanuel Durand, a French civil engineer who volunteered for the government-commissioned team of experts said the north block of the silo was "on path for catastrophic failure" and that a collapse was inevitable at this point.
He told the AP the grain fire could not be extinguished by water, which actually fuels the process of fermentation and can make the silo tilt faster.
Durand, who is based in Switzerland, has been monitoring the silo for two years via sensors, and sending warnings to the government.
The silo that has been tilting at no more than 0.5 millimetres a day two weeks ago is now moving at a "cruise speed" of 2 millimetres an hour, he said.
Earlier this year, the Lebanese government moved to demolish the silo but was forced to suspend the decision following protests from families of the blast's victims and survivors, who have yet to see justice served.
They argue that the silo may contain evidence useful for a judicial probe.
Some also say the silo should stand as a memorial for the tragic incident.
The judicial probe revealed that a range of government officials knew about the dangerous substance that had been stored at the port for years but did not take meaningful action to remove it or dispose of it.
No officials have been yet been convicted.
Those implicated have lodged legal complaints against the judge leading the investigation.
A rally organised by the Hezbollah group against the judge leading the probe turned deadly last year, with six people killed and dozens wounded.
