Hundreds of firefighters are battling three uncontrolled bushfires, one at Dimboola and Wail near the South Australian border and two in the Grampians National Park.
The Dimboola fire was sparked by dry lightning at Little Desert National Park on Monday and has grown to more than 65,000 hectares.
Residents were told to evacuate as the fire reached the outskirts of town but some at Dimboola District Hospital and the town's aged care facility didn't get out in time.
About 190 people and their pets sought refuge at a makeshift relief centre in Horsham, with 27 staying overnight.
The fire was downgraded to watch and act on Tuesday afternoon.
"I'm incredibly thankful that no lives have been lost from this fire, and we have no reports of injuries either," Emergency Management commissioner Rick Nugent told reporters at the State Control Centre in Melbourne.
Firefighters say a blaze spread from public land to private property and has grown to 65,000ha. (HANDOUT/STATE CONTROL CENTRE)
Popular wedding and conference centre Little Desert Nature Lodge was destroyed in the blaze
"The lodge just exploded, it's totally gone," Hindmarsh Shire Council mayor Ron Ismay told AAP.
"It's very sad, it was a great venue for people of the area."
Mr Nugent confirmed a farm house west of Dimboola was lost and another near a river may have been damaged.
Dimboola resident and Hindmarsh Shire Council deputy mayor Chan Uoy said the smell of smoke lingered in the town as he and other locals returned home on Tuesday afternoon.
Nearly 200 people sheltered at a makeshift relief centre in Horsham as fires threatened their homes. (Diego Fedele/AAP PHOTOS)
He had just 30 minutes notice to evacuate late on Monday in hot, dry winds and smoke haze.
"It's like this massive cloud covering following you, it's quite dramatic," he said.
At the relief centre in Horsham, municipal recovery manager Mandi Stewart said locals were very relieved the blaze wasn't more severe.
"It looked fairly bad," Ms Stewart said.
"The towns been pretty all right, there's one area sort of on the south of the town that there's still no access to."
Little Desert National Park and Wail State Forest remain closed and the rain line between Victoria and SA has been suspended.
A second emergency warning issued for Strachans, Victoria Point and Victoria Valley in the south western part of Grampians National Park has been downgraded to watch and act.
Hours later, watch and act alerts were issued for a blaze at the Wallaby Rocks Road area near Zumsteins in the park's north affecting Brimpaen, Laharum, Wartook, Cranage, Zumsteins and Glenisla Crossing.
The fires are in the remote areas not scorched by fires earlier this summer.
"It's incredibly dry in there," Mr Nugent said.
It took 21 days to bring blazes in the Grampians, sparked on December 17, under control.
A bushfire sparked by dry lightning has been downgraded but authorities remain on high alert. (HANDOUT/STATE CONTROL CENTRE)
Forest Fire Management Victoria chief fire officer Chris Hardman said the fires would present a "similar scenario" as a cool change gave way to gusty conditions.
"The Grampians fire will be a real challenge for the coming days and potentially weeks," he said.
Premier Jacinta Allan said there were up to 150 fires across the state as high heat and winds brought dangerous conditions and fire bans.
20°C