Mr Taylor was awoken by what he thought was a picture falling off the wall in the early hours of Monday morning.

Dazed and groggy, he looked up, and through the exhaust fan above his bed, could see the moon from his Horsham house.

"I thought, "What the heck is going on here?"' Mr Taylor said.

"I felt the bed around me, and it felt like I had crumbs all through my bed. I got up, shined a torch on it, and it was all insulation, plaster, and rubble.

READ MORE: Bureau of Meteorology update on freak Horsham storm

"I went out to see how bad the damage was, and I realised my roof was completely gone.

"I couldn't believe it."

A freak weather event - that officials say could have been a tornado - moved the roof from Mr Taylor's Frederick Street house into his backyard.

Mr Taylor's front veranda also collapsed, windows were broken, and half of his brick chimney was strewn 30 metres away, all in a matter of minutes.

"It stripped everything," Mr Taylor said.

"That's the bit I'm peeved about; I spent 16 years renovating, putting love and money into this place, and in half an hour, she's all gone. It's a bit upsetting.

"We knew we were going to have wind and rain last night, but nothing like this."

Mr Taylor however said there were silver linings to take from the wild night.

While the roof of his shed was also torn off, his priceless classic car inside somehow remained untouched.

Mr Taylor, too, along with everyone else caught near the unforeseen storm, avoided serious injury.

"It's upsetting ... but I'm still here. I'm not injured," he said.

"It's just the house."

Mr Taylor was one of many in north Horsham to be severely affected.

The roof of his shed was lifted and transported into neighbour Spike Smith's yard on Wawunna Road.

Mr Smith said he was awoken by what he thought was a car crash.

"I thought someone had crashed into our neighbour's caravan out the front," he said.

"We came outside and we could see sheets of iron flying through the air.

"But within about three minutes, it had all died down."

Mr Smith also had a hole in his roof, likely caused by flying debris.

He said he was incredibly lucky to avoid serious damage to his home.

"The roof in our backyard has bounced off our shed, and how it's missed everything else has got me buggered," he said.

Emergency Services Minister Lisa Neville said five homes were now uninhabitable.

"Although the bureau is yet to formally declare it a tornado, it certainly has the appearances of it," she told reporters on Monday.

"I know that it was a terrifying experience. We've got supports from local government, CFA and SES up there to assist them in any way that those communities need."

Bureau of Meteorology forecaster Christopher Arvier isn't ruling out the possibility it was a tornado, but noted the strongest winds only reached 50km/h.

"There was quite severe damage that occurred as a line of showers went over Horsham," he said.

"We did see very strong winds across Victoria across the weekend. So it is also possible that it was just associated with some very strong downbursts during some showers."

Mr Arvier said tornadoes are quite rare in Victoria, with only one or two forming across the state each year.

But a team of severe weather meteorologists is evaluating the weather event, with a preliminary assessment expected to be completed on Monday afternoon.

"Unstable air can lead to lots of updrafts and downdrafts in the atmosphere and when you get that sort of stretching of the atmosphere, you can start to see the winds turn and become tornado-like," Mr Arvier said.