Police arrested five people and three officers were injured as the deadline arrived for the camp to be cleared so that work could begin on upgrading the railway station.

One police officer was punched in the back of the head amid frenzied scenes that brought the intersection of Flinders and Elizabeth streets to a standstill.

The day started quietly, with fewer than a dozen people sleeping on the Flinders Street pavement, but took a tumultuous turn after protesters hijacked the camp and special operations police descended on the area.

Onlookers described the scene as "a circus", with concerns raised about the impact the angry protest would have on people sleeping rough in the CBD.

A large media contingent and onlookers filming with their mobile phones followed every scuffle as it broke out.

Homeless man Glen had been living with his dog Tonka outside the station for more than a month. When police arrived and formed a line around the camp he took his three suitcases and left.

"But I look like an activist now because the professionals [protesters] brought banners and took over the joint," said the 32-year-old.

"I wasn't involved in it."

Glen said he had not decided yet where he would go. At least half a dozen rough sleepers have relocated to other spots in the CBD.

In recent weeks the state government and Melbourne City Council issued repeated warnings that the camp would be removed so work could continue on a $100 million upgrade of the station.

The Andrews government had announced the rough sleepers would have "immediate" access to 40 transitional housing units.

"The advice I have got is everyone in that camp has been offered those services," Housing Minister Martin Foley told ABC radio.

It's understood some of the rough sleepers were placed in crisis accommodation in a Coburg motel.

Stuart Bateson, Victoria Police's Acting Commander for North-West Metro said most of the rough sleepers complied with police requests to move on, but a large crowd of protesters gathered and became violent.

Commander Bateson said police first spoke to the rough sleepers at 10.30am, telling them they would have to go.

"We do not under any circumstances think we have solved the homeless issue. Yes we will probably see them somewhere else in the city," he told reporters.

About 2pm, silence descended on the area as trams were halted along Flinders Street and tensions boiled over after special operations police wearing red vests arrived on the scene.

In front of a large crowd of onlookers that spilled across Flinders Street a man in a mobility scooter and a woman who had been stationed at the camp were arrested, as people yelled "let them go" and "f--k the police".

Afterwards another man began pulling down the yellow barricades trucked in for the start of construction work.

One person was arrested for assaulting a police officer, another for throwing an object at an officer, one for public drunkenness and two for resisting the police. Only one of those people was homeless, a police spokeswoman said.

Commander Bateson said one person repeatedly rammed their personal mobility scooter into police.

He said police expected to lay charges in all cases.

But Commander Bateson said the protesters were "regular starters at any far left protest".

Earlier on Wednesday, Mr Foley said 15 people from the camp had been "successfully housed" while four others had been placed in drug and alcohol rehabilitation facilities.

He had urged the last remaining rough sleepers at the Flinders Street Station camp to accept a "pathway" out of homelessness or face eviction.

Public Transport Minister Jacinta Allan said scaffolding works had been "planned for months" as part of the station's $100 million upgrade.

A City of Melbourne spokeswoman said the council's role was to collect "unaccompanied belongings" and co-ordinate housing and support services for those wished to accept the offers of help.

Next week the council will consider a report on homelessness.

With Nick Toscano, Emily Woods