The state's virus testing chief, Jeroen Weimar, said ADF officers checking home isolation compliance caught the returned travellers breaking the rules on Friday, and they will each be fined just under $5000.

Mr Weimar said after weeks of work to contain the latest outbreaks, his biggest fears are returned travellers failing to comply with restrictions, and more leaks out of the hotel quarantine system.

"Those are the two things that give me sleepless nights," he told reporters on Saturday.

"Our main concern is when I've got 13,000 people here in Victoria who have come from interstate on red zone permits, each one of them represents a risk that they are incubating the coronavirus."

Of the 13,000 people who have travelled from red zones, 44 per cent have so far returned negative tests.

Victorian residents returning from red zones such as Sydney must self-quarantine for 14 days and get tested twice during that time.

About 48,000 people have recently entered Victoria using orange zone permits, which requires them to isolate and get tested, and so far 41 per cent have returned negative test results.

On Friday 50 defence force members visited just under 200 households to check people are following isolation rules.

Authorities also checked 800 people who arrived in Victoria by plane, of which 600 people came from red zones.

Five were forced to turn around as soon as they arrived, while another three were immediately taken to hotel quarantine.

The state has notched up its third day in a row with no new local cases.

That leaves 30 active cases, with 210 primary close contacts from the recent local outbreaks in Sandringham and Oakleigh still waiting to clear quarantine.

Mr Weimar said any further easing of restrictions in Victoria depends on local efforts as well as the situation interstate, with 35 new local cases recorded in NSW.

"That's a pretty high fuel load that's firing off north of the border and we're rightly concerned about that," he said.

But as the situation in the NT improves, Alice Springs, Darwin, Palmerston and Litchfield have been downgraded to orange zones.

On Saturday morning authorities sent messages to about 1300 red permit holders who have returned from the NT, to tell them they can leave self-quarantine early if they have received a negative COVID test.

"When we can release you early, we of course will, that again is part of the whole border permit system," Mr Weimar said.

The state administered 19,623 vaccinations on Friday, with 24,247 test results received.

Also last week, the state's Health Minister Martin Foley told anti-vaxxers to keep their "tinpot theories" to themselves after a protester threatened staff at a Melbourne hub administering the vaccine.

An anti-vaxxer barged into a mass COVID-19 vaccination centre at Cranbourne in the city's outer south east on Friday morning.

Hundreds of police are continuing to monitor the Victorian-NSW border, stopping 2100 people in vehicles and on public transport on Thursday.

Australian Associated Press