The pair exchanged barbs on Thursday amid the ongoing political furore over street violence, sparked by Mr Dutton's claim last week that some Victorians were scared to eat out because of gangs.

Mr Andrews scorned those suggestions and invited Mr Dutton to join him for dinner at a restaurant, an offer quickly rejected by the Turnbull government minister.

"He can offer to shout me dinner all he likes, but all it is is a distraction," Mr Dutton told Sky News on Thursday night.

"Mr Andrews has taken this left-wing approach to the law and order system and it has resulted in this gang violence in Victoria."

Mr Dutton accused the premier of appointing lenient judges who were wrongly allowing bail and imposing "very soft sentences" in the name of political correctness.

"He needs to admit that he's made a problem," Mr Dutton said. "If he doesn't, as I say, then he needs to consider his position."

Returning to work from leave, Mr Andrews said there had been some nasty criminal incidents recently but expressed confidence Victoria Police was "turning this around".

"We can be confident they are the best police force in the nation," he said.

Mr Andrews' return to work comes amid heightened tension between the state and federal governments over youth violence in Victoria.

At the end of last year Prime Minister Malcolm Turnbull hit out at the Andrews government's handling of "gang violence" in Victoria.

But on Thursday morning Mr Andrews said the Prime Minister did not raise the matter when they spoke late last year.

"I must have spoken to him four or five times towards the end of the year about a whole range of issues."

"I think Mr Dutton's comments were designed to get a rise out of people," Mr Andrews said. "They were designed to be as controversial as possible.

"I don't know how often he spends time in Melbourne. He's always welcome to come and have dinner."

But on Thursday Mr Dutton stepped up his attacks on the Victorian government, stressing he does not want to see a "small element" of the African community tarnish the reputation of others.

In an interview on Adelaide radio station FIVEaa he cited a Fairfax Media-ReachTEL polling conducted last week that found unease in suburbs affected by recent high-profile incidents.

Mr Dutton said there was a "problem with some of the judges and magistrates that Daniel Andrews has appointed" and criticised people for denying the existence of gang-related violence.

But, Mr Dutton's office declined to provide specific examples of concern in Victoria's court system when contacted by The Age.

The Law Institute of Victoria said it was "extremely concerned by the ongoing political attacks on Victorian judges, magistrates and the legal profession".

"There is no place for political attacks on the judiciary and undermining the independence of our judges and magistrates," the institute said.

The Judicial Conference of Australia also raised concerns about "repeated personal attacks on judges and magistrates".

Victorian Attorney-General Martin Pakula said: "While Peter Dutton may enjoy going on Adelaide radio to take ill-informed pot shots at Victoria he has clearly not been paying attention. No Victorian government in recent memory has appointed more prosecutors and ex-prosecutors to our courts than this government. "

Mr Andrews said he had termed recent criminal activity as "gang crime".

However, he said Deputy Commissioner Andrew Crisp had expressed concern that some people might be "fuelled or pushed on" by being called a gang.

"I do have some sympathy for the difficult balance that Victoria Police has to strike," Mr Andrews said.

Despite some high-profile criminal incidents recently, crime in Victoria has declined, the latest data shows.

On Wednesday Chief Commissioner Graham Ashton rejected suggestions of an "African gangs crisis" and said some people of African origin had been subjected to racially motivated death threats.

South Sudanese leader and solicitor Kot Monoah said members of the community had been unfairly threatened and racially vilified.

Together with African-Australian leaders Victoria Police will form a taskforce to tackle public order offences.

Victorian Opposition Leader Matthew Guy said his office had been inundated in recent days with "hundreds of examples" of Melburnians who were frightened about the state of law and order.

Mr Guy accused the Andrews government of failing to acknowledge the seriousness of the issue.

With AAP