The Legislative Assembly's Legal and Social Issues Committee inquiry into the state's anti-vilification protections also called for the laws to be extended to cover gender identity and sexual orientation, disability and HIV status.

The committee, made up of three Labor MPs, two Liberals and two independents, said there had been an increase in public displays of Nazi symbolism and a rise in racially motivated incidents during the coronavirus pandemic.

They cited the storming of the US Capitol building, which was participated in by neo-Nazis, including one man pictured wearing a "Camp Auschwitz" shirt.

Further, it was revealed the Australian Security Intelligence Organisation was directing up to 40 per cent of its counter-surveillance at far-right extremist activities in 2020.

"The committee believes it is important to send a clear message to the community that Nazi symbolism is not acceptable in any form and has wide-ranging, negative societal impacts," the report reads.

It recommends the Victorian government establish a criminal offence that prohibits the display of symbols of Nazi ideology, including the swastika, with considered exceptions to the law.

"This would allow Victoria Police to immediately remove Nazi symbols that are on deliberate display to vilify targeted communities," the report said.

The committee also recommends the government monitor the public display of other hateful symbols to determine whether a broad-based offence should be established.

Committee chair Natalie Suleyman said efforts to protect minority groups in the past have fallen short.

"It is essential for the Victorian government to implement legislative reform and to also develop complementary prevention-based strategies to reduce and eliminate vilification in Victoria," she said in a statement.

The committee recommended various initiatives in the areas of research, school-based education, public awareness campaigns and responsible media reporting to address the causes of discrimination and hostility towards minority groups.

The inquiry was set up in 2019, just months after the government was powerless to stop a neo-Nazi music festival from taking place.

In January 2020, the government was also unable to stop a family from flying a Nazi flag above their home in the small town of Beulah.

Opposition police spokesman David Southwick said the swastika has since been displayed at events in Cranbourne, Geelong, the Grampians and Kyabram in a "deliberate attempt to intimidate others".

"After a long battle we are one step closer to seeing a ban on the Nazi swastika, a universal symbol of hate which has no place in our proudly multicultural state of Victoria," he said in a statement.

Dvir Abramovich, the chairman of the Anti-Defamation Commission, has been campaigning to outlaw the public displays of the swastika for three years.

"I will be lying if I didn't admit to shedding tears of joy when I read the report," he said.

"The committee has declared in a clear and unmistakable voice that the ultimate emblems of inhumanity, genocide and racism, that are meant to break our spirit and instil fear, will never find a refuge in our state.

"This is a triumph for the victims of the Holocaust, the survivors and our brave diggers who died to vanquish the demonic Third Reich regime."

Premier Daniel Andrews on Tuesday indicated the government would take action against the use of the Nazi symbols.

"There's no place for those views, there's no place for those symbols, there's no place for those attitudes and conduct in a modern Victoria," he said.