And they say the company that purchased the Corkman Irish Pub for $4.76 million last year should be forced to surrender the land to Melbourne City Council.

The 159-year-old pub, formerly known as the Carlton Inn, was knocked down without a demolition or planning permit – sparking five separate government investigations into what laws may have been broken.

The pub, which had a heritage overlay, was already at the centre of a police arson investigation over a fire that broke out there the week before.

The site is owned by developers Stefce Kutlesovski and Raman Shaqiri, who are building apartment projects all over Melbourne.

The pub was demolished by Shaq Demolition and Excavation, which is half owned by Mr Shaqiri, who holds valid demolition and building licences.

The pub's destruction has led to community anger, with a petition to see it rebuilt now signed by almost 13,000 people.

On Wednesday, the Construction Forestry Mining and Energy Union, which has thousands of members working on large-scale building projects, said a "green ban" was now in place on the site where the pub once stood.

The union and the Victorian Trades Hall Council have instructed their members not to work on the site. It is believed to be the first such ban by construction unions in at least a decade.

CFMEU state secretary John Setka said Melbourne City Council and the Andrews government needed to "take this site off these heritage vandals, and put it to public use".

Mr Setka said the fines the developer faced for illegal demolition – likely up to $380,000, although lord mayor Robert Doyle has speculated they could top $1 million – had not been a disincentive to demolishing the building.

Trades Hall secretary Luke Hilakari said more than fines were needed to deter developers who wanted to "profit from the vandalism of Victoria's heritage".

"The only way to permanently protect our heritage is for illegal demolition to amount to surrender of the site itself," said Mr Hilakari, who added seizing the land from its owners would "stand as a warning".

The Age has attempted several times this week to contact Mr Shaqiri over the building's demolition, but his mobile phone has gone unanswered. He could not be reached on Wednesday.

The Victorian Building Authority, Melbourne City Council, Worksafe, the planning department and the Environment Protection Authority are all investigating what happened at the site on the weekend.

It came as the Environment Protection Authority Victoria confirmed it had found asbestos among the rubble.

EPA manager Daniel Hunt said there would be a notice issued on the site's owners over the waste, that would also "require it to prevent any asbestos from leaving the site".

Planning Minister Richard Wynne said he was pleased the CFMEU was "taking a stand against illegal demolition", but stopped short of backing acquisition of the site.

He said the government was considering increasing fines for destroying commercial buildings in the wake of the pub's demolition.

The current fines for illegal demolition largely target residential construction, where hundreds of thousands of dollars are considered punitive.