However residents still don't have a clear idea on what will specifically be located on the site, and it won't be clearer until private sector interest is revealed.

Premier Daniel Andrews visited the site on Friday claiming he would "fast track" the project which has long been mooted as one of the biggest in Ballarat.

Labor pledged $25 million to the project in the lead up to the election, however funding from the private sector will still be needed to see the vision turn into reality.

"The important thing is getting the sequencing right, the $25 million we have pledged to spend and invest in is helping to clear the site and deal with safety and asbestos issues and getting this site and this precinct ready for a partnership between the public and private sector," he said.

Friday's visit was to announce the fast track process which will advance the planning and governance arrangements for the site.

The project has been one of the more high profile for Ballarat, with the previous government spending much of their term talking up the project and developing plans.

In fact, ex-premier Denis Napthine stood in front of the same exact plans on the same site only six months ago spruiking the project.

However the Labor commitment was the first time significant funding has been committed to the project.

A masterplan for the project was unveiled in July which envisioned a complete revamp of the area including construction of a four and half star hotel.

City of Ballarat mayor John Philips was on hand for the announcement with the council pushing particularly hard for the project during the 2014 election race.

"This building has sat underutilised for such a long period of time and it will slowly fall apart as underutilised buildings do," he said.

"There are businesses waiting to get involved and we are just treading water at this point to get it going  as quickly as possible."

Minister for Regional Development Jaala Pulford echoed this sentiment saying there was private sector interest waiting for the site to be ready so they could invest.

"I can understand the residents frustration with the lack of progress on this project over four years," she said.

"We are really hopeful in starting construction before the end of the year but we need to walk before we can run.

"We need to get the governance arrangements for this part of the precinct sorted first."