"Every time you go there, you go there with trepidation," Ballarat Bulldogs coach Gary Sternberg said.

"You have one eye on the guys on the oval, and another eye on what is going on over there (the grandstand)."

For the past six months, Mr Sternberg, and residents in the surrounding area say White Flat Oval has become home to a group of rough sleepers who have set up an encampment at the oval's grandstand.

Ballarat Bulldogs, a football team for people with intellectual disabilities, use the ground for training and home games during the team's regular season.

Mr Sternberg said many of its players have copped abuse from the people camped out in the grandstand, to the point where some of the team's players have quit the game for good.

"Halfway through the year it began at training, with guys in the grandstand just yelling out abuse at training," Mr Sternberg said.

The coach said the abuse had reached a peak at the team's semi-final match against Cranbourne, where abuse escalated to threats of violence, and police were called.

"We had a big crowd, we had members of the public coming up to us and making complaints that people at the grandstand were making threats, to stab kids and rape women," Mr Sternberg said.

"At the end of the game our side had won and we were going to be in the grand final. All of the guys were up and about. Both teams were having something to eat and drink and having a chat.

"Three to four of the guys from the grandstand came up to us with threats of violence. People had to intervene, and it just got really nasty. We had to call the police, reports were made."

It is a delicate issue for Mr Sternberg, who works in the disability sector when not coaching the Bulldogs.

He said while he sympathizes with the plight of people experiencing homelessness, he was worried the issue may escalate to a point where someone is injured.

"We are sympathetic to the fact that they are homeless, through pure bad luck really, for anybody. It is the anti-social behaviour that has been getting worse," Mr Sternberg said.

"It is on us to provide a safe environment for these guys. They have lived with this disability all of their life, they have all of these challenges ahead of them. They are actually really inspiring, this team.

"To have that thrown at them now, it is very, very disappointing. You imagine, if we go down and play a game, it is up to us to go down and sweep the oval to make sure we find a syringe. How can we play a game there in all good conscience."

Other residents living around the oval have said they are fed up with having to call the police multiple times a week - with some even giving up calling the police and leaving the public space.

"The current state of our community oval and heritage grandstand is that a small group of people have taken over possession, making it difficult for the community to access these cherished spaces," local resident Darren said.

"The lack of intervention from the council and police is disheartening. It is our right to use the oval safely, to walk our dogs and enjoy the space without fear."

"The cops said if the people in the grandstand muck about, they will do something about it. So why haven't they? You can't just put your head in the sand and expect the public to be the ones that just have to deal with it," another resident Rob said.

"Why are they even still there? They have been there causing havoc long enough. We need to be safe and people need to do their jobs and follow through not just have meetings. When they behave that way they need to be moved on. Simple."

One local resident, who did not want to be named, said she had found human faeces and syringes at the site.

City of Ballarat mayor Des Hudson said he had spoken to residents living in the area and to the rough sleepers themselves, and that the council was taking a "coordinated approach" for the "best possible solution" to the issue.

The mayor did not say what the "best possible solution" would be however.

"Together we are trying to support a coordinated approach to find the best possible solution for White Flat Oval user groups, rough sleepers and the broader Ballarat community by balancing the rights of all people to be in public places, whilst respecting the right of communities to live in a safe and peaceful environment," Cr Hudson said.

"I have been regularly meeting with user groups, some of the rough sleepers themselves and other relevant stakeholders in recent times and will continue to do so in the future.

"Homelessness and rough sleeping is often a complex matter to resolve and we appreciate the patience of White Flat users and the broader community while we work towards a solution."

In October, three tents were pitched at the entrance of the White Flat Oval grandstand, two of which have since been removed.

Police are aware of the issue, and a spokesperson said they are working with the oval's rough sleepers to find them support and accommodation.

The police do have the capability to move on rough sleepers if incidents of anti-social behaviour occur, but the spokesperson said the police were more focused on making sure rough sleepers had access to essential services.

"The priority for Victoria Police is to link people who are sleeping rough to the services available to them, and we work closely with local councils and support organisations to address these issues," the spokesperson said.

"Police have increased proactive patrols in the area to address and concerns or reports of anti-social behaviour at White Flat Oval. If an offence is detected, police will deal with the matter at hand."

Support services include Uniting, who have been in "regular" contact with the group camped at the White Flat grandstand.

Uniting Homelessness senior manager Adam Liversage said the service was working on finding the rough sleepers long-term accommodation.

"We're in regular contact with those currently sleeping rough at White Flat Oval, providing assistance such as material aid and working to secure long-term accommodation in an area with limited affordable housing options," Mr Liversage said.

"Many are also regular visitors to our Breezeway community meals program and attend our local Uniting offices for any additional support they may need."

Mr Sternberg said he had attended meetings with the City of Ballarat, police and Uniting, and while all had been supportive, they were not able to address what he called the "elephant in the room".

"We have a grandstand at the moment that cannot be used as a grandstand. That seems to be the centre of all the anti-social behaviour," Mr Sternberg said.

"I know some members of the council are reluctant to board it up, but I still think there is a way of securing it with user groups having keys to it.

"Somehow it just seems to me that if we can tidy up the grandstand area we can help these guys."

Another resident who attended the meeting, who did not wish to be named, said the meetings had not achieved much.

"We had a meeting three weeks ago, there were 21 people in the room, and the outcome was to have another meeting with less people," they said.

"The red tape and the bureaucracy has just been ridiculous... the amount of people that are meeting, the amount of services involved, and the amount of expense that ratepayers will have to pay to fix the grandstand is ridiculous."

Meanwhile, western Victorian MP Joe McCracken has brought the matter to Victoria's parliament using question time to address the Minister for Police, Anthony Carbines.

"I ask the minister to take urgent action to make sure that the criminal activity that is going down at White Flat Oval is stopped immediately," Mr McCracken said in parliament.

"It is really quite a serious matter, and I hope that the matter is taken care of urgently rather than being just as another issue that is on the backburner, like so many other things in Ballarat are."