A Daylesford man told police he drove to another man's house to verbally sort out an issue but it turned physical after he was pushed.
Shayne Adam Petkovic voluntarily took part in two recorded interviews with detectives over the violent incident on January 17, 2017.
The interviews were played to a Ballarat jury on Thursday during a trial being held at the County Court.
A day after the alleged incident, Petkovic, 41, told two detectives he received a text message from a woman complaining about the victim.
The woman had taken out an intervention order against the victim, who was not allowed near her.
"I was feeling really stressed and worried for (the woman). It's been going on for so long now and nothing has been done about it," Petkovic told police.
"There has been other incidents and he just keeps doing it."
Petkovic said he decided to drive his BMW to the man's Musk house after feeling very annoyed to sort the issue out verbally.
But he told police in the second record of interview on February 6, the conversation "obviously didn't go that way after he pushed me".
The pair were involved in a wrestle inside the victim's house where Petkovic allegedly punched the victim twice and threatened to kill him four times.
When detectives asked Petkovic why he punched the victim he replied, "fear of him hurting me" and "he is quite a big guy".
He then said he did not know why he punched the victim and he wished he hadn't done it. He strongly denied making the threats to kill.
The jury was told the victim received bruises, small cuts and scratches during the fight, while Petkovic's knees were bruised.
Petkovic faces nine charges, including four counts of making a threat to kill, two counts each of intentionally causing injury and recklessly causing injury and one count of aggravated burglary.
He has pleaded not guilty.
The trial before Judge Michael McInerney will continue on Friday with closing addresses from counsel.