The baby boy had moved from South Australia to Ballarat with his uncle, grandmother and father in April 2017, five days before he was diagnosed with the serious head injuries. 

While police allege the baby's father, Jesse Harvey, caused his son's injuries sometime between 2.30am and 1pm on April 19, the accused's barrister asked his mother and brother if they picked up the baby at 2.30am and shook it.

Defence barrister Tim Sullivan asked the two family members the confronting questions at the County Court in Ballarat on Wednesday during Harvey's trial.

Harvey's mother Catherine Scott replied with, "no way, no way known", while brother Peter Scott said, "I suggest that is a load of b******t that just came out of your mouth", and "why would I shake a baby?".

Mr Scott said he saw Casey possibly at 9.30am the day his injuries were diagnosed and he was fine but might have been a bit whingy.

The jury was told the baby, Casey Harvey, was discharged from the Ballarat Base Hospital on April 14 after he was admitted with a low-grade fever and vomiting.

Five days later, Harvey used a taxi service to take his son back to the hospital emergency department where his condition deteriorated quickly. He was then transported to the Royal Children's Hospital.

Deputy director of the Victorian Forensic Paediatric Medical Service at the Royal Children's Hospital, Dr Jo Tully, told the jury the baby's diagnosis included multiple small subdural haematomas of varying ages, retinal hemorrhages, swelling on the brain and a diffuse axonal injury. 

She said a MRI scan showed there was tearing and swelling around the posterior ligaments at the back of Casey's neck.

She said this combination of injuries could only be caused by violently shaking a baby.

 "The combination of the injuries he had really result from quite significant forces," Dr Tully said.

Mr Sullivan suggested the injuries could have been caused before Casey's first hospital admission, but the doctor said children became unwell quickly if they suffered a head injury.

Ballarat Base Hospital emergency department doctor Heather Crook told the jury her initial diagnosis notes of the baby included he was unresponsive, pale, floppy, he had tachycardia and anemia.

She said he was in the resuscitation unit, radiology and intensive care before he was transported to the Royal Children's Hospital in Melbourne.

Harvey has pleaded not guilty recklessly causing serious injury and negligently causing serious injury.

The trial will continue before Judge Gavan Meredith on Thursday.

Have you signed up to The Courier's daily newsletter and breaking news emails? You can register below and make sure you are up to date with everything that's happening in Ballarat.