The suburb has come a long way since 1940, when the novelist and former resident Christina Stead called the village of fishermen and sailors a "civic disgrace".

A heated battle is mounting over an 80-year-old villa on beachfront Victoria Street, Camp Cove, Watsons Bay.

With millions of dollars riding on its heritage value, facts about the Villa Porto Rosa are in dispute. It was built by Hugh Victor Foy, but the property's importance is under question by its new owner, who says Mr Foy was not prominent in his famous family's retail empire.

George Miller, who owns three properties in the street, has written to Woollahra Council, along with 14 other neighbours, to express his concern about demolishing "a remnant of a period of Sydney of which not a great deal remains today".

"Camp Cove is important not only to local residents or to Sydney but indeed to the whole of Australia," he and his partner, Margaret Sixel, told the council.

One of the first beaches inside the harbour and the site of the First Fleet landing, Camp Cove has been the secret hideaway for "city merchants" for decades. Its spectacular setting lured three retail dynasties. First was the Foy family, who built the villa in about 1928. It was home to Hugh Victor Foy, who ran Mark Foy's from 1918 until he died in 1943, his daughter Mitta Farrell said.

Next came Beryl Phillips, the daughter of competing retail giant of the day, Albert Grace, who bought the villa in 1955 from Mr Foy's daughter. Now 87, Mrs Farrell had subdivided and built a house next door after World War II. She still lives there with her son Michael, having brought up seven children with her late husband, the local postman.

Finally, the Lowys bought property in the 1990s and built their home on two blocks on the other side of the villa.

The retail dynasty string was broken last December when Vaughan Blank paid $28 million for the villa after Mrs Phillips, a centenarian, died on March 16.

Mr Blank wants to spend almost $4 million demolishing the villa, replacing it with a house that will include two laundries, six ensuites, three guest bathrooms, a media room, a rumpus room, a casual living room, a formal living room, a kitchen and a lap pool.

To get approval, he must justify knocking down a building that the council's development control plan says contributes to the heritage of Camp Cove.

The new owner's heritage consultant, Graham Brooks, says Mr Foy "does not appear to have any prominent role within the Foy empire". The villa has no heritage value, Mr Brooks writes.

A council heritage officer, Louise Thom, agrees in her assessment of the property.

"Hugh Foy was never directly involved in the family business and is not listed as a significant person in the Australian Dictionary of Biography ," Ms Thom writes. "The property has no special association or meaning for a contemporary group of people. The property therefore has low heritage value in its own right."

She has ordered Mr Blank only to retain some plantings that contribute to the heritage of the beach.

Mr Foy's grandson, Michael Farrell, who grew up next door to the villa and lives there with his mother, disagrees: "He actually ran the thing for 30 to 40 years."

Mr Blank has already shored up support from his next door neighbour, Mr Lowy.

A letter to the council from Mr Blank's architect expresses support, giving an insight into how the rich and powerful co-exist in the city's crammed eastern suburbs. It describes a meeting that included two of Mr Lowy's architects, Peter Stronach and Scott Norton, and his landscape architect Sacha Coles.

After Mr Blank agreed to change a bedroom and a deck, and add a wall and more plants, Mr Lowy provided his "conditional support".

The Farrells have lodged an objection to the plans but also decided to move rather than live through the noise of the demolition and construction.

"I don't want to leave on a sour note," Mr Farrell said.

He has sold the last of the Foys' 100-year-plus association with Camp Cove to a "charming" English friend of Mr Blank, who will keep it as a holiday home.