The owners of secondhand art and literature-focused Bookhouse say they are gutted after the City of Port Phillip upheld a complaint it received last August that the shop was trading illegally in a residential zone.

Irate locals and customers of the shop in Robe Street, just off the Grey Street red light district, described the ruling as mystifying and grossly unfair.

Mayor Bernadene Voss told The Age that an "existing use rights" clause could have let trading continue – meaning the owners had to prove the premises had been used as a shop for at least 15 years, without a break longer than two years.

But Bookhouse had failed to prove this, so council ordered it to close by June 5.

Ben Kemp, who runs Bookhouse with wife Margot McCartney, described the situation as "Kafka-esque" and would like to know what upset his neighbour.

"It's like [Kafka's novels] The Trial, or The Castle. I cannot get my head around it," Mr Kemp said. "It's like, the council says 'yes you can do this', and it is a shop. And then it's not a shop. And you've got a complainant who won't even tell you what the complaint is, who won't talk to you.

"It's so mad. Every time I tell someone, it messes with their heads."

"I feel gutted, because I can't imagine anyone finding issue with a bookshop."

Mr Kemp believes that 52 Robe Street has continuously operated as a shop since the 1920s. Locals made statements "saying they bought lollies here in the 40s when it was a milk bar". More recently, it was the Nefertiti video store, a homewares outlet and a gallery.

Mr Kemp said council's sticking point was the years a costume-maker had occupied the space (2009-2012), because it was deemed not to have been enough of an obvious shop.

While Mr Kemp's landlord is considering appealing the decision at VCAT, Mr Kemp says he and Ms McCartney aren't willing to operate "in limbo" and will move out. But they will continue online and look for another location for Bookhouse.

Ms McCartney said the shop was "incredibly well loved and well frequented", holding exhibitions and book launches. Its book groups include TV host Brian Nankervis, musician Tim Rogers and comedian Colin Lane.

A longtime customer, architect Phillip Schemnitz​, said Bookhouse "brings the community together" and its closure was "terrible" and "grossly unfair".

"I'm really angry and disappointed," he said.

"I offered to support Ben financially if he'd wanted to go to VCAT over this, and some of my friends would have done the same thing. We feel the diversity factor in St Kilda is diminishing and there's less for locals in St Kilda to actually keep us here."

RMIT writing lecturer Lucinda Strahan, said Bookhouse was her "magical literary bolthole".

"This Bookshop brings so much joy to my everyday life as a local ... the thought of it being kicked out is appalling and completely illogical considering that everyone in St Kilda knows that building has been occupied by some kind of business for decades."

Writer Bill Garner, who has lived in Robe Street for almost 35 years, said it was "just strange".

"Everyone I've spoken to is delighted that there's a bookshop there and we'll be very pissed off if he's forced to leave.

"It's impossible to imagine it being offensive in any ordinary understanding of what offensive would mean. Even for a bookshop it's a quiet one. It's not like there's hordes of people outside going in and out."

The Age approached the neighbour who made the complaint but he didn't want to comment.