Housing criticism a distraction: Shorten