Does anyone else have a house rule on raiding unprotected cities?

I'm relatively new to the game, we've played it 5 times so far, and there's just one thing about the rules that has rubbed my play group the wrong way. It seems deeply unintuitive that you can't raid a city if there are no ships there. Thematically, you're obviously attacking a city with raid dice since a city needs to be there and it's also fairly silly that, somehow, not having any ships on a planet to defend a city makes a city invulnerable. We ended up trying out just letting people raid undefended cities and it seemed totally fine, but I'm curious if others have experimented with this house rule, and if there are downsides to it that we just aren't seeing yet.