I always set up my houses 5 spaces apart. I’ve never had them stop working on houses because of this, but even with multiple ladders and ways to get the house done they refuse to touch them.
I also have a red exclamation mark over a finished custom building.