The level at which he was built isn’t the issue here.
I may be wrong, as I am by no means an expert - but enemies are spawned at the edge of your visibility +/- 80 as long as there is an AI path to your settlement.
Take this as an example, I BELIEVE that only the yellowy-green areas are available as spawn locations.
Hence the potentially very quiet game.
Original for comparison
Ladders would open up the other areas as potential spawn locations. @sdee could say for sure though.