Sure, it breaks the immersion of the game a little bit, but I think they should absolutely be able to use ladders. Otherwise, when they spawn on a different “level” as your village, they wouldn’t be able to get to it. Or, they could just be given the ability to jump from one level to the next. Either way, they need to be able to get to your town.