Sometimes this happens - if you manually place ladders on the red blocks they’ll manage to go there and build them (and you can order the ladders to be removed later)
I’ve helped someone with similar issues and provided some nice screenshots, let me find a link here…