Hay supper quick suggestion. The guys shouldn’t have to climb down the scaffolding to make the next level of a structure. I have worked in construction and have had to assemble scaffolding before and when we assemble scaffolding we do it from the top not the botom. So it would be more realistic and dramatically faster if the villagers built the scaffolding first then built the walls.
I have linked a video of some british people putting up scaffolding so you can see what I meen.
Edit- (I posted this clarification a little down the page but im going to move it up here)
Also what im suggesting dosent change anything except that the civilians dont climb down the scaffolding. they just build it while there standing on it. I would look 90% the same, all you would do is stop the climb up and down and up and down animation. Thus saving time with no additional craziness being added to the game.