As to the scaffolding I actually have seen my workers do that. continue to put scaffolding in place as the builders build. Usually it is my miners that are doing that since those are the ones I have marked not to build, and when there are no mining jobs; else it is the other peoples. I guess my builders build fast enough for that to happen in certain instances.
But generally yes it seems to be a switch off. However I have noticed that when they lay the bricks down, and go to get wood for scaffolding; it is not on their mind to continue to use that very same piece they were working from. I know this due to fact clay bricks are not individually stored at the moment, and I refuse to use the All function for a stockpile beyond the initial one that eventually gets reduced…
When scaffolding is needed they will go for wood and do that, and seems like they will get a new piece anyway, and sometimes will use the pieces laying around. 50-50. The things that can be stored, they will return the piece they are using and switch out with the wood; stone as the example.
I’ve watched them do what I mentioned. Love watching them build, and never use the speed options, as I don’t mind the slowness that speed 1 has. Less to miss that way. 
To make scaffolding not a ladder task, if it is the case, it might have to be separately defined I’d imagine. But wouldn’t think would be all that complex since it is almost a replication to be sorted with something else.
However the ladders they build along with the scaffolding, there is no difference between those and the ones you place beyond probably a flag. Meaning hearthling temporary more likely and one that you can issue to tear down. I think anyway… Only guessing… But I do agree on that part for it possibilty being a nice gameplay tweak to have. Then again I don’t mind, as long as the building itself is being built.
So to the initial point I got what you meant, but I don’t actually see that happening all the time. After all it is one task at a time per say. Unsure they would be thinking reusing the resource they just used. It would be more like ok scaffolding check, now where is close resource for building. So possibly timing on that. At least if I am understanding the hearthling mindset they have coded in. Of course I am kind of thinking on the processes of which makes their actions.
Also on that same bit, I have seen em’ drop the resource (clay bricks) and someone else takes and uses it instead; not always the same person that dropped it. Else like mentioned above if not clay bricks they seem to return to stockpile to do a somewhat trade off for scaffolding supplies. I make mix material buildings. Stone/clay with the obvious wood in roof the RC normally has. Is the primary faction I play at the moment. Well that is my observations from my own experiences. 