Found a situation where part of the scaffolding can’t be removed. It looks like the problem is with multi-story buildings with a roof on more than one level.
As you can see in the images below, the hearthlings have built the scaffolding up through the lower roof to build the higher wall. But they seem incapable of removing that same scaffolding when the job is done. You’ll note that in the third image, I have a bunch of idle workers and the scaffolding left behind.
For quick duplication, use this building file. It’s the building in the third image. Only uses 1 wooden door and some wood.