Here is a picture just showing about how i have buildings to build and items to hual but still lots of idling, except those Farmers! man they try to do too much!
I have sort of seen something like that in my games too, especially when I created a lot of items during early stage where I am clearing an area for the settlement. No matter how many stockpiles (for wood) i designated, they hearthlings always seem to have something else to do/haul. (they will haul if it is part of another job like crafting or building, but not to stockpile) But I ignored it and just continue playing and after some time, when they are idle, they will start to haul the wood.
No idea what is going on, but if I have to make a wild guess, it will be timeouts during path-finding(?)