When ever someone has a special job like anything to do with a crafting table or crafting, the workers wont help the builders, if you still don’t understand, here is the summery.
Builders or Workers build Buildings. Other Jobs (Ignore Guards) could help them build than being an idle.