I tried building pretty simple buildings and I am still having issues. I noticed scaffolding on both the inside and outside of the building (causing one hearthling to get stuck inside the building). After a couple days nobody tries to complete the building. I tried hitting build several times, there are plenty of resources and lots of idle workers.
Once the workers gave up on completing the building, I attempted to remove it. This also caused an issue. Work started on building removal and then stopped just like construction. As you will see by one screenshot, one of the workers got himself stuck on the roof. (Almost as funny as it was frustrating 
) Now i cant remove or complete the building…and two of my workers are stuck forever lol. Time for a restart/new game.
