Update: Found this thread: 2 story building bug
Related, I removed the building that had ladders in it, and built ladders so the hearthlings could remove the stuck ladders. After they removed those ladders, they finished the building without further issue.