Not sure if it was just a fluke, but when I went to remove a ladder I had built to the roof of a building so my hearthlings would remove some of the logs they left up there, there was no indication of it given (like there is for other commands, such as harvest, move item, undeploy item, etc.) (see attached video, 27:42). I think it would be nice to get one of these for this designation also.
Incidentally, the ladder was indeed removed (barely visible at 30:19), so in that sense I cannot confirm some of the bug reports of ladders not being removed at all, although that might still be an issue under other circumstances (see these bug reports: 1 2 3).