I don’t know if I just haven’t found the way to remove it but I tried moving my village flag inside a building but none of my villager were moving it, instead, I ordered them to move it somewhere else outside and they did immediately. The problem is that I can’t remove the «ghost» of the previous order, I am stuck with 1 «ghost» flag inside my building and cancel order is doing noting.
It is like the flag is now considered a furniture to be moved inside but nobody is able to move it that way.