Normally the ghost would remain if no hearthling has placed yet the item, but if you played during 4 hours and they didn’t have enough time to pick it and place it, it’s a bug.
Was the fence reachable by them? Maybe it was in some place that they couldn’t access so they didn’t go to find it.
Otherwise, for removing it, try either with the “Clear” command from the harvest menu (the one with the axe and pick icon) or open the console with Ctrl-C, select the ghost fence, and type destroy, then press Enter.
right ok, you’ll be pleased to hear the ghost fence is no longer haunting me (Ctrl-C, who knew it was such a good exorcising tool)
however it seems the bug has evolved as i have checked all my stock piles and crates and the fence is nowhere to be found, but i still have the facility to place it