Hm…
dirt blocks blocking my progress by filling the inventory to cap.
geomancer comes into game in a far away future version.
“Ghost dirt”
…
doesn’t strike me as obvious.
How about this approach:
-
Dirt is a block like anything else but each hearthling can not put it in his/her backpack but it occupies thier hands. so every worker can carry only one dirt at a time. The geomancer might magically be able to move more later so there is some difference in classes.
-
Dirt gets its own stockpile type which counts internally but does not count towards the towns inventory cap.
These stockpiles can get placed near building places and can be moved later on.
From these "only dirt holding stockpiles (or short: “dirtpiles”) workers can grab dirt and place it anywhere the player wants them to: -
So some gobling dug out his private nest and you threw him out? Fill the gap with dirt again.
-
So you wanna build some farming grounds on the 2nd floor terrace of your castle or up in the mountains? Move some dirt up there!
-
The dirtpile itself is some type of crate which costs one wooden block and can be created by any worker. It has the shape of a decent heap of earth regardless of the amount of dirt stockpiled there and can hold an unbelievable mass of 512 blocks. It cannot be moved or destroyed until all the dirt and grass blocks inside are moved elsewhere (not like the crate which can get moved with the items inside).
This way dirt does not clog up the inventory and you can fill gaps or build your first walls out of it if you don’t mind that it takes fairly long (moving one block at a time instead of working with wood!) and leaves some not-so-beautiful brown holes in your town.
This solution provides you with the posibillity of moving dirt/grass with finite resources (no ghost dirt!), does not increase the game’s workload / calculations through the roof (mined dirt blocks don’t have to be displayed! only an integer in the stockpile 
) and gives some perspective to future applications (geomancer).
It’s no exploit towards early wall building either because of the slow building speed and the visual scars it leaves on the surroundings. It could actually become an addition to wall building as the “moving dirt”-option might encourage some players to build “earth walls with a stockade” .
It’s just a suggestion though. 