In the middle of the night my unit decided to sleep walk into the goblin camp.
He went and picked up a wood and started taking it to the goblin camp.
You can see in the video he is walking to the camp with wood like he knows what he is doing.
Report of findings:
I noticed that the stockpiles in the goblin camp were editable by me. I think that the unit was actually trying to fill those stock piles up.
Once I clicked on the stock piles and deleted them my units stopped trying to move stuff to the goblin camp.
In the game I did click on the fire pit of the goblin camp and try to move it the day previous.