Unfortunately, all the goblins will do is hold onto your stolen supplies for later; they won’t use them to upgrade or anything like that. As you get further into the goblin campaign they’ll start sending bigger encampments and if multiple camps exist then they’ll send patrols between them, so it does look and feel like they’re progressing if you leave them to expand; but this happens independently of anything else they do or that you do to them – e.g. if you kill all the workers at a camp but leave the firepit, goblins from other camps will still come out to patrol it.
You can change the difficulty mid-game using commands or the debug tools, I’m not super familiar with how to do that but there should be a guide on these forums somewhere. Another popular tactic is to settle in an area with limited access, and then build a wall or dig a moat in order to completely cut your settlement off from the rest of the map until you’re ready to face enemies. Be aware that this can lead to performance issues if many enemies “stack up” outside your defences though.
On normal difficulty, you can easily defend a settlement with a couple of footmen in the early-game as long as you control access (i.e. create choke-points like walls with limited entry gates.) Once you get to the point that goblin wolves attack you then you’ll probably want a cleric, and to start working towards a knight. By the time you face Ogo (whether you choose to fight or take the other path to learn about Mountain’s history) you’ll definitely want a knight, cleric, a well-equipped footman and an archer using spiky or fire arrows as your bare minimum. Those four will give you all the tools you need to disrupt the attacks of powerful enemies, and take control of battles. Past that point, it’s up to how you prefer to play – a firing line of archers, an immovable wall of knights backed up by clerics, a swarm of footmen… all work pretty well if you take the time to learn how they fight.