I’m not sure how the goblin camps are set to spawn, but I would imagine right now they’re focused on x distance from the fire pit. At least that’s how it seems as I’ve met them. Yes, they spawn close to some of my buildings, but that’s sort of on me, as I never seem to put my fire pit anywhere near where I initially set up (I’m weird, I know!). I do know that there are really bad spawns out there, as @c00de mentioned, but even with the closest town I’ve had (around the corner in my current game), it just took a little bit more preparation. I’m running with three footmen who each started with a wooden sword and a shield as soon as I could make it. Also, what @martyrsvale said is right, that if you fiddle with your resources a bit, you can prolong that spawn for a few days until you’re more comfortable.
Ultimately @TekDragon84, there’s still a ton of growing and tweaking to happen. There’s a thread here that’s looking for feedback for the goblin campaign, which also might be a good place to share your findings so that they can look at it and tweak it if need be. 
Don’t give up on Stonehearth, but certainly take a break if you’re frustrated.