If you do not click either option when the goblin leader messages you, you can instead just close it and ignore it. The goblins will never attack.
Reporduce:
Start a game
When the goblin talkes to you, close the window
This may be just missing functionality, but figured I would report it anyways, because of some oddities that tie in to it.
When I did this, I didnt recieve any of the new shop caravans. I still got the old fashioned traders and such, but none of the new stores. Then, when I finally did talk to him and told him to eat my shorts, I got an error message, but was able to continue playing. The goblins never sent anyone to attack after that, so I just wandered over and killed them.