I know the feeling, I too have a somewhat quite good hardware (also a 1070 and an i9 9900k) but start facing similar issues after 40~50 hearthlings if I try to play on speed 3 or 4.
It’s not that much about our hardware but about SH’s optimization and threading, etc… Unfortunately the game suffers a bit from some early decisions (like building a custom engine, no easy task) and they didn’t have enough time to polish and optimize it as it would’ve been in an ideal world.
So this is sort of an issue that everyone faces in SH, regardless of their machines (some people start feeling that as early as 20 lings or so, so consider ourselves the lucky bunch)
We might see some improvements over time (as hardware improves without changing its paradigm) so it’s funny to think that SH is a game that will get better over time but as of now, I wouldn’t feel bad. Just try to enjoy the minimum speed and have fun with your lings slow speed is great for building or just admiring them working
And if you would like to make your town feel more bustling without sacrificing much performance, I’d recommend expanding on traveler beds, I have like 25 traveler beds and the town looks so amazing with so many hearthlings running around - and they’re quite cheaper in terms of performance