Multiplayer lag performance issue (resolved... kind of...)

Support Performance
1

Hi,

Yes an other guy, with an other post about bad performance and lag. Like I saw, I’m not the only one. So I read a lot about that on this forum to find a solution… and found some usefull ones (reduce number of heathling, reduce number of stuff on ground, reduce number of owned stuff, make ladders everywhere…) that reduce bad CPU performance but I’m still having a barely playable game because of huge network_send percentage (70-90%).

So I’m making this topic, with hope, to send my save files and, also, to say that I LOVE this game, guys you did an amazing work. More I play, more I enjoy it (70 hours now) and learn how to manage the engine and the IA, it’s really clever, funny and interesting. Well done ! And the game is so stable, no crash at all !

The little cloud in that shiny blue sky is the lag :frowning:

Player 1 : host - desktop PC :
Intel i5-3470 OC @3.8Ghz
16GB RAM DDR3
Nvidia GTX 1080
Samsung SSD 850 EVO
1Gbit/s optic fiber bandwidth, LAN connection no wifi

~50-80% CPU usage, 40% GPU usage, ~5-7Mbits/s upload, 3GB RAM usage

Player 2 : guest - laptop PC :
MSI GT72VR 6RD Dominator
Intel Core i7-6700HQ 2.6GHz OC @3.2Ghz
32GB RAM DDR4
Nvidia GTX 1060
Kingston SSD
1Gbit/s optic fiber bandwidth, LAN connection no wifi

~30-50% CPU usage, 40% GPU usage, ~5-7Mbits/s download, 2GB RAM usage

Game version on both PC : W10 64 bits, Stoneheath v1.1.0.r949 x64 with ACE v0.9 - no mods

We play in multiplayer last AAA games or Battlefield without any performance issue on both PC so I don’t think it’s a hardware issue… ? Still we are above recommended requirement according to https://www.stonehearth.net/some-technical-stuff/

Tweaking the ingame graphic settings doesn’t seems to do anything on FPS or LAN. The GPU work more or less harder (15% min - 60% ultra), but the FPS are still the same (25-35 FPS). Did you said CPU bottleneck ?

We didn’t try “32 bits” but read that it wont change anything. Will it ?

~5-7Mbits/s seems to be a cap in bandwidth ??? I think our hardware can support much better than that, maybe uploading more than 5Mbit/s will be a solution ?

When game paused, the network drop (normal :slight_smile: ), the GPU gain ~20% of activity and the game is much smoother.

I didn’t try a real game hosted on PC 2, but the loading time is way more reduced on this one… maybe a clue… I tried to copy my save from PC 1 to PC 2, the game start but impossible to join (it says server full). And the clan was switched by the way :frowning: I change some data in server_metadata.json for switching clan but it produces an endless load game…

So, the problems :

  • on dirty maps (stuff on ground everywhere, around 60 heatling on the map) : there is a huge CPU problem, very bad FPS performance (around 10-15 FPS)
  • on every map : the host use 80% for network_send and it’s really laggy

If you have any kind of solution… bonus question : is there a way to force the game using local multiplayer (we are on the same network) instead of using internet ? Maybe the bandwidth will be better… and big bonus : is stone road are faster than wood road !!! It’s seems not… ??

Here’s my saves :

  • 1546896757352 : Alternatives to
    messy map, 2 players, more than 20 hearthling each, bad FPS and LAN performance
  • 1547264586319 : Alternatives to
    clean map, 2 players, 20 hearthling tops each, good FPS (~30-40), not too much stuttering, LAN issue
  • 1547053513025 : Alternatives to
    messy map, 1 player, more than 20 hearthling, bad FPS (CPU issue ?)
  • 1547157942111 : Alternatives to
    clean map, 1 player, few hearthling, good FPS but still stuttering

And log file :

2

add saved files

3

Welcome to the Discourse! And, not to sound too snarky here, but congrats on having done some research and tried out things, and especially for taking the time to understand the nature of “simulation creep.” Clearly you’ve seen how many people don’t, and when it comes to offering assistance, there’s only so much we can do without very specific information about what’s going on. Thankfully, with the details you’ve given we (or the devs) can hopefully narrow down the issues at play here :merry:

It should be possible to use LAN only for local play – check the “how to play multiplayer without Steam” thread, I’d guess the instructions there should include LAN or at least put you on the right track.

When you mention that on PC 2 you get much faster load times, that makes me suspect that the read/write speed of the SSDs on PC 2 is significantly faster. On my own machine, the main bottleneck is HDD write speed and while there are some tricks to “soften the blow” (basically it comes down to trying not to have too many items updating at once), being a hardware issue there’s only so much I can do about it. I doubt that it’s the core cause of your multiplayer lag (although it may contribute), but if you’re able to upgrade your storage drives to have a faster read/write speed then it should improve the number of hearthlings you’re able to run smoothly.

All roads have the same speed boost, regardless of material.

For the fourth save file, you mention good FPS but still having some stuttering; that sounds like “sim creep” to me – the kind of situation where the computer spends so much time trying to clear a backlog of failed tasks that it runs out of time between ticks to complete the full task list. This is an incredibly vague issue because just about anything can cause it – too many tasks ordered at once, too many AIs to process, too many items to keep track of, errors/bugs/glitches, memory leaks… If you’re able to find a game log associated with that save, I’d recommend uploading it alongside the save. I’m sure the devs will be interested in such a save which should, by all appearances, be smooth and stable (it certainly sounds like you’re doing everything right in-game to reduce the workload for your computer.)

Good luck getting these issues sorted out :merry:

1 Like
4

Hi, thanks for you reply :slight_smile:

Here’s the log (all of them…) : Framadrop - Share files confidentially (first topic edited)

I will try a new game hosted on PC 2 when we finish this one, or when it will too laggy to play with. So, if we do that, I’ll put some report back here.

Hmm I don’t think the SSD on PC2 is better than one on PC1 but… maybe… I didn’t stress-test them. Anyway, the game don’t access the drive a lot while playing, I’m not sure it’s the main problem… ?

I’ll try Multiplayer without Steam?. I had this in my plan so, I will and report :wink: With an end-to-end (wire, router, bandwidth) 1Gbits/s network system, it make me sad to think it’s the bottleneck :frowning:

I forgot to say :

  • with wifi “On” it’s just unplayable (very laggy and crashy)
  • it helps the game, I think a lot, to pause it every 5-10 min for few sec to help him to breath, and to don’t use the 2x speed.
1 Like
5

Hello!

Well, in my opinion I wouldn’t compare Stonehearth with a AAA game. Especially one that is designed around multiplayer.

Multiplayer in Stonehearth. takes a lot of bandwidth. Like a lot… Think in the neighborhood of up to 10Gb/s. So your 1Gb/s is a bottleneck for the game. If you have a 10 Gb/s network at home, it should better on LAN. Basically, the game sends a lot of information to all the clients. This gets worse as you increase the number of clients and hearthlings. But at that point, you’ll actually start running into AI performance issues which will be based more on your CPU rather than your network.

Use less hearthlings, you will see better network send. I can comfortable play with about 30 or so hearthling between two players with one player being a host.

As for LAN, Steam will automatically connect you via LAN before it uses it’s Steam relay server. So you don’t need to use multiplayer without steam.

Also, running game on SSD will be faster mostly for loading. Once it’s loaded it shouldn’t be a major factor.

With Wifi On you will reach the network bottleneck faster so LAN would be the best way.

Good luck!

Edit: Let me clarify, even with 1 Gb/s network, the cpu would still need to be able to send the the reading and writing of the data. So technically, it probably won’t even reach the full 1Gb/s transfer rate. The point is the game sends a lot of information to the client.

The host handles everything and the client acts like a Video Player with the capability to send commands to the host.

6

I ran some test :

  • LAN speed from PC1 to PC2 : up 900 Mbits/s, down 700 Mbits/s
  • LAN speed from PC2 to PC1 : up 700 Mbits/s, down 850 Mbits/s
  • Internet speed on PC1 & PC2 (speedtest.net) : up 350Mbits/s, down 900 Mbits/s

No one, or maybe just a few, can do better than 1Gbits/s over Internet connection in France for personnal usage.And again, just a few, in France, can just ever dream about reaching those speeds… And for local bandwith, I think 1Gbits/s is far from necessary too.

If the game needs 10Gbits/s or more… sorry, but I’m affraid I’ve to say it’s a very bad technical choice… for french gamers anyway.

And why the game send around 7Mbits/s in game, with a 1Gbits/s usable network ? If I was laggy with 1Gbits/s sent, so, yes the network would be the bottleneck. But here, the game usage if far away from my computer can give.

How to force the game to use LAN connection without Steam ? I don’t know how to join a game without using Steam Friend List… I trust what you said (Steam use local if it can) but still, wanna try :stuck_out_tongue:

edit because your edit : yes the CPU is probably the problem (if we don’t speak about code refactoring and optimization)

7

I’ll just point out that in the 10Gbit/s example. There was 5 players. each player had 10 hearthlings. The host was sending data to those players.

I would make sure the game isn’t “messy” and I heard roads and optimizing item placement in game will help reduce CPU load which results in better performance overall.

Basically, edit the user_settings.json, below is an example setting for client

	"multiplayer" : {
		"server" : {
			"default" : {
				"max_players" : 1,
				"remote_connections_enabled" : false
			},
			"port" : 57093,
			"ip" : "0.0.0.0"
		},
		"remote_server" : {
			"enabled" : true,
			"ip" : "192.168.2.63",
			"port" : 57093
		}
	},

Below is settings for host

	"multiplayer" : {
		"server" : {
			"default" : {
				"max_players" : 1,
				"remote_connections_enabled" : true
			},
			"port" : 57093,
			"ip" : "0.0.0.0"
		},
		"remote_server" : {
			"enabled" : false,
			"ip" : "192.168.2.63",
			"port" : 57093
		}
	},
8

On Host I had "remote_connections_enabled" : false, tried true but didn’t change anything. Still “lost connection” on Client.

Anyway how to launch the client and tell him to connect to a multiplayer game without Steam Friend List ?? (You answered : just launch the game after Host create game and place the camp)

One thing more : without Steam, no ACE (and no mods) so the save is not compliant. I tried anyway, but got 75 errors :stuck_out_tongue: I tried with a fresh game, still “lost connection” (but didn’t try fresh game without ACE activated…lazy me)

An other way to test : launch the laggy save on PC2. But, how to backup the save and make it run on other PC ? I tried without succes…

edit : ok finally it worked (did the same thing than before… well…) and I think it’s worth, more laggy. And the game still send only 5Mbits/s

9

Do you have a firewall blocking 57093?

10

I think not, disabled firewall on both PC to be sure, and port forwarded in router…

11

I looked at the 1547053513025 save. A whole lot of the lag was caused by draining the waterfall landmark. In its default state, that landmark is in equilibrium with an equal number of wet and dry stones producing and consuming water. When the pond water is drained, this equilibrium no longer exists and water is constantly being added and removed. I suspect you would experience this lag regardless of whether you’re using ACE. Unfortunately, since at this point the water has already been drained, and the pond area is too big to be refilled (the same mechanic that prevents water spreading endlessly across a plain when it floods somewhere), you’ll have to mine into the mountain area to remove the wet stones (or use slice view and debug tools to shift+click and destroy them).

I’m unable at this time to do more in-depth performance profiling, but performance is certainly a long-term concern for ACE, and these saves may prove useful for identifying optimization opportunities.

12

Have you seen github.com/stonehearth/lua_profiler_scripts? I don’t know if it’s in a state that’s usable by modders, but that’s the main tool we used to profile the Lua side of the game.

13

Yes, thanks, I’ve been using that and gotten a lot of help from Linda with it. I was just running into an assert when trying to profile this game (right when profiling finished) that caused the profiling files not to be saved, and I don’t have the time to sort that out right now.

14

The water system is probably a good source of lag… but the game was very laggy before draining the waterfall landmark. But it’s good to know by the way, and thanks for your time. I’m glad if my saves and reports helps to make Stonehearth a better no laggy game in multiplayer.

15

Hi there, with more tests :

  • removing the wet stones doesn’t make any difference
  • after a while, the all water landmark was almost completely dry, but my irrigation system still worked

So, for now, I think the only way to play in multi is to have 20 hearthling max on the map (10 per players if 2 players), specialize jobs and trade a lot. I didn’t test this solution… but i’m sceptical anyway, I think after few hours the game is just unplayable :frowning:

16

I tried a new game again, this time with 10 hearthling max per players (2 players by the way) and the game is totally playable :slight_smile:

Player 1 focus on fighting, healing, building roads and mining. Player 2 on crafting. We trade a lot, but it works.

So for now, here’s my solution to play the game in multiplayer. And it’s an interesting way to play it by the way.

I’ll be able to play it with more hearthling maybe in few patches :wink:

17

The next stable update of ACE should improve online/multiplayer performance with some structural changes we did trying to reduce the huge “network_send” load. We still don’t know when it will roll out, but it might help you folks :slight_smile:

2 Likes
18

Good news ! Thanks for working (hard I suppose !) on this :slight_smile: