I am in the middle of creating a mod to enhance the headless server capabilities. The main feature I want to add is the capability to save the game. I figured the C++ code handles the saving logic. But I was hoping to find a solution to be able to trigger it from a remote client without being forced to run a ‘host’ client.
The lua code calls the ‘radiant:client:save_game’ event which is listened for by the client. The server doesn’t accept this event.
I put together that the ‘radiant:client:save_game’ triggers the save event handled by C++ which then takes a screenshot before it calls ‘radiant:server:save’ on the server.
I tried calling _radiant.call(‘radiant:server:save’) directly but it immediately crashes the server with no output.
Is there an existing solution in the C++ code to save game from the server without having to go through the client? Or is dll injection the only option to add the functionality?
I want to avoid need to run a full client and server. My goal is to run this on a computer with no graphics card so opengl won’t work.
I attached my github project, but all it does is enable multiplayer settings when you load headless server without a saveid and trick the client/server that the first player to join is the host player.
Is there an actual headless server support from the game, or is that something you have added/billed as such? I’m quite out of the loop.
Technically, if it’s a radiant call, it should not matter where it comes from, since it is likely treated all the same… technically. I’m not too familiar how multiplayer works, actually.
When you tried to call radiant:server:save, did you supply the necessary parameters as well? Maybe it’s missing one and that’s crashing it. If any client could call this, however, then it would sound like something’s borked… unless the client calls it in the server portion of itself.
I suppose what could work would be if you sent a message to the server/the player that is the server, and then run the save from his context. But like I’ve said, I’ve never looked at the multiplayer part of Stonehearth, let alone played it since, so this is just general guesswork.
So! there is actual already headless server support, but it is mostly on the C++ side of the code. You can see the hook for ‘radiant:headless:init’ in stonehearth mod.
To enable it, just modify your user_settings.json to something like so:
Now, there are two ways of running the game. The easiest way is by dropping a save game into the save game folder on the server. change the multiplayer.server.headless.saveid=“save_directory_name”
Second way, is to let the stonehearth mod auto generate a world which you change the options to by filling in multiplayer.server.headless.options.
The second way doesn’t enable multiplayer by default, so you need to run the code to enable multiplayer. My mod does that. Also, you need to start some stonehearth server services which get started if you place camp as a host. Since there isn’t a host, you need to add in code to start those services. I got around this by faking a ‘host player’ as the first player that joins. Then it caused some issues with building placement since building placement and saving is handled different if you are host or not.
Based on my “research”, the server and the client runs as two different process. Even running the game without headless creates a server process listening on port 57093 and the client connects to it. Just enable console and you’ll see it create logs that it running two different processes.
You can see indications of this by looking at functions in manifest. You need to specify whether the function hits the client or the server. Another indicator of this would be the lua console. There is a ‘server’ and a ‘client’ console. Both of which hit different things. If you ran _radiant.call on the server it will do something different than if you ran it on client based on mods/c++ code.
So ‘radiant:client:save_game’ does have arguments that you need to pass for it to work properly. If you don’t atleast pass in a string as a first argument it actually spits back out an error ‘expected string value for node’.
Using the same logic, you can try ‘radiant:server:save’ with or without arguments and none of it will fail. In fact, the server just crashes.
If you run a headless server and call the client save, it creates a screenshot in your saved_games folder but on the server side, it doesn’t create or do anything.
So I could definitely run the server as a regular client + server, load in a player as host and kill all of the player’s entities and then handle ‘reassign’ the host to the first player that joins. At that point, I would just have to do what you say which is save the game from the real host. However, this would mean opengl needs to be used and therefore defeats the purpose of having a headless server functionality.
At this point, after taking a look at the assembly code, I think there are missing arguments that need to be sent to the server.
Without having to reverse engineer the solution, I was hoping someone could just give me the info needed to be able to run that save.
edit: In fact, @RepeatPan, you just gave me a great idea. I could watch the tcp traffic and find out the exact data that is sent to the server. You are brilliant! I can just imitate the packets sent to trigger the save. A proxy server should work. Though a lua version of being able to save would be better, but this might work for my goals.
There’s also a third realm, the aforementioned GUI. Technically, server and client are at least different lua states, which share nothing directly. It’s possible that they are in different threads as well, but I somewhat doubt it. The new process you see is most likely definitely CEF, which spawns new processes for the rendering. I would assume that 57093 is the port for the GUI (since it communicates… or communicated using HTTP).
Sounds to me like it’s bugged.
Which makes sense… you wouldn’t want a malicious client so fill your server’s disk space up by spamming saves.
This is something I absolutely did not wish to imply and something I wouldn’t imagine either working or recommending… If a client can trigger a save on the server, then the logic is majorly broken, period. For me, there’s three possible reasons why it doesn’t work:
A) It’s bugged and that’s the reason it doesn’t work
B) For one reason or another, headless servers aren’t supposed to save/force-save
C) There’s a check somewhere that prevents saving from non-trustworthy sources.
In any case, crashing anything should be filed as a bug, because you shouldn’t be able to kill the game using lua that way.
If you really wish for a third-party way, you can do it the same way I’ve added HTTP support to Stonehearth. Make an own lua5.1.dll, provide your own patches/lua methods, and use those. That way, you can control it from lua, but have native abilities.
I figured the CEF + OpenGL runs on the same process which is why you can touch the GUI while in game. In addition, you can see it actually use GPU and if you look at the command line triggers you can see arguments related to running CEF for the process that uses GPU. There also another process that is ran which is the server.
When you run the headless server, it only creates one process, the one that runs the server. Though you had more experience playing with the mods. So I am just guessing here.
For sure, we don’t want the client to trigger the save on the server since it can have a malicious use case.
Oh, no. You didn’t recommend or imply it. I just didn’t think of it until I read your post. Having a proxy server would just make things easier. Rather than modifying the dll, I might be able to imitate the save trigger and the remote clients would never know or even have access to the functionality.
I am not a lua programmer. So I didn’t consider being able to swap out the lua dll library. This may have some good use cases. But I would still need to figure out the proper way to call radiant:server:save. Wouldn’t the save games still be handled on the C++ side. So if it is hooked in the client portion of it which doesn’t get executed when running headless, then it still won’t work.
Technically, ‘radiant:server:save’ is not in any mod code. It only shows up in the stonehearthd.dll. Technically no one is supposed to call it from lua. I only found it after poking around.
I found The Happy HTTP Hack (1.0.0). I can see some fun features being added with this http functionality! Awesome idea to create this.
After the multiplayer update, all bets are off I suppose. The architecture was bound to change. Before the multiplayer update, CEF definitely spawned a new process for itself that it used for… either rendering or command processing, I can’t quite recall. Stonehearth’s logic itself was contained in one process.
Well, lua’s C and not lua, so you’re good to go.
You’re right that you still need to call save. But like I’ve said, if the game actually crashes then it’s most definitely a bug. I suppose nobody looked at headless servers so far, which is why this probably went under. It’s possible that nobody has considered how save/load should work with headless servers, or it’s been a long time since.
Nope, you are right. It all runs in one process + a CEF process for the UI. I just tested by killing off the CEF process. The game still runs with AI and all still working. No commands work though.
Also, in single player or when connected as a “host” provider in multiplayer, there is no tcp traffic. If I connect as a remote server, there is tcp traffic.
looking at the traffic during multiplayer though I did find the right parameter for the radiant server save.
radiant:server:save*{“saveid”:“1544997715276”}
response:
{“error”:“radiant:server:save returned false”}
This gets triggered after the client creates a save folder and a screenshot. Anyway sadly, I wasn’t interested in the game sooner. I may just have to scratch idea to improve headless server if I can’t get saves to work. Hopefully someone on Radiant team could shed some light.
Hey this is awesome. So as you’ve noticed radiant:server:save has a check that make sure that the ‘host’ is the caller. The way we determine host is if clients.host.player_id in the server persisted metadata matches the callers player id. So you should be able to manually change the host by editing clients.host.player_id in server_metadata.json to reflect the desired host player id – the value is set internally when the host connects though, so this would only work for headless mode.
I’ll look at this today to see if it works, but you still have to mod in a way to trigger the save.
Shutdown should be easy, you can mod in a command that calls radiant.exit(0) – another option would be to listen for clients joining and leaving and auto-pause when no one is connected.
Oh! shoot! I have been so focused on being able to run headless mode with auto generation (without loading a save). I did not consider to check out if saving was possible after you loaded a saved game. I remember glancing at server_metadata.json and ignoring it completely. I will mess with the server_metadata.json when I get home.
I did notice radiant.exit(0) as well. I was considering just leaving the headless mode server up the whole time since I can just pause the game when my wife and I stop playing. However, after a 4 hour gaming session we noticed the network_send spikes up to 70% to 80%. This caused significant slow down when both players were connected. This goes away completely if there is only one player left in the game. Anyway, I figured if I could restart the server it might help.
In case anyone is curious, the slow down could be caused by various reasons but it definitely goes from 25% network_send to 80% network_send over the span of 2-3 hours. Re-connecting to the server helps drop it back down to a more playable rate. The only other mod we run is ACE mod.
I have been testing this on an c5n.xlarge AWS instance with no elastic graphics attached because I figured AWS would have the best bandwidth network performance compared to my subpar gigabit router. The instance runs it pretty well. c5n.xlarge runs the server at 30% cpu and barely any memory usage. I tried to run it on c5n.large but the cpu usage maxed out to 100% after an hour or so.
How recent is your version of ACE? Over the past week I’ve made some significant performance changes so the only lag that should be happening is when large bodies of water (with lots of fish/plants) are changing volume (e.g., via wet/dry stones).
It’s Bolune. I always grab the latest version nightly since I’m actively working on it.
Also, yesterday, I set the water setting to a higher number 5 or 6 I think. Not sure if it affect it any.
My wife likes to dig into lakes to watch the water level sink…
It may be ACE affecting it since we always make our town next to water for the above reason.
This setting (if the settings code is working properly, which it might not be) should only show up for the host, because it will only take effect for the host. This might not work properly on a headless server. Do you notice a difference playing with normal hosting compared to headless?
I overridden the host player_id manually by overriding the _radiant.sim get host player id function and the _radiant.client is authenticated as host. Keep in mind, this mod is currently only for my own personal usage so I wouldn’t recommend this hackery to trick the server and the client to think a player is the host for public usage. Though it sounds like Team Radiant did a good job locking down possible vulnerabilities from client to host.
Anyway, I honestly, don’t notice or can’t tell the difference. Based on the code, in ACE, it should properly trigger the update and change the setting. But I haven’t played Stonehearth enough to see the difference. Is there a telltale to indicate the difference?
Heck! I finally played my longest game which was 4 hours and got a shepherd for the first time! It was super exciting to see my sheep run across the map to get grass.
I have spent more hours looking at all the lua and assembly code than play this game.
Settings are a little weird because they actually get set separately by each player, but if you get them on the server, you’re getting them for the host. If you get them on the client, you get it for that client, so for anything that’s related to rendering or things that affect the UI, that’s probably going to be accessed on the client and so it doesn’t matter. Things like Auto-Loot, I had to patch in the proper checks to consider the individual player settings, since that’s being checked on the server.
This setting is similar in that it’s checked on the server… you can verify that it’s being set properly by running radiant.util.get_global_config('mods.stonehearth_ace.water_signal_update_frequency') in the Lua server console.
I’ll mess around with the water_signal with headless server sometime after I look into the pet tasks.
For anyone that cares, you can save to the server once you tell the process that you are the host. This can only be done on server side by loading a existing save game with a modified server_metadata.json.
^ use this technique to set the host player_id on the server process. Then you can do the saves. You can even reload the save using _radiant.call(‘radiant:client:load_game_async’, ‘save_name’);
However, to actually create the same the folder must exist first. ‘radiant:client:save_game’ triggers the first part of the save where it creates the screenshot and the folder based on the first argument passed. This doesn’t happen on the server side.
First, create a folder on the server.
Second, call ‘radiant:client:save_game’ on the client with the first argument as the folder name and the 2nd argument will be the metadata.json content. Examine the stonehearth UI’s code for saving for more details.
Saved! then on client you can call ‘radiant:client:load_game_async’ with the first argument as folder name to reload the save on the server process. This will disconnect the client so you will need to restart the game to reconnect. Sad! Would be awesome if we could reconnect without having to restart our client.
