REPEATFEED RᴇᴘᴇᴀᴛFᴇᴇᴅ RepeatFeed


#181

image

So much for that, I guess. Whatever dm/streamer.cpp:215 in r834 is, it’s currently ruining my mood.

After the mod workshop/repository discussion I’ve had a while ago, I’ve said that most stuff could be done if there were an extremely limited HTTP API. Well.

image

Ask and you shall receive. By compiling my own lua-5.1.5.dll, I’ve managed to sneak in HTTP functionality into the core game. This allows for a few goodies, like this for example:

The example mod I’ve written queries the Discourse’s latest.json, then fetches the avatars as well and displays them in a list. It was relatively easy to write, and the API technically supports more (as seen above - POST’ing to sources as, say, JSON or using form-data would be possible as well).

Since this is technically breaking the sandbox (hard), I’ve thrown out several of the more dangerous suspects out of the lua libraries - then later found out that in newer Stonehearth builds, they’re not even loaded anymore. So much for “My lua can HTTP and is safer!” that I wanted to use as a slogan. So it goes.

It also technically prevented me from adding my http library’s method and objects. So I’ve just thrown the loading code into the base library loading code, and there it was again.

image

It’s been quite a while since I’ve done anything C/C+±ish related, so I’ve had quite a few issues at the beginning. For example, doing throw new std::illegal_argument(...) and then wondering why the catch didn’t work. Hint: “new” is not required and I was throwing pointers around.

In total, there’s quite a bit of code involved and I went through approximately three ways of doing HTTP until settling on libcurl (again). In the end, I’ve had 245+419+354+94 lines of .cpp (including everything) written - and funnily enough, the lua binding (which was written by hand) was the biggest file, whereas the whole HTTP manager was the smallest one.

Because HTTP’ing is nice, but being able to HTTP anywhere is kind of a no-go these days with privacy and security concerns, I’ve utilised a whitelist approach. In the game’s base directory, there should be a http_whitelist.json which contains a list of rules that define what kind of calls are allowed. The idea was that, if a mod requires some phoning, the user would need to manually adjust the file him or herself, therefore kind of “acknowledging” it. The whitelist I’ve used is as follows:

{
   "allow": [
      {
         "hostname": "discourse.stonehearth.net",
         "schemes": [ "https" ],
         "methods": [ "GET" ],
         "paths": [ "/latest.json", "/user_avatar/" ],
         "ports": [ 443 ]
      },
      {
         "hostname": "avatars.discourse.org",
         "schemes": [ "https" ],
         "methods": [ "GET" ]
      }
   ]
}

The format is also simpler and allows just hostnames, in which case all methods are allowed. Paths were must-start-with (therefore, adding “” or “/” or just not specifying anything allowed all). Attempting to perform a query against a non-whitelisted source throws a lua error. Because I wanted to have some fine-tuning like methods and stuff, I could have written an own format. Or just include a 584kB json.hpp and use JSON. :see_no_evil:

The HTTP requests themselves could be somewhat sophisticated (or at least, had more options available than most HTTP libraries in games I’ve seen):

static const luaL_Reg instance_methods[] = {
	{ "get_url", http_get_url },
	{ "set_url", http_set_url },
	{ "get_method", http_get_method },
	{ "get_hostname", http_get_hostname },
	{ "get_scheme", http_get_scheme },
	{ "get_path", http_get_path },
	{ "get_port", http_get_port },
	{ "get_status", http_get_status },
	{ "get_response", http_get_response },
	{ "get_response_base64", http_get_response_base64 },
	{ "get_status_code", http_get_status_code },
	{ "get_response_headers", http_get_response_headers },
	{ "get_response_header", http_get_response_header },
	{ "set_request_header", http_set_header },
	{ "set_callback", http_set_callback },
	{ "is_completed", http_get_completed },
	{ "send_get", http_fetch_get },
	{ "send_post", http_fetch_post },
	{ "send_post_form", http_fetch_post_form },
	{ NULL, NULL }
};

So you could set request headers and read response headers, pass some payload, and even set the content-type for the payload and basically send whatever you want. This could easily be expaneded; for example, you could write a (static) lua method that would have taken a (completed) request, checked it for validity (headers/source url/content), and then wrote the specified .smod - effectively writing an own repository client in the game. That would actually have been the next step: writing (but not really hosting) an open-source repository that could be accessed with this client, enabling auto-mod updates even for non-Steam-users.

At some point, I’ve also misunderstood the API for lua_pcall and effectively managed to create a stack with -1 items on it. Debugging that was fun, because it only was an issue when two requests at the same time finished.

Speaking of requests, in order to be able to properly call the callbacks, I’ve had to “get back” into lua somehow. For that, http.update() is doing actually work on the game. Because I don’t have a proper line to the game’s internals, I’m just using the gameloop-callback - except on the client, where I’m using the good ol’ frame tracer hack that I’ve discovered long, long ago and put in jelly at the time. It still works, four years later. However, this means that requests aren’t handled when the game is paused (on the server side; client works all the time). That’s a bit sad, and could probably be circumvented by designating a client and just telling it to call a method on the server as often as possible in order to poll it that way… but that’s really nasty.

At one point I also forgot to free resources properly, so I’ve had a gigabyte of garbage in my memory after roughly 30k requests. That was fun to debug, but afterwards, I’ve managed to leave no garbage (neither in lua nor in the application itself) lying around, even after 120k+ requests, so that worked nicely considering how much magic I’ve had to do for the whole thing.


Soooo, where did it go wrong? Unlike C#/.NET, where everything is nice and dandy and somewhat standardised (for a certain degree of “standardised”), C/C++ is pretty much chaotic. There’s ultimately a ton of knobs to turn and I don’t have the necessary skill to figure out which ones. If something is wrong, you’ll just notice it way further down the road. An example of a few knobs to turn:

  • The version of lua to use - well, since the file was called lua-5.1.5, this was kind of obvious. Hopefully.
  • The C-runtime to use. If you pick the wrong one though, things are usually over really quickly.
  • The version of the toolset. Because I don’t have anything installed but VS2017, I’ve used v141,
  • Exceptions or longjumps? lua supports both, depending on whether it’s compiled as C or C++. As far as I can guess, Stonehearth expects longjumps here.
  • Patches to lua. Just digging around the DLL revealed cocos, so I’ve patched my lua with coocs as well.
  • Probably a few dozen #ifs that could be used to further change the behaviour of lua.

SH itself uses luabind for the bindings, so I’m not sure if there’s additional patches or something required for it, nor what its settings are. I mean, if a dev chimes in and tells me the required configuration for the lua dll, I can surely continue tinkering on this a bit. Until then, it’s working really nicely until you start a game and the AI starts doing stuff - I think something with the coroutines is upsetting it.

Or maybe it’s just streamer.cpp. I’ve never understood this whole streaming business, really, what’s so interesting watching somebody play a game?

For those interested in the API bits, here’s the companion mod that had the library extensions and the Discourse example:

httplib.zip (5.1 KB)

Because the lua dlls literally crash the game, I can’t really release them with a good conscience (unless somebody really wants to make a fancy main menu I guess?). Because I haven’t written C++ in ages either, the code is also nastly, so I’m not sure if publishing it would be a benefit for the world. I mean, look at this. Seriously:

static const char HttpManagerPtr = 'H';

#define LUA_HTTPREQUEST "HttpRequest"
#define LUA_HTTPMGR "HttpManager"
#define LUA_HTTPMGR_PTR (void*)&HttpManagerPtr

#define CHECK_ARG() HttpRequest* request = *reinterpret_cast<HttpRequest**>(luaL_checkudata(L, 1, LUA_HTTPREQUEST))
#define GET_MGR() lua_pushlightuserdata(L, LUA_HTTPMGR_PTR); \
lua_rawget(L, LUA_REGISTRYINDEX); \
HttpRequestManager* mgr = *reinterpret_cast<HttpRequestManager**>(lua_touserdata(L, -1)); \
lua_pop(L, 1);

static int http_new(lua_State* L)
{
	GET_MGR();

	HttpRequest* req = *reinterpret_cast<HttpRequest**>(lua_newuserdata(L, sizeof(HttpRequest*))) = new HttpRequest(L, mgr);
	luaL_getmetatable(L, LUA_HTTPREQUEST);
	assert(!lua_isnil(L, -1));
	lua_setmetatable(L, -2);
	return 1;
}

static int http_delete(lua_State* L)
{
	CHECK_ARG();
	delete request;
	return 0;
}

#define HTTP_GET_STRING2(strName, strMethodName) static int http_get_ ## strName (lua_State* L)\
{\
	CHECK_ARG();\
	const std::string& str = request->## strMethodName ();\
	lua_pushlstring(L, str.c_str(), str.length());\
	return 1;\
}
#define HTTP_GET_STRING(strName) HTTP_GET_STRING2(strName, strName) 

#define HTTP_GET_STRING_NIL(strName) static int http_get_ ## strName (lua_State* L)\
{\
	CHECK_ARG();\
	const std::string& str = request->## strName ();\
	if (str.empty())\
		lua_pushnil(L);\
	else\
		lua_pushlstring(L, str.c_str(), str.length());\
	return 1;\
}

and that’s just an excerpt. At one point I got so fed up with WinHTTP and libcurl that I’ve tried to see if Stonehearth would accept a C++/CLI DLL (spoiler: it did). So basically, the once-C-dll was then suddenly calling .NET to do the whole HTTP stuff. It worked really nicely, but it felt “wrong” (and a bit boring). Plus, I know what a pain that HttpClient/WebClient/WebRequest can be, so I’ve decided to avoid what I’m doing all the time (namely, install-package Flurl.Http) and try to go the C++ way. Even if it’s ugly. And might not work in the end.

But for a short time, I’ve hacked HTTP requests into some game, and it was gorgeous. I mean, technically even more since I’m just full-on linking curl, but filtering access to it over the lua interface. Technically it would be somewhat easy to allow sending e-mails, or accessing FTP, or whatever else curl allows. But that would really go a bit too far, I think…


#182

Good god, man, that’s pretty hardcore! Sadly, I don’t think we can allow HTTP calls natively because then a mod could wreck all kinds of havoc, security-wise. :frowning: If you could give some practical use cases, we could try to come up with a compromise.

As for the streamer assert, that’s because an object is being destroyed while we’re trying to encode it to send over the network. We have a patch for Lua that allows pausing the GC to prevent this. While I don’t want to encourage you down this insane path of monkey-patching Lua, I’m attaching it to this post cause why not.

gc_hibernate.diff.txt (3.1 KB)


#183

Applied the patch, and now I’ve been able to play with it a lot longer than before - so I think it worked.

Let the tinkering begin…


#184

In the middle of my finals, I’m taking a little break to do a bit of reporting here. Although I haven’t much coded on this project in the past few weeks, there’s been a few interesting, new ideas floating around that could utilise a http library - most notably, the idea of utilising a masterserver/game directory service for multiplayer outside of Steam.

There’s also a whole new dimension of security that I completely forgot about, but which I realised when I thought a bit about what max said. So far, my whole security premise was merely based on “Prevent the game to access bad websites”/“Prevent the game to download bad mods”. That’s one dimension, but there’s another.

Assuming we have some sort of workshop, or mod repository, how would you upload things? You would likely want to authenticate the user somehow, usually using some sort of access token, API key or JWT. But here’s the problem: What if badmod is actively taking these credentials and uploads them somewhere else? It doesn’t even need to have a special kind of website to do so; it could utilise the comment section of a mod, for example. You could even go completely nuts and encrypt it with a public key, embed it in a file (e.g. PNG, JPG), then have an external service download these files and extract the API token out of it again. The possibilities are endless! And quite interesting to think about. This is a scenario that’s probably never going to happen, but I think it’s still important to prevent it. Just because something is unlikely doesn’t mean it’s not going to be abused.

Steam doesn’t have those issues at all - it’s handled by the game, your Steam login is transparently handled by Steam itself.

So, let’s introduce a new concept: A “keystore”, keyring, keychain or whatever you want to call it. The idea is simple: Give each mod the ability to safely store and retrieve credentials and similar, delicate data. The lua library, written in C, can identify the mod by looking at the file the callee was in. In lua, this isn’t necessarily possible, because one could simply overload debug.getinfo (trust me, I’ve been that person for another game in the past - lots of fun) and spoof the identity. However, you can’t spoof the C method lua_getinfo as easily.

Now, the idea would be that a mod could simply do something like local token = http.keystore.get('access_token') and then have access to the token. As long as the mod itself does not leak the token, it’s impossible for other lua files, mods or methods to get it.

In normal lua, that statement would be wrong. You could use the debug library to get local variables, upvalues, or monkey-patch vital functions for example, to get the token. All these methods have been removed as part of the MP update however, and are therefore not available anymore in Stonehearth - so saving the token in a local variable seems safe. I actually have not played with all possible attack vectors in here yet, but so far it looks pretty promising.

The only problem then is how to get the initial token. Let’s assume we have a GUI for it, then JS knows about the token at some point, and a lua handler knows about it. Both could be intercepted, if an evil mod was already listening in. So that’s not a good idea. Unfortunately, I don’t think it’s possible to completely “isolate” a GUI component. Even if there were, there’s still the possibility that the backend was hijacked and the key would be sent to an attacker. Ultimately, there’s no way to prevent any of this - unless it were possible to add methods into the Javascript space, that call back into C/lua/a mod directly, but that’s unlikely to happen. All that would be required, engine-wise, would be the possibility to create a keypair for a certain mod, then query it (i.e. JS: let key = createKey('modname', 'keyname'), then radiant.call('modname:set_credentials', encrypt(key, this.username + '##' + this.password));, and in modname/callbackhandler.lua something like function ModHandler:set_credentials(session, response, encrypted_secret) local secret = decrypt('keyname')). However, if we were to do this in the public JS namespace, somebody could - again - simply hijack createKey or encrypt and abuse it for themselves. To be super sure, you would need to be able to redirect the client to a completely different HTML page, that has no other mods injected, then enter your stuff, then go back again. But even then, somebody could simply override the library JS used to communicate with Stonehearth… so you would need to ship it with a barebone stub of radiant’s JS library, in order to avoid that.

So, the only really confidential way of adding external secrets to the game would be to add them outside of the game. That adds new questions to the game, such as “where?” and “how can we make this somewhat easy for the user?” Messing around with JSON files would definitely be a no-go in my book - too much can go wrong, and “post your JSON” doesn’t quite work if it’s full of potentially confidential tokens.

The best way to do this would be, I suppose, to have the secrets as physical files on disk. Use some fancy naming schema (i.e. modname##secretname.dat), and filter by secret name. A mod repository or what not could simply say “Download this file, put it into Stonehearth/keyring, and restart the game” - which is about the maximum I would say a user can be entrusted with.

Is that a satisfiable solution? Not really, no. It’s probably quite safe, but also extremely unwieldy. Having proper engine support could help a lot here, but also create a maintenance nightmare. I think some sort of keyring/keychain - along with a rewrite of the http library, since reading/hijacking HTTP requests of other mods is also a no-go - could help here, but even then, I’m at odds. What happens if someone were to simply monkeypatch the HTTP library itself? There’s not a good way to prevent that - short of actually installing a metatable and/or vital lua functions (require, setmetatable, getmetatable, debug.getinfo).

This is a nightmare.


#185

I think I’ve finished writing the readme.md for the whole thing. It’ll be probably the last thing that I’ll do with Stonehearth, but I feel like it ought to be finished and released, even if I can’t imagine anyone using it.

Most of it was written in the dead of night (like right now, at 3am), so sentences might be weird and the English might be borked. But in general, this is what I imagine an absolute minimum of an API documentation to be.

I’ll make a proper build tomorrow, test it a bit, then release it. I’m not sure if I should release the source code or not. On one hand, there’s a bit of a licensing wumbo jumbo (lua, coco, the patches provided by @max99x, curl), on the other I feel like opening the gates of hell by showing how to do this stuff easily.

Then again, I already wrote about what I did, so it can’t be that bad. I don’t think the library will take off, so impostors are probably not an issue either. Maybe somebody can take a look at my C++ and say “Yepp, that’s a C# coder that wrote that thing, sure enough” and maybe get a giggle out of it. Or a brain aneurysm, depending on his attitude and my quality of code. Bit hard to guess, but I’ll take my chances.

If anyone has any nice idea (besides the discourse-in-game one) for a demonstration, I’m all ears. It just shouldn’t require too much work, as I don’t plan to invest a lot more into this.


#186

-snorts through nose-


#187

Let’s talk about MIDI.

In the last millenium, digital games often used MIDI music, instead of say OGG or MP3. Instead of containing the audio data like those formats, MIDI is more like a score: It contains information about which instrument should play what note at which point in time, for how long, how loud, and so forth. Because this uses very little data, each note in a midi file just about 2 bytes or so, they were easy on the memory. I’m talking about times where having a 40 GB hard disk drive was considered posh. There’s also a slight difference between MIDI streams, such as an attached keyboard, and MIDI files. The first can be done event-wise with APIs, the latter needs to be parsed manually (usually), which means some kind of library. In my case, I went with NAudio.

If you’ve ever come across any Synthesia video, congratulations, you’ve likely encountered a MIDI. They usually sounded the same-ish, unless you had your own MIDI synthesizer (software) installed, you were using Microsoft’s. That’s why a generation of games, from RPG Maker to small Indie games, all kinda sound similar.

Because you have information about each note, each instrument, this makes it very interesting for me. I’ve always found synchronised music shows kinda attractive, like the ones that pop up around Halloween at some point.

So let’s go on a journey. The year is 2016, and I’ve found out that I can actually interface with Windows’ MIDI API relatively easy within Unity. So let’s have a real MIDI device input stuff in Unity, and then simultaneously output something through Microsoft’s MIDI synthesizer. The result was a very successful concert of mine:

Now, let’s plug in a MIDI file for once. The player was still kinda broken, because the format is non-trivial, so bits sound off (compared to the original midi that I’ve found).

Add some minor script improvements to it…

That already was kinda neat. Now, I’ve tried some shenanigans. For example, what would songs sound like if we took the notes, arranged them by count, removed the bottom half and re-mapped them to their nearest surviving brother? The result is a perfectly balanced paradise, as all paradises should be.

Using the same approach, I’ve also made something that allowed me to map songs to Factorio semi-automatically, which is nice. However, this kills the performance. Fast forward to 2018, I’ve then tried to resurrect the idea, this time using self-rigged models. They’re super poorly rigged, and edited by yours truly, who barely knows that Blender is also a 3D program and not just a kitchen utensil. So here we go, the first attempt at mapping MIDIs with “actors”:

The premise was simple: I parse the MIDI and save it in a more convenient, local format. I keep the original midi codes intact however, so I can re-create the song accurately - something I didn’t do with previous implementations. The parsed song is then split in channels, which can be accessed by an actor.

An actor is a simple behaviour which gets the song and a channel as input, and supplements some data for other classes. For example, it calculates the current “pitch” by taking the active notes, averaging them, and then does some smoothing so the value doesn’t jump around too much. Based on these actors, we can now define sub-actors, such as the “singer” on the left. The singer just takes the base actor’s value and throws it at the animator, which in turn interpolates between the “mouth closed” and “mouth opened” animation. We also have the “mover”, which interpolates between two transforms (e.g. position/rotation/scale). This was a nice experiment, so let’s skip to the latest iteration.

There’s some small changes here compared to the previous video. Alright, I’m kidding, there’s a lot of changes. Using Unity’s Timeline feature, which is still desperate in need of some loving touches, and some more instruments (that I had to severely change in some cases just to be able to do the animation somehow) and animations. There’s a lot to be learned here.

First, rigging and modelling is hard, and I still can’t stand that stuff. The basic rig worked nicely for basic stuff, but as soon as you try more complicated things, it breaks down. Additionally, if your rig isn’t humanoid, then Unity is somewhat unpleasant about it if you try fancy things, such as IK. I’ve not done it here because the results would be messy. But basically, a good rig should conform to the model, and this isn’t the case here.

Then, animations are painful, especially if you don’t know how to play the instrument. I had to improvise a fair bit with the drums, and I think anyone who plays the drums and sees my implementation will probably attempt to behead me on the spot. I’m sorry! I tried my best! Google can only get you so far. But to be fair, it’s really hard to play a conventional drum with such tiny arms. Here, a more detailed model, which really fits the style and limitations, would be much better than any real-world example. Hence, the lesson learned: A good design fits functionality and aesthetics, and if it seems somewhat realistic on top of it, then that’s a nice bonus. Again, as I’m not a modelling guy, I had to take what I could. Also, it’s important to set the properties right. In the video above, I had a wrong exit time for one drum animation, which results in it being played twice per trigger instead of once, which looks kinda out of place.

Furthermore, trying to do stuff on-the-spot results in weird behaviours. The video has a slight delay between video and audio, because the video is just evaluating the MIDI as you go, with some pre-warming delay in order to make it “just in time”. Well, it doesn’t, and it’s somewhat noticeable if you really focus on it. In other versions, I’ve fixed that by pre-calculating a complete plan on how to “play it”. I’m creating an animation curve and set the keyframes, then I simply evaluate the animation. That way, I get a perfectly synchronised animation, but it also means that more complicated things that require more than just an interpolation between two values, get more difficult. Animation curves that are 4min are also kinda immutable in Unity, as the editor simply refuses some operations (like selecting a keyframe) by pretending to be dead.

Also, god drums are annoying! Most of the instruments can be faked pretty easily by interpolating one or two animations, but drums are something else. I think in order to have a fully working, nicely looking drumset you would need a pretty beefy state machine to handle it. That’s not impossible to do in Unity, of course, but would require some serious amount of animations and coding in order to evaluate the proper state of it.

I’ve already modified the guitar to use an animation curve approach, and the result does look much better. Like pretty much anything in this little side project, there’s possibilities to improve pretty much everywhere. I think it would have been nice to be able to do something like that in Stonehearth as well, but the animation architecture doesn’t quite allow for it.

I don’t think I’ll continue this for now, or maybe again in two years. I got it to a point where I can say that it’s pretty nice. To improve it now, I would have to spend a fair amount of time for rather small details, and while I love those, I’m not sure if it’s worth it, especially since a limiting factor is still the models/animations, which I know I can’t really improve anyway. As MIDI files get rarer these days, and let’s not start talking about the copyright nightmare that YouTube can be, I don’t think pursuing this any further is worth the time.

So, onwards to new adventures. And if you’re interested in the evolution of Megabawkania, here’s the playlist with all parts.

And in case you’re wondering why I’m re-using (or abusing depending on your point of view) assets this much (for example, there’s a fish or two repurposed in the last video), it’s because I have limited access to assets, but I like working with them. They’re simple enough to be used with almost everything else I can imagine. So hooray for that.