The Happy HTTP Hack (1.0.0)

The Happy HTTP Hack

Ladies and gentlemen. I understand that you have come tonight to bear witness to my journey on an insane path of monkey-patching Lua. I regret to announce that this is not the case, as instead I come tonight to bring you the codified recreation of the end of the world.

HHH is a patch for Lua that allows Stonehearth mods to do HTTP calls to remote endpoints. This allows the game to interact with web services, because we all know HTTP and the cloud are the future.cough

Using HHH is pretty straight forward with the included httplib.lua:

local http = require 'httplib'

if not http then
   return -- HTTP is not available
end

-- Fetch the resource. Make sure the endpoint is whitelisted!
http.get('https://my-awesome-web-service/feed.json')
   :on_success(function(ctx)
      -- Feel free to do whatever you want in here. As example, let's get the response content:
      local response = ctx:get_response()
   end)
   :on_failure(function(ctx)
      -- Uh oh, something went wrong.
      logger:error('Could not fetch %s: HTTP status %d, error %s', ctx:get_url(), ctx:get_status_code() ctx:get_error())
   end)

Download v1.0.0

Included is everything you need to get started: The two patched lua-5.1.5.dlls and an example mod that loads the latest topics from this Discourse, sends it to the GUI, and displays them there along with the avatars of the last posters.

Simply copy the content of the zip over your base game directory - not your mods folder! If done properly, Windows should ask you to overwrite lua-5.1.5.dll twice. Keeping a backup of the original files is recommended if you wish to go back to the official Lua.

What’s different in this Lua dll of yours?

It’s built completely from scratch, as I had no access to Stonehearth’s original Lua sources. Thankfully @max99x helped me here and there with a few missing functions, so that overall the dll should do mostly the same. In fact, there is a one-in-a-million chance that it actually runs somewhat (probably in the unobservable realm) smoother, as I’ve fixed one (1!) undefined behaviour that happened in one of the patches, but that’s mostly just semantics.

My patched DLL comes with a HTTP client that can open HTTP and HTTPS connections (technically, FTP, SFTP and whatever else curl offers would be possible, too, but for safety reasons, only HTTP is enabled for the time being), as well as lua bindings for it. To top it off, there’s a proprietary (read: not the one SH uses) JSON parser included to parse the whitelist.

To be able to actually use it, I’m hitchhiking on some other calls that Stonehearth necessarily needs to make in order for Lua to work.

Additionally, I’ve removed several bits that could be used to do bad things, such as the io and debug library. Turns out, those have already been removed as part of the multiplayer update, so they’re gone… twice now, I suppose?

On the topic of Security

You might ask now, isn’t this kind of insecure? Well, yes, it is! There’s a good reason that Stonehearth doesn’t allow HTTP connections out of the box. Lua itself is a sandbox (let’s ignore the fact that Lua has native support for binary modules), especially in games. By allowing outgoing requests, we’re breaking the sandbox.

In addition, you’ll need to trust the .dll that some stranger made. This is inherently unsafer than just any mod, because if I have messed up somewhere, bad people could do bad things. The chance for that is slim, but it’s not exactly non-existent.

If anything, this is a proof of concept. Depending on the interest generated and the use cases, there might be additional development, and the whole source code might go Open Source as well.

Any mod that is requires this library will also require its users to download HHH, as it cannot be distributed over the Steam Workshop.

To get some level of security going, the user needs to maintain a whitelist himself. This whitelist cannot be overridden, mixinto’d or anything else, as it exists completely outside of Stonehearth’s realm of influence. Details on the whitelist are on the GitHub README.

More examples!

Posting some JSON

If you wish to control what your request body will be like, simply pass a string (containing the data) and optionally a Content-Type. In this example, I’m posting some simple JSON and specify the Content-Type as application/json.

http.post('https://my-awesome-webservice/api/search', '{ "query": "sheep" }', 'application/json')
   :on_success(function(ctx)
      local response = ctx:get_response()
      -- Feel free to do whatever you want here. Parse it, print it, send it to someone!
   end)
   :on_failure(function(ctx)
      -- Uh oh, something went wrong.
      logger:error('Could not fetch %s: HTTP status %d, error %s', ctx:get_url(), ctx:get_status_code() ctx:get_error())
   end)

Posting a form

Forms are posted by passing a lua table. Both keys and values of the table must be strings, or stringify-able.

http.post('https://my-awesome-webservice/api/search', { query = 'sheep' })
   :on_success(function(ctx)
      local response = ctx:get_response()
      -- Feel free to do whatever you want here. Parse it, print it, send it to someone!
   end)
   :on_failure(function(ctx)
      -- Uh oh, something went wrong.
      logger:error('Could not fetch %s: HTTP status %d, error %s', ctx:get_url(), ctx:get_status_code() ctx:get_error())
   end)

Reading the response

There are multiple ways to dealing with a HttpRequest as is passed by the on_success and on_failure callbacks. The most interesting properties are:

  • HttpRequest:get_status_code(): Returns the HTTP status. Defaults to 0 in case the request wasn’t started yet, or there was a transport error.
  • HttpRequest:get_response(): Returns the response content as string.
  • HttpRequest:get_response_base64(): Returns the response as base64 encoded string.
  • HttpRequest:get_response_header(name): Returns the value of the response header with the name name, or nil if the header was not present.
  • HttpRequest:get_error(): Returns a string containing the error (if any), or nil.
  • HttpRequest:get_error_code(): Returns the libcurl error code associated with the error. Defaults to 0, which is OK.

More advanced usage

It’s also possible to go barebone on HttpRequest. In this case, usage of httplib and especially http.prepare is still recommended, because it’s still making things somewhat easy.

In this example, I’ll perform a PUT request on a resource, using a custom request header (Authentication) and as payload some plain text.

local req, promise = http.prepare('https://my-awesome-webservice/api/content/12')

req
   :set_method('PUT')
   :set_request_header('Authentication', 'Bearer XYZ')
   :execute('Baaaawk!', 'text/plain')
   
promise:on_success(function(ctx)
      -- Same as the other examples.
   end)
   :on_failure(function(ctx)
      -- Same as the other examples.
   end)

And more!

The full documentation of the interface can be found on GitHub. The README contains more bits too, so I’ll keep this one short.

4 Likes

That’s all very exciting!
I’m not sure what I’ll do with this, but it’s neat to have a other tool available!

My very first thought was OMG the security issues. I’m glad you thought ahead.

It’s certainly not a trivial matter, and especially when dealing with authentication, it kinda gets wonky. I’ve described in deeper detail in my RepeatFeed on what could be done to what extent, but the issue is always that with greater security, the usability suffers - at least with the abilities that we have available without proper engine support.

1 Like

-raises hand-
as a representative of those hard-of-thinking, what would the theoretical end user-use be? /i am sorry, i has the dumb today

I’m afraid I don’t understand the question.

I think the question was, “who would get the most use out of this when it’s complete?” / “what is the potential application of this?”

1 Like

Oh, connected things.

For example, it could allow you to create a new form of multiplayer. Imagine having a trader and a market, and said market is shared between players and in real-time. You could buy and sell goods on it, which wouldn’t be magically spawned but actually having been created by other players. Of course there’s certain issues with abuse and cheaters, but you might be able to get around it by having “private” markets (i.e. “you and your friends”).

In Frostfeast, we had the idea to make gifts across players/games possible, i.e. you can wrap something up and send it to somebody else. You could do that here too, simply select a friend from a list, send it away, and he’ll get it as soon as possible/the next time he plays the game.

You could also do some approach to social media. Sims 2 had a feature that allowed you to take screenshots, write descriptions, and upload that album to the official site, ready to be shared with everyone else. This could work here too: Collect stats (net worth, hearthlings, weather), capture screenshots, and send them to your blog. It’ll automatically post it, maybe update the stats, to show everyone what you’ve been building and how’s it going.

On the other hand, you could pull data in. News from the game, latest mod recommendations, and so forth.

Technically - but this would require more work on the DLL - you could also have independent modding repositories, unrelated to Steam. I did some thinking about it, and how to make it safe, but haven’t fiddled around with it yet. But it’s definitely possible, because the first step - actually breaking the sandbox - has been done.

4 Likes

Exactly @ kitty

So yeah, basically you can share stuff with the outside world. For example, if you wish to do so, you could actually “lend” hearthlings, too. Send your hearthling away to another player, and get it back with more experience/what not.

The possibilities are pretty endless. In the end, I think this can provide an additional multiplayer experience. Each player is on their own map, but still connected in a way with the others.

But it’s not limited to that either. Pull data from the internet, or push data from the game to the internet.

If anything, this was a proof of concept and to be honest, I don’t expect anyone to use it. I’d be happy to, but it’s unlikely given the format. This is also very likely the finishing piece of mine, so I’m not going to invest the effort to make more useful representations or demos.

4 Likes

Really cool with lots of potential, although if the game has access to the internet it’s only a matter of time before Fox News starts calling it Pornhearth or something catchy like that. :stuck_out_tongue:

Another use for this: Connecting with that template sharing site and basically use it as a cloud storage for your buildings. Heck, with enough dedication and good internet one could store whole saves in the cloud

1 Like

I suppose this would require support from my side as well. It’s something I’ve thought about, actually. You would likely need some API to upload or download files. Basically, it would be something like

req:execute({ save = http.get_savegame('my_save_name') }).

Maybe it would be a magic table instead, and I would make the parsing smarter (i.e. { type = "savegame", name = "foobar" } would attach the savegame instead of a string to the form), maybe some userdata that is worthless outside of HTTP but identifies it. The details would depend on what is requested.

Vice-versa, to fetch data from the response, there could be something like req:save_response_savegame('foo'), which would allow you to save the response as a save game (providing it is one). Some basic validation would need to be done, e.g. the server might need to respond with a certain header.

These are only required for things that run clearly outside of the user realm, though. Since templates can be loaded and stored easily by lua/JS (I think? Or is it a radiant.call?), there should be no need to provide additional settings.

Now mods, on the other hand, would be a different topic. That’s something that I would definitely need to provide an API for (uploading and downloading mods). Not something too difficult mind you, I believe…

In any case, extra features would needed to be whitelisted, too. Example: It shouldn’t be possible to download mods from any endpoint you’ve whitelisted, and it shouldn’t be possible to upload mods to any endpoint you’ve whitelisted. Within the whitelist, an explicit (complex) rule that allows it would need to be specified.

But that’s not a deal breaker either.