The Modding Requests

You can find the response to this batch here.

This batch has been submitted and answered. The next modding requests can be found here.


Hello and welcome to the first The Modding Requests.

This thread has two purposes: It lists (and prioritizes) modding API requests and modding-related questions that the community has for Team Radiant. Those will be collected and sorted by me. Every week, the five or so most important requests and questions will be checked by Team Radiant. Please read the short FAQ below before posting :slight_smile:

To discuss the entries, the prioritization or anything else that is not a request or question, please visit this thread!


Proposals and questions that will be submitted:

Proposals

  • @RepeatPan: File API

  • read access to all files inside /mods and some dedicated “data” directory

  • write access to a dedicated “data” directory

  • write, read, rename, create, delete, list files & directories

  • long term bonus: virtual directory for per-game/per-save data that does not require to be in lua 24/7?

  • @RepeatPan: render_info:set_scale

  • currently only working when switching to a model variant that has a different .qb

  • @RepeatPan: modify rendering behaviour inside mods

  • materials, textures, shaders (?)

  • @RepeatPan, @Asterai: access to qb matrices and model_variants

  • read and write matrices/models (on voxel level)

  • create new, assign and modify models/model variants during runtime

  • @RepeatPan: disabled mods and mod list

  • user_settings.json entry that skips mods with said name

  • event radiant:modules_loaded gets a list of modules that were loaded

Questions

  • @RepeatPan: How is stonehearth:entity_created accessed? It’s called on the created entity - is there some way to listen to all entity_created events?

  • @RepeatPan: Is there a way to enable the micro map you use to test things?

  • @RepeatPan: Can we access the Javascript debugger/refresh UI stuff on the fly already?

  • @RepeatPan : What’s the difference between radiant.resources.load_json and radiant.resources.load_manifest? Is it merely some sort of programming sugar?

  • @RepeatPan: Is the JSON parser non-strict on purpose? (i.e. [ "abc" "def" ] being merged to abc""def, [ [], [], [], ] being valid)


How can I propose a modding-related feature?

Simply reply to this thread. Please keep the following things in mind:

  • Your proposal should be related to modding. This includes (but is not limited to!) a new interface for lua, a new event in JavaScript, the possibility to tweak something using JSON, …
  • Keep it short but precise. Don’t be vague and don’t be too general. If you have an idea but are unsure how to phrase it, come and visit us in the discussion thread. We’re happy to help :smiley:
  • Propose things that you think are useful. The main priority is to get things into the game that modders can profit early on - so things that can be utilised immediately will have a higher priority than something that, for example, requires another game feature to be present first. Explain what could be done if your proposal got implemented.

Your proposal will then be added by me, which means that I may need to rewrite parts of it. In any case, I’ll post an update to the discussion thread, so you should get a ping about the status.

How can I ask a modding-related question?

Just reply to this thread. As with proposals, a few tips:

  • Your question should be related to modding.
  • Keep it short but precise. Don’t be too vague.

As with proposals, your questions will be added to the list by me. Questions that have already been answered will not be added to the list (or get removed early).

Who decides what will be submitted this week?

That would be me! I volunteer to collect everyone’s modding feature requests and questions and sort them by a few points:

  • Generality: Something that many different mods can profit from will be ranked higher than something that is useful to a few.
  • Time investment/Feature gain: Smaller or simpler things are more likely to be implemented soon and therefore get better spots.
  • Distance: Something that is related to mods that currently exist, or mods that are currently in development take priority over stuff for “upcoming mods” or “mod ideas”.
  • Community: I’ll follow and participate in the discussion over at the discussion thread. If you disagree with a decision, feel free to start a conversation about it. It will be a fairly democratic process, so your voice does matter!
  • Reasoning: Entries that have a reason or goal attached to it (e.g. “I am currently developing mod X and could use Y”) take priority over “pure curiosity” proposals and questions. It might make sense to include the “what” in your proposals (questions not necessarily) to get a bit of context what you are doing.

Both proposals and questions can move between “submitted” and “moved to next week” at any time. Once the deadline’s up, all entries that were not done this week will be carried over to the next week’s thread.

For questions, criticism, concerns and similar, here’s the discussion thread!


Things considered for the next thread:

##Proposals:

  • @RepeatPan: support for lua binary modules

  • either directly in archives or dedicated folder

  • security issues: certification required/user acknowledgement?

  • @RepeatPan: mixintos to remove/modify content

  • alternatively/additionally event for radiant.resources.load_json that allows modifications

  • @RepeatPan: event listener ordering

  • allow to tag events with a priority (number)

  • bonus: define mods whose events should run after/before this event, including a magic “all” element

  • @RepeatPan: Point3/Point3f compatibility

  • _add, _sub both without errors (likely returning a Point3f)

  • implicitly convert Point to Point3f (and vice versa)

  • @RepatPan: configuration management

  • save configurations (similar to radiant.util.get_config)

  • bonus: (simple) validation of configs using JSON scheme or similar

  • bonus: pass mods their configuration as argument (e.g. ...?), add default_config section in manifest.json

  • @RepeatPan: harvest_node implementation

  • complete implementation of ResourceCallHandler:harvest_node

  • add events on the specified component that are called by the AI: on_harvesting (during AI action), on_harvested (after AI is done)

  • @RepeatPan: component/buff removal

  • functions to remove buffs/components from an entity

  • @RepeatPan: disable input in lua and JS

  • add function sto enable/disable is_down and capture_input events in lua/$('keydown keyup') in JS

  • useful for <input>, cinematics or similar

  • @RepeatPan: modify start menu

  • add, query and remove current start menu by lua/JS and JSON mixinto

  • @RepeatPan: recipe management

  • enable, disable, add, remove recipes from carpenters by lua/JS

  • @Teleros: bone attachment managing (already somewhat possible?)

  • attach (other) entities to bones of entities, specified with relative position/rotation

  • query bone position/rotation

  • @Teleros: dynamic reload files during runtime (medium term candidate)

  • reload textures/models/materials during runtime (automatically)

  • probably not scripts

  • @RepeatPan: start camp overhaul (incomplete)

  • JSON based approach for easier mixinto/override

  • entity ref/model that is used for choosing the location (i.e. default: the banner)

  • list of entities that are initially spawned (i.e. default: banner, firepit, workers, logs, saw)

  • lua events to add more dynamic content/events

  • make this part of the faction data?

  • @RepeatPan: faction overhaul (incomplete?)

  • reduce factions to a simple name/object instead of (name, json).

  • add callbacks/virtual functions/events to determine gender, entity ref, name, personality/stats

  • event when a new faction is created (or json-defined lua file that is executed to extend/modify the faction’s object), or a new citizen is spawned

  • provide a function that spawns a “more complete” citizen (like NewGameCallHandler:place_citizen)

  • relations between factions (i.e. Faction:can_use(Faction other, Entity entity, Action action)) instead of hardcoded comparisons

  • @Miturion: fine-tuned control about building creation (incomplete?; announced)

  • define which block/brick is used at which location in a building/wall

  • general look of buildings (?)

  • using any (scaled) entity/model as brick, not just 1x1x blocks (?)

  • @RepeatPan: world generation (incomplete; research being done)

  • define flora in JSON

##Questions:

(None yet!)

3 Likes

I moved a post to an existing topic: The Modding Requests: Discussion Thread

Hi,

Thanks for pulling these together! These are really great questions! Looking forward to the next batch. Feel free to elaborate on any of the points below. The idea is to start a discussion with the modding community on the most important topics for them, not for this to be a dissertation of the “One Source of Truth” or anything like that.

Proposals

@RepeatPan: File API

A general purpose File API would be really powerful, but is also a double edged sword. It would enable a whole class of very powerful mods which would be impossible otherwise, but we don’t want to do anything would could compromise the end user.

For example, if we provide a general purpose write file API, an unscrupulous modder could easily write a virus to your system and try to trick you into clicking on it via their UI. Even if we only let you write well-known file types with well known extensions, modders can exploit bugs in the underlying operating system to still wreak havoc (e.g. the recently revealed .png metadata overflow attack).

Could you elaborate on what types of things you’d like to do with a File API? Maybe we can enable them with a specific and impossible to exploit API instead. For example, World of Warcraft has a combat log feature which is used by World of Logs and others to give really awesome combat summaries. Since Wow’s determining the content of the file, there’s no way to exploit the combat log to break out of the sandbox.

@RepeatPan: render_info:set_scale

I’m working on a fix and hope to have it in the next release. Note that render_info:set_scale only affects the appearance of the entity, not any of the other actual entity state. For example, trees define a region where someone must be standing to harvest a tree (the ‘destination’ component’s adjacent region). Changing the scale of the tree in render_info won’t resize that region, so things are probably going to look a little silly if you don’t also change that region. The same thing is true for collisions. Eventually, we hope to have a system where you can easily scale the entire entity, including the destination and region_collision_shape components.

@RepeatPan: modify rendering behavior inside mods materials, textures, shaders (?)

You can change the material of a model using render_info:set_material(). For an example, see “horde/materials/default_material.xml”. In the near future, we want to move the entire horde directory into the Stonehearth mod. After that happens, you should be able to do things like render_info:set_material(‘mymod:materials:translucent_shiny_glass’) to point to your own material files and shaders. This probably won’t make the next release, but I’ll see if we can get it done soon. In the meantime you manually copy stuff into the horde directory.

@RepeatPan, @Asterai: access to qb matrices and model_variants

We have the framework for implementing this, but it would need a lot of work to apply it generally. It’s currently used by the construction sub-system to make walls of arbitrary size out of qubicle templates. Check out the voxel_brush stuff in the disassembled code if you’d like to see what it looks like now. I’d like to see this, but it probably won’t come along until we have a game-system or class that desperately needs it. It’s a lot of work, and doing it now would come at the expense of working on more gameplay features (like save/load… need… save… load…!)

@RepeatPan: disabled mods and mod list

We definitely want more sophisticated mod loading, including the ability to pick what’s enabled/disabled for your game. It’s probably something that will get tackled on the way to save/load, which means sooner rather than later.

Questions

@RepeatPan: How is stonehearth:entity_created accessed? It’s called on the created entity - is there some way to listen to all entity_created events?

I’ll add the same trigger to radiant.events. It should be in the next build we release.

@RepeatPan: Is there a way to enable the micro map you use to test things?

Not easily, but I’ll add it my list of things to try to expose soon. If you add this to your user_settings.json file, you can create a small (though not as tiny as ours) world, which should dramatically speed up the game start time:

"mods" : {
	"stonehearth" : {
		"world_generation" : {
			"method" : "tiny"
		}
	}
},

@RepeatPan: Can we access the Javascript debugger/refresh UI stuff on the fly already?

It works in the shipping build, but you need the devtools package from Chromium Embedded to enable it:

  1. Download the latest CEF3 build
  2. Open the archive and copy cef.pak and devtools_resources.pak in the resources directory to your Stonehearth folder (right next to the locale’s folder).
  3. Run the game and point Chrome at http://localhost:1338

You can hit F5 in Chrome to reload the UI in Stonehearth.

**@RepeatPan: What’s the difference between radiant.resources.load_json and radiant.resources.load_manifest? **

There’s no real difference.

**@RepeatPan: Is the JSON parser non-strict on purpose? **

We use libjson to read/write JSON. I’m guessing it’s a bug in their implementation or we’re not catching an error somewhere. At any rate, it’s not-intentional.

Looking forward to the next batch!

7 Likes

Thanks for the answers! I would like to start the discussion by answering/addressing a few points:

[quote=“Ponder, post:4, topic:5317”]@RepeatPan: File API

Even if we only let you write well-known file types with well known extensions, modders can exploit bugs in the underlying operating system to still wreak havoc (e.g. the recently revealed .png metadata overflow attack).

Could you elaborate on what types of things you’d like to do with a File API? Maybe we can enable them with a specific and impossible to exploit API instead.
[/quote]

The question is somewhat where your (i.e. the game developer’s) responsibility ends and the user’s responsibility begins. I would say that opening a file in the data folder (or simply just accessing it - it could be stashed away, hidden in %APPDATA% for example) is kind of the user’s area. They could always download a mod which contains nasty stuff and asks them to extract the mod and run modding_enhancements.png.pdf.doc.pif.exe or have it installed using a “mod installer” - both those possible security risks aren’t really your problem (nor something that you could possibly fix/prevent).

However, GMod’s approach - which I would be in favour of too - would simply allow writing .txt files (with a limited size?). Although certainly not free of exploits, it should give enough freedom to modders to deal with their issues. I can’t think of all possible uses, but a few one on top of my head:

  • configuration (.json): Could be put into user_settings.json if there would be radiant.util.set_config or similar (that’ll be a candidate request for next week!). I would have used fs.write/fs.read just for that.
  • images (.png/.jpg): This could/should probably be provided by some sort of bitmap library from the game itself. First use that comes into my mind would be stats plotting, map generation, …
  • data export (.json/.txt): To have dynamic stats for example, export a json that can be imported somewhere and then plots it (for example, some website that uses the Google APIs). The opposite to load_json in most cases.
  • generic save data (.txt/.blob?/.shblob?): Data mods need to save. Combine this with the matrix manipulation to have some sort of “Save this work/building” or “Create blueprint”. This could also include things like custom/advanced log files (that want to be kept separate from the stonehearth.log).
    This partially works just as well with a json (even if it’s just { "data" : "endless string of (binary) data" }, but might be inefficient to store as JSON (bloated up through needless escaping for example, improper formatting for logs or similar).

I’m sure there’s lots of uses that I’ve forgotten or that wouldn’t fit into these four general areas. The last point is probably the weakest and for now I can just find use for the first three.

[quote=“Ponder, post:4, topic:5317”]@RepeatPan: render_info:set_scale

Eventually, we hope to have a system where you can easily scale the entire entity, including the destination and region_collision_shape components.
[/quote]

Looking forward to that! For the current known applications, it’s not too much of an issue that the other things do not scale - visual is enough.

[quote=“Ponder, post:4, topic:5317”]@RepeatPan: modify rendering behavior inside mods materials, textures, shaders (?)

In the near future, we want to move the entire horde directory into the Stonehearth mod.
[/quote]

That was the core of this request - it’s possible to add things right now, but it’s a little more hassle than just putting a file into the /mods directory and auto updaters have to deal with it in special ways (if even).

[quote=“Ponder, post:4, topic:5317”]@RepeatPan, @Asterai: access to qb matrices and model_variants

I’d like to see this, but it probably won’t come along until we have a game-system or class that desperately needs it. It’s a lot of work, and doing it now would come at the expense of working on more gameplay features (like save/load… need… save… load…!)
[/quote]

How difficult would it be to have the following things, theoretically speaking…

  • Loading a new qb model from a lua string (binary lua strings work like a charm with lua_tolstring, again I’m not sure how luabind handles those) or lua table/array
  • Adding/overwriting a model variant on an entity at runtime
  • Loading .qbs inside /mods into a lua string or lua table/array

If that is doable, I would take care of the rest (after talking to Tim about that of course) and that would definitely be a request for the next thread.

That information is really useful - thanks! Reloading the GUI works, but the interface in Chrome itself looks rather broken - like it’s missing some JS or CSS. Is that normal or did I mess something up? I did extract the whole 7z/resources into the SH directory, although my locales/ files differed in size from the ones delivered with the game.


As for the future of the series, I’ll start the next thread tomorrow. Likely it will be two weeks this time (or not have a “fixed” schedule per se) - the timing is still something that can be tuned I believe.

1 Like

Seconding what @RepeatPan is saying on the File API. @Tom maybe you have seen my ideas about a “Chronicle”-Mod. The idea is to save key milestones / events in a book which can be also shared with other players (and loaded into new games). To do so I would need the possibility to save what is written in the book (pre-defined text and pictures, but also custom text and screenshots) into a file and to load it again.

I have also mods in my mind where there will be some kind of a “score” or “progress” (title, points, etc.) involved. To implement them properly I would also need to be able to save this values.

2 Likes

Hi guys. The next build will move horde into the Stonehearth mod directory and add methods to dynamically change model variants at runtime. More on that as we get closer to release.

7 Likes

Nice. Does this also account for things like height for example? Or a broader question, ability to change the rigs used in a model?

I’m not sure what you mean by height. Can you give an example?

You should be able to change rigs, though it would be a bit complicated. In order to animation a model, we need to know the local origin of each body part int he Qubicle file. That’s what the skeleton in the rig format is for. It describes the position of all of the origins in all of the Qubicle matrices attached to the model variants. Ideally you could just change the rig out by calling the render_info component’s set_animation_table method and everything would “just work”, but if the position of the bones of the skeleton in the new rig doesn’t match the positions in the old rig, everything will look really wonky.

The best fix for that is to figure out how to tie the local origin of each matrix to the matrix file itself rather than having the supply it from the skeleton file. Once’s that’s done, you should be able to switch rigs by just changing the animation table pointer. It’s on my list, but probably won’t make this patch. In the meantime, everything should work fine if you change out the animation table and all the model variants to ones built to work with the new right, though I admit I haven’t tried it. :slight_smile:

1 Like

Can we perhaps have a little peek into what exactly “dynamically change model variants at runtime” means to you? :smile:

1 Like

I changed the male rigs and used a different body.qb with a higher torso. Some way link matrices and rigs would be nice. But you kinda answered it already that you are working on it.

1 Like

Check out customize_citizen in stonehearth/servcies/population/population_faction.lua and stonehearth/entities/human/male_1/male_1.json.

customize_citizen uses the stonehearth:customization_variants entity data to dynamically create new models on newly created entities. It basically walks the tree from root, adding more and more qbs onto the model_variants component via model_variants_component:add_variant(variant_name):add_model(random_model). The game engine will take all the qbs the ended up on a certain bone (in this case, they all end up on the head), squash them together, mesh it, and send it off to the renderer.

4 Likes

From what I’ve seen it seems like a pretty nice setup, though it doesn’t seem to work well yet (unless im doing something wrong). Wish I could decompile the lua to take a peek but I’m exhausted trying to figure that stuff out.

As for the entity customizations, it’s still very much WIP, but I’ve torn apart the basics in another post of mine.

What it does, currently, is building a tree, starting with the “root” variant and then go all the way down until it reaches a variant that has no child-variants. In simplified pseudo-lua, it would be

function check_variant(variant)
   if variant == nil then return end
   if variant.models then add_model(variant.models[1]) end
   if variant.variants then check_variant(get_random_variant(variant.variants)) end
end

check_variant("root")

In theory, I suppose, you could even have loops, but I strictly advise against that… It’s not that difficult if you visualise it perhaps.

root is the first variant to be loaded, that’s pretty much hardcoded. After that, it is taking a random sub-variant until there’s no sub variant. In this example, circles are just variants for ordering’s sake, whereas rectangles set a model. An arrow indicates a sub-variant, i.e. root has only sub variant, young, which will always be taken, whereas young has three sub-variants and young_blonde, young_brown and young_sandy are all taken at approximately 33% chance each.

If a variant adds a model, it’s really just thrown onto the default model variant - which, unless your mod does something else, is the one always used. There are only a few cases where a different model variant is used by SH, most prominently the berry bush. Villagers, so far, use only the default one.

I’ve tried using unwrp and it just keeps taking my smod file and renaming it. I don’t know why, and I haven’t had the ambition to figure it out.

The chart does help visualize it, though I think I understand it decently. Just can’t wait for it to be finished so it doesn’t just choose the same branch every time. . .[quote=“RepeatPan, post:14, topic:5317”]
I suppose, you could even have loops, but I strictly advise against that
[/quote]

But loops are fun! Especially infinite loops! :smiley:

It should extract the things into the folder though? Have you tried running it as administrator?

That part is finished. It’s taking a random variation every time; the only thing it doesn’t do is taking a random model out of a branch. You can easily fix that, or rather work around that by splitting up my_variation into my_variation_entry -> [variation_with_model_1, variation_with_model_2, ...] -> my_variation_exit - which would have the same effect. Probably.

As for “taken the same every time”… unless things have changed, I’m afraid the random number generator in lua (math.random) might still not be properly initialized (something RP fixes iirc). That means that no matter how often you start the game, it will always return the same random numbers - hence, the same variations. You can fix that by placing math.randomseed(os.time()) somewhere in your code before the citizen is spawned - in that case, pretty much anywhere in your mod would be fine. Preferably the server_init_script.

The fact that I’m a little hung over didn’t inform me that I was looking in the wrong folder. I found it.

Doesn’t seem to even do that for me, just chooses the original model and no variations, even after I fixed that error we found.

1 Like

Yepp, that might be because of the random number generator as mentioned above. You can either try installing RP or this little smod. The latter is probably more durable than RP and can easily be re-distributed.

I think that this might solve the issues, but I’m not in a deep “uninstall RP and debug stuff without the console” mood today… Not while I’m working on the kitchen.

I’ll have to check it out soon.

My limited c++ knowlege isn’t helping me understand the code very much, looks like I need to spend some time looking into lua format.

Though what I can sort of understand it’s a very interesting set up. More complex than I thought.

Edit:
Just out of curiosity, this code:

function PopulationFaction:generate_random_name(gender)
  local first_names

  if gender == "female" then
    first_names = self._data.given_names.female
  else
    first_names = self._data.given_names.male
  end

The function generate_random_name takes in the argument containing the gender. Then the if/else statement determines what the gender is and then accesses the (self meaning the folder the lua file is in) self/data/ascendancy.json and uses the correct category to choose the first name?

If so than many I understand this a little more than I thought.

The periods in things such as radiant.mods.require is what was throwing me off.

Periods are syntax sugar for table accesses, where table["key"] is equal to table.key. It’s therefore accessing the table self (the “object”, similar to this in C++ and friends, another syntax sugar; function class:bar(first_param) is equal to function class.bar(self, first_param) - note the difference between period and colon), then _data, then given_names and at last female. self is not the “folder the lua is in”, there is no such thing. lua (at least in SH) is not relative to anything; every require is relative to its mod folder (with radiant.mods.require being relative to mods/). There is no hidden magic to load files - files are loaded using radiant.resources.load_json, which require an absolute path (… relative to mods/) or an alias.

I recommend reading the lua reference if you are completely new to lua. It explains the very basics of the language and you can usually skip, although it might provide useful insights, all the programmy techy parts.

This got me rather curious and I didn’t really know where to ask it, but I’d hedge my bets that you (Pan) are the most likely to know.

Would it be possible to change the mods of a currently existing save file? So, say I had Nihonjin and I wanted to switch to the Egyptian Mod, could it be done? Would it be impossible because they override the same model and thus would end up horribly or are there other problems? And if that is the case, would it be possible with mods that never touched each other, such as one to say change Stonehearth’s language and one that simply replaced the original worker with Batman? I’m just quite curious and naïve as to the whole interaction between mods and saving. :slight_smile: