Engine Performance Improvements

Suggestions
1

Dear Radiant Developpers,
I’ve been playing this game since alhpa 5 and really enjoy this game. I this you guys have made a huge progress and will continue to shape this alpha into a powerfull and popular game.

Lately you’ve been adding new classes, a new storyline, and better war mechanics. And you even aim to nake this game a multiplayer in the future!! By looking at the road map, you guys say the performance and engine is 90% done. I think this game firstly needs a much bigger upgrade in performance before even tackling new advanced classes that we can’t even achieve due to the maximun hearthlings we can get into our towns (mine is 20). After a certain number of hearthlings, the game slows down and the x3 speed is forgotten, even sometimes the x2 speed.

I sincerely think and recommend you guys to focus your next alpha on major performance inprovements!! I know it is very boring, hard and umpleasant to tackle, but i think it would really strengthen the basis of this game and enable many more people to be able to play it lag free!!

Then you guys can focus on crazy combats, classes, storylines and even multyplayer!!

I even have a pretty decent desktop computer!

Enaugh said!! i congratulate you on the hard work and initiatives you guys put in daily in this wonderful game!!
Sincerely,
Rob

2 Likes
2

I think on their last update on alpha 17 they made cpu performance enhancements. I imagine the other 10% of the engine is going entirely to help the cause of performance against high hearthling towns. I would very much like to have 30 hearthlings and have it on x3 speed without it being choppy. We both need to keep in mind that this game still is in alpha, not beta.

1 Like
3

Ohhhhhhh, so you haven’t dealt with programmers before. 90% is half done~

But yeah, 17 is definitely faster than 16, so they’re working on it. I wouldn’t be terribly concerned at the moment. Optimization is usually a lower priority than making the game itself play well, so if we haven’t even hit beta yet, it’s pretty reasonable to give them time on it. If we hit beta and the default villager limit isn’t 40-50, then worry~

3 Likes
4

I’m sorry, but… NO.

Take a look at similar games like Gnomoria and Dwarf Fortress. They handle around 80 gnomes/dwarves before lag starts to be noticeable; but their job priority systems are far more advanced/complicated and their stockpiles/storage is far more advanced/complicated, and that adds extra overhead. These games are also single-threaded (they’ll only use 1 core of your quad-core CPU). Sure Stonehearth has much better graphics, but that’s mostly done by GPU (unlike Gnomoria and Dwarf Fortress where the graphics is done by CPU) so it doesn’t make much difference.

Taking all this into account (given that Stonehearth is far simpler, and multi-threaded) Stonehearth shouldn’t have trouble with 200+ hearthlings. You think 50 is “good”? Pure nonsense.

Currently, once the game reaches about 20 hearthlings things start taking too long and the game stops bothering (in what I consider a completely misguided attempt to hide lag). The forums are full of symptoms of this - hearthings standing around idle for half a day when there’s work they should be doing, crafters not doing jobs at workbenches for no (sane) reason, scaffolding not being taken down, buildings not being completed, sending a military party to a distant target and only half doing what they’re told. All of this is the same “bug” - the game not bothering when things (path finding, finding a job, etc) takes too long

From 20 to 35 hearthlings this just gets worse. Things take too long to get done, so the player decides to get more hearthlings, so there’s more overhead, so hearthlings stop bothering even more, so things take longer to get done.

Once you go above about 35 hearthlings you start seeing lag because the “stop bothering when it takes too long” hack alone isn’t enough to hide the massive performance problem.

Basically; the game is about 10 times slower than it should be.

The reason? There’s 2 reasons actually.

The first reason is LUA. LUA is a scripting language designed for secretaries to use when they get bored with diddling with VBA in excel spreadsheets. It’s not intended for anything performance critical, it’s intended for things like quest systems that aren’t executed often enough to matter. Stonehearth uses LUA for almost everything; so almost everything is much much slower than it should be.

The other reason is the way the game handles time and threads. It’s completely broken and wrong. Normally you have a “game time” and “wall clock time” which are only loosely related. You update entities, etc and increase “game time”. If the game is running too fast (“game time” gets too far ahead of “wall clock time”) you do nothing for a while to let “wall clock time” catch up. If the game is running too slow, then too bad - “game time” is allowed to fall behind “wall clock time”; which means you can have (e.g.) a “1000x speed” and the game will just run as fast as it can without any issues. For threads, let’s just say the game is full of race conditions - 2 hearthlings will go pick up some wood on the other side of the map, the first will collect it, the other will arrive and have nothing to collect (oops); miners will mine things in a certain order to prevent things getting stuck, but due to race conditions this order won’t be followed properly and mining jobs will get stuck, etc. I’ve even seen hearthlings trying to build a simple road; where one hearthling will place a block and another (who is still mining in preparation) will remove it immediately, leading to “place block, remove it, place block again, remove it again, …” until something upsets the timing.

To fix the problems; you’d have to rewrite about 90% of the game. This is not going to happen (it’d mean “no progress” for about 2 years, which will kill the game). Also note that “multi-player” won’t ever happen either (the game can’t handle the load from one player’s hearthlings, so handling “load from 2 player’s hearthlings plus Internet lag” is a fool’s dream).

What will happen is that the performance problems will never be fixed, the developers will continue to with “20 hearthlings only” as the default limit, and will continue to disable the “only 3x the (painfully slow) normal speed” button. Then the game will be abandoned soon after it gets out of beta; because everyone that wants it would’ve already bought by then, and the performance problems will prevent the addition of new features needed to convince more people to buy it.

Again; sorry. I know it’s a nice game (and fun), and I do want it to grow and thrive, and I do wish I was wrong; but the damage has been done and it’s too late to fix it now. :cry:

4 Likes
5

Oof. Yeah, I just found a site for comparing various scripting language speeds, and the thing is only a small, small fraction as fast as Java, which is itself not as fast as a native language like C++. That…really makes me sad.

I dunno, is it outside their budget to hire a couple people whose entire job is to convert existing LUA script into Java or something? I know I’ve definitely heard of having individual applications written in a mix of languages, but I’m not sure if it works as a patchwork thing?..

6

Half done? The sheer optimism.

The last 10% takes 90% of the time.

EDIT

@Brendan, let me play the devil’s advocate. You are correct in that Lua is slower than C++, but I don’t think that’s really a problem. The reason why:

I’m not sure if you’ve been following the update blogs, but basically, Lua speed is a known issue in Stonehearth. SH’s pathfinder used to be written in Lua. However, as part of a (big) upgrade to game speed and capacity, they rewrote the pathfinder in C++.

You gotta think that Radiant do have a balancing act to keep up. On one hand, blah blah blah Alpha blah blah not full game blah testing not playing bloodly bloody blah. On the other hand, come on: that’s not the fact. The fact is that people put money into kickstarter with the mindset of getting a game out of it. If Radiant didn’t spend at least a little bit of time on making the game as opposed to just tweaking the engine, we would never even have found out that the engine still needs work.

Following from this, I think it’s reasonable to say that as a small company, Radiant can commit their resources to only so much at a time. I do not presume to speak for them, but I do think that their priority right now is simply not on hearthling optimization. They want the game to run well with 20 hearthlings in every way – that is, combat, quests, gamemaster, progress, score, building, whatever. After all those things look solid, then they can relax and go back to their profiler and tweak formulae and shave milliseconds. I may sound dismissive here, don’t get me wrong, I know the importance of performance. You can’t have a hearthling kingdom with 20 hearthlings, that’s really more of a hamlet. But I also think that currently, Radiant has bigger fish to fry than performance. As long as performance is “good enough” for what THEY are doing, it will not be the focus of development.

EDIT 2
For some reason, the bit about Lua being for VBA-bored secretaries got to me. If it makes it any different, the Flame virus, aka the most sophisticated cyberwarfare weapon discovered to date, was mostly written in Lua.

4 Likes
7

I’ve been following development since Kickstarter, but I’ve missed a lot of posts over the years. It’s cool to hear that their is room for them to rewritten parts of the engine into C++. It’s surprising the more process intensive sections aren’t already written as such, but it’s great to hear they can identify and fix those bottlenecks.

Obviously, it’s hard to retroactively change some elements. Stephanie said on Stream how she thought if they had known more about how the building editor would be used, they might have written it completely differently (in response to comments about building block by block or more sophisticated builds). I’m happy to hear there is room to be adaptive, because as pessimistic as it may be, the comments about the limitations of the current engine to hold weight in its current state

8

Another important thing not to misunderstand: the engine is not written in Lua. Low level things like the renderer, pathfinder, etc are all in C++. All Lua files are “game content” in the same way that models and textures might be game content. They aren’t what one would classically call the “engine” of the game. So yes, if a particular Lua component is found to be bottlenecking, you can be sure they’ll rewrite it to go faster, as they have done so before.

4 Likes
9

Well, I won’t claim to know much about programming, so I cannot participate to your lua vs. C++ conversation, but I just want to add my voice to say that performance NEEDS to be addressed in a major way.
I just feel like with all the new classes and enemies and so on, the 25 hearthlings my PC can handle don’t suffice. And yes, I know it’s not strictly fair comparing games with stonehearth which are completely different, but still… I can run Witcher 3, total war warhammer, Assassins’ Creed Syndicate etc. All new, pretty system intensive games without an issue. So I feel like 25 bits of pathfinding shouldn’t be too much either…

10

I did not know the game was written in LUA, that explain the lagging game.
If they want a serious game, sooner or later, they have to port it to a low-level languish (C++).

All high-level languish like LUA, or Java have to be translated to assambler-code, instruction by instruction, before executing it, that take mostly more time then the instruction itself.

A low-level languish like C++ are after compiling, pure assambly code, and execute the instruction direct, without translating delay.

To think to build a serious game, out over some simpler browser games, with a script-languish, is not a good idea. What start out as a simple game, often grow out to much more when people like it.

I have follow some games that started out as scripted, or Java games, and they got 2 choices. Port it or leave it.

Last one i saw was the game Boundless, that started out as a java game. They had to take a 6 months timeout to port all code to C++. After that it run smooth, lags was gone and new content flow into the game without problem.

To be limited to 20 heartlings is very sad, it could be a very nice game, with big cities and lot of fun, but then it need a port.

Other city-building/rts game can do it, with hundreds and hundreds of in-game chars in full 3D, so way not this one.

11

Non sequitur but I’m assuming you meant language every time you said languish.

Did you read the previous comments? Moai explicitly addressed some of your comments. The reason that Stonehearth cannot handle hundreds of characters in its current form is because of the high level of ai and autonomy every single one has.

2 Likes
12

Well… the Ai is not exactly impressive. It’s all about choosing a task. I’m not sure I’d call that AI…

13

Then what exactly is AI? The ability to feel love and compose a beautiful sonnet? … even those things are the process of choosing what tasks to finish to accomplish your goal.

But seriously, what would it have to do for you to call it AI?

Would it really? “Lua is usually slower than C++ so therefore the game is slower”? I know someone who worked on a bank server, handling transactions, sessions, all that super important, super secure stuff. The server’s processor was worse than that of a midrange laptop, and it was running on an old version of Java. Why? The old version has been around for a long time, and is reasonably secure after constant updates for years. The server doesn’t NEED a superprocessor, just a large data bandwidth. The said server performed (and still performs) wonderfully, without any lag, in spite of running on a substandard language on substandard hardware. So without running a profiler and seeing exactly what’s lagging, what is giving you the idea that you know what it is? Tell me first why they wrote the game in Lua, and after you explain this to me to my satisfaction, I will listen to you why they shouldn’t have written the game in Lua.

It could be the fact that the game’s scripting is written in Lua, and Lua by itself is slow, yes. OR it could be under-optimized subroutines – write poor code in any language and it’ll perform poorly. OR it could be the renderer. OR it could be some particular section of the renderer, or the pathfinder, or whatever. We don’t know what it is. If you’ve profiled the game, please create a thread, I’d love to see your findings. If not, be patient.

I left the SH community for a while, because I felt game development was moving slowly, and I still didn’t have a solid game to play. I returned recently, played a game, and was blown away by the progress. This will sound crappy, but I’m saying it genuinely, from my heart: if you don’t like it, leave. Come back in six months, and it’ll blow you away, I promise.

3 Likes
14

Someone who knows what they’re doing can make a low-level language run faster than a high-level one. But the trade-off is that the lower-level a language is, it takes longer to write and test, and it’s harder to port to different systems.

Were performance the only issue every big game would be written exclusively in assembly language. But that takes far too long, doesn’t allow adding on gradually as nicely, is locked to one OS [Chris Sawyer’s having a heck of a time porting Roller Coaster Tycoon 2 to Android], and won’t even offer a performance benefit unless the programmers are much more skilled than actual compilers at writing assembly code. Also, you’d see a lot more programmers quit for the sake of their sanity.

For a Stonehearth-sized team, in Early Access, I think it makes much more sense to develop quickly with a little cost to performance than the other way around. But is it even that big of a cost? Stonehearth has a built-in profiler, showing how much time’s used for the Renderer, Pathfinder, and Lua, among other things. Unless the Lua bar’s much bigger than the others, there’s not much of a problem.

2 Likes
15

Oof. Yeah, I just found a site for comparing various scripting language speeds, and the thing is only a small, small fraction as fast as Java2, which is itself not as fast as a native language like C++. That…really makes me sad.

I dunno, is it outside their budget to hire a couple people whose entire job is to convert existing LUA script into Java or something? I know I’ve definitely heard of having individual applications written in a mix of languages, but I’m not sure if it works as a patchwork thing?..

In extremely broad terms, there’s always a compromise between development speed and execution speed. If they rewrote most/all of the code in C++ it’d be a lot faster, but harder to write/maintain and harder to debug. It’d also make it harder to create mods for the game, as (without the C++ source code) anything in C++ can’t be changed by modders. I’d assume this is why they’ve chosen to use LUA so much (faster development time, easier for modders, worse for everyone else).

I also suspect there’s an element of “prototyping” involved. The idea of doing things (e.g. water/liquid physics) in LUA where it’s easier, and then convert it to C++ after they’re sure it doesn’t need to be changed (e.g. after they’ve experimented with different algorithms, etc). Of course when programmers say “we’ll convert to C++ one day” that day rarely comes (there’s other features people want that are more interesting, deadlines from marketing people, etc).

For budget, it’s better to think in terms of profitability - companies see decisions based on “return on investment”; so It becomes a question like “If we spend an extra $X and make an extra $Y in sales, is Y greater than X?”. The problem with (e.g.) Steam’s “early access” stuff is that most of the people that will ever buy the game will buy it during “alpha” or “beta” (and few people wait for a final release), which means that the potential for that “extra $Y” is severely diminished before the game reaches final release. Basically; it’s more like “If we spend an extra $X (to rewrite most of the LUA in C++) and make an extra $Y in sales (where Y is tiny because most people already paid), is the tiny little Y greater than the huge X?”.

2 Likes
16

I’m not sure if you’ve been following the update blogs, but basically, Lua speed is a known issue in Stonehearth. SH’s pathfinder used to be written in Lua. However, as part of a (big) upgrade to game speed and capacity, they rewrote the pathfinder in C++.

For games like Stonehearth, path-finding is single largest performance problem, and the fact that they’ve (been forced to?) shift path-finding to C++ shouldn’t surprise anyone (beyond the fact that they actually wrote it in LUA in the first place).

The second largest performance problem for games like Stonehearth is normally job control. Essentially, when a hearthling needs to find something to do you search through a list of jobs, filter out things the hearthling can’t do (e.g. only carpenter can make a wooden chair, cook can’t make a stew unless the raw ingredients are in inventory, etc), find the most important job they can do, then mark items needed for the job as “in use” (e.g. so that some other hearthling doesn’t eat the raw ingredients for the stew after a hearthing takes the “make stew” job) and either remove the job from the list or mark it as “in progress” (so other hearthlings don’t try to do the same job). Of course (for performance) this wouldn’t be a single master list of all jobs; but could/should be a list for each type of job (e.g. one list for carpenter jobs, one for cook jobs, one for mining jobs, one for hauling jobs, etc). This is currently all in LUA. Because it touches on a large number of things (all work done by all roles) it’s not easily separated from everything else and it would take significant amount of work to convert to C++.

Also note that for multi-threaded code some parts of job control (e.g. the “mark items needed for the job as in use”" part) need to be thread safe. Currently there are several bugs caused by this part either not being thread safe or not being done at all; and I suspect this is because LUA lacks any built in support for thread safety (and makes it virtually impossible to do properly, especially handling errors that occur while locks are held).

Following from this, I think it’s reasonable to say that as a small company, Radiant can commit their resources to only so much at a time. I do not presume to speak for them, but I do think that their priority right now is simply not on hearthling optimization. They want the game to run well with 20 hearthlings in every way – that is, combat, quests, gamemaster, progress, score, building, whatever.

Why 20 hearthlings? Why not 10 hearthlings? Why not 1 hearthling?

Currently for “end game” you need about 25 military, 5 farmers, 2 cooks, carpenter, blacksmith, engineer, herbalist, 3 or 4 trappers/shepherds, plus about 10 general workers for hauling/mining/building. That’s about 50 hearthlings.

With only 20 hearthlings the game is unplayable. Your military is too small so you get trashed by enemies, your workforce is too small so any kind of large scale project is like watching paint dry, etc.

To add new features, they need more quests with harder enemies (but you’d need more than 50 hearthlings for that), or new roles like geomancer (but each new role adds to the total workforce needed). Performance problems limit total number of hearthlings which limits scope for new features.

For some reason, the bit about Lua being for VBA-bored secretaries got to me. If it makes it any different, the Flame virus, aka the most sophisticated cyberwarfare weapon discovered to date, was mostly written in Lua.

I’m quite biased towards lower level languages - I’ve spent the last 10 years writing highly optimised/scalable code in C and assembly language (in a different field, not game development). I’m the sort of guy that would take 20 times longer to write the code, and then offer a very sincere apology if I can’t make it handle over 400 hearthlings simply because I know it wouldn’t take much to handle 200 hearthlings. While I don’t expect this level of optimisation for any game; to me, LUA really is scraping the bottom of the barrel - it’s a toy for people that just couldn’t handle real programming. :wink:

2 Likes
17

What would your ideal solution be?

If you wrote it in a low level language, you’d have to do something special to make it moddable. Writing it in a high level language makes it easier to mod, but requires finer tuning for performance. It feels like I’m saying “they haven’t gotten to the tuning yet”, and you are saying “the game cannot be tuned”, and I am not sure on what basis you are saying that.

18

What would your ideal solution be?

Give me some time and I’ll put together a description for it and post it as a suggestion (in a different/new topic to avoid cluttering up this topic more).

If you wrote it in a low level language, you’d have to do something special to make it moddable.

Why? For some things (rendering, path finding, job control, liquid physics) performance is more important than making it moddable (and it’s worth the extra development time because it has a significant impact on performance). Modders would still be able to create new roles, new types of jobs, etc; they just wouldn’t be able to change the “which job gets done when” code.

Writing it in a high level language makes it easier to mod, but requires finer tuning for performance. It feels like I’m saying “they haven’t gotten to the tuning yet”, and you are saying “the game cannot be tuned”, and I am not sure on what basis you are saying that.

You can fine tune LUA as much as you like, it will never be comparable to the performance of the same code written in C, C++ or Java. LUA is fine for things that aren’t executed often enough to matter. It’s not fine for the 2nd largest performance bottleneck in games like Stonehearth (and not fine for what is currently the single largest performance bottleneck in Stonehearth).

I’m saying that to convert job control over to C++ is definitely technically possible, but would require significant changes throughout a lot of existing code (especially if it’s done properly), and therefore won’t ever be considered profitable and will never be done. I hope I’m proven wrong; but I’ve seen enough commercial software development over the years to be sceptical.

1 Like
19

Lua is pretty fast for a scripting language, but in general is not fast enough for high-performance code. For performance sensitive components, we write high-speed primitives in C++ and call them from lua. This includes pathfinding, rendering, ai scheduling, computational solid geometry, and object tracking.

The in-game performance bar shows a breakdown of the time spent in each area. If your game is slow, you can click on the bar and see what the rough bottleneck is. It will also help us if you post a screenshot so we can see the layout of your town and where the time is being spent.

1 Like
20

So wait, does that include the job scheduling? Is the job scheduling already in C++?