Engine Performance Improvements

Someone who knows what they’re doing can make a low-level language run faster than a high-level one. But the trade-off is that the lower-level a language is, it takes longer to write and test, and it’s harder to port to different systems.

Were performance the only issue every big game would be written exclusively in assembly language. But that takes far too long, doesn’t allow adding on gradually as nicely, is locked to one OS [Chris Sawyer’s having a heck of a time porting Roller Coaster Tycoon 2 to Android], and won’t even offer a performance benefit unless the programmers are much more skilled than actual compilers at writing assembly code. Also, you’d see a lot more programmers quit for the sake of their sanity.

For a Stonehearth-sized team, in Early Access, I think it makes much more sense to develop quickly with a little cost to performance than the other way around. But is it even that big of a cost? Stonehearth has a built-in profiler, showing how much time’s used for the Renderer, Pathfinder, and Lua, among other things. Unless the Lua bar’s much bigger than the others, there’s not much of a problem.

2 Likes

Oof. Yeah, I just found a site for comparing various scripting language speeds, and the thing is only a small, small fraction as fast as Java2, which is itself not as fast as a native language like C++. That…really makes me sad.

I dunno, is it outside their budget to hire a couple people whose entire job is to convert existing LUA script into Java or something? I know I’ve definitely heard of having individual applications written in a mix of languages, but I’m not sure if it works as a patchwork thing?..

In extremely broad terms, there’s always a compromise between development speed and execution speed. If they rewrote most/all of the code in C++ it’d be a lot faster, but harder to write/maintain and harder to debug. It’d also make it harder to create mods for the game, as (without the C++ source code) anything in C++ can’t be changed by modders. I’d assume this is why they’ve chosen to use LUA so much (faster development time, easier for modders, worse for everyone else).

I also suspect there’s an element of “prototyping” involved. The idea of doing things (e.g. water/liquid physics) in LUA where it’s easier, and then convert it to C++ after they’re sure it doesn’t need to be changed (e.g. after they’ve experimented with different algorithms, etc). Of course when programmers say “we’ll convert to C++ one day” that day rarely comes (there’s other features people want that are more interesting, deadlines from marketing people, etc).

For budget, it’s better to think in terms of profitability - companies see decisions based on “return on investment”; so It becomes a question like “If we spend an extra $X and make an extra $Y in sales, is Y greater than X?”. The problem with (e.g.) Steam’s “early access” stuff is that most of the people that will ever buy the game will buy it during “alpha” or “beta” (and few people wait for a final release), which means that the potential for that “extra $Y” is severely diminished before the game reaches final release. Basically; it’s more like “If we spend an extra $X (to rewrite most of the LUA in C++) and make an extra $Y in sales (where Y is tiny because most people already paid), is the tiny little Y greater than the huge X?”.

2 Likes

I’m not sure if you’ve been following the update blogs, but basically, Lua speed is a known issue in Stonehearth. SH’s pathfinder used to be written in Lua. However, as part of a (big) upgrade to game speed and capacity, they rewrote the pathfinder in C++.

For games like Stonehearth, path-finding is single largest performance problem, and the fact that they’ve (been forced to?) shift path-finding to C++ shouldn’t surprise anyone (beyond the fact that they actually wrote it in LUA in the first place).

The second largest performance problem for games like Stonehearth is normally job control. Essentially, when a hearthling needs to find something to do you search through a list of jobs, filter out things the hearthling can’t do (e.g. only carpenter can make a wooden chair, cook can’t make a stew unless the raw ingredients are in inventory, etc), find the most important job they can do, then mark items needed for the job as “in use” (e.g. so that some other hearthling doesn’t eat the raw ingredients for the stew after a hearthing takes the “make stew” job) and either remove the job from the list or mark it as “in progress” (so other hearthlings don’t try to do the same job). Of course (for performance) this wouldn’t be a single master list of all jobs; but could/should be a list for each type of job (e.g. one list for carpenter jobs, one for cook jobs, one for mining jobs, one for hauling jobs, etc). This is currently all in LUA. Because it touches on a large number of things (all work done by all roles) it’s not easily separated from everything else and it would take significant amount of work to convert to C++.

Also note that for multi-threaded code some parts of job control (e.g. the “mark items needed for the job as in use”" part) need to be thread safe. Currently there are several bugs caused by this part either not being thread safe or not being done at all; and I suspect this is because LUA lacks any built in support for thread safety (and makes it virtually impossible to do properly, especially handling errors that occur while locks are held).

Following from this, I think it’s reasonable to say that as a small company, Radiant can commit their resources to only so much at a time. I do not presume to speak for them, but I do think that their priority right now is simply not on hearthling optimization. They want the game to run well with 20 hearthlings in every way – that is, combat, quests, gamemaster, progress, score, building, whatever.

Why 20 hearthlings? Why not 10 hearthlings? Why not 1 hearthling?

Currently for “end game” you need about 25 military, 5 farmers, 2 cooks, carpenter, blacksmith, engineer, herbalist, 3 or 4 trappers/shepherds, plus about 10 general workers for hauling/mining/building. That’s about 50 hearthlings.

With only 20 hearthlings the game is unplayable. Your military is too small so you get trashed by enemies, your workforce is too small so any kind of large scale project is like watching paint dry, etc.

To add new features, they need more quests with harder enemies (but you’d need more than 50 hearthlings for that), or new roles like geomancer (but each new role adds to the total workforce needed). Performance problems limit total number of hearthlings which limits scope for new features.

For some reason, the bit about Lua being for VBA-bored secretaries got to me. If it makes it any different, the Flame virus, aka the most sophisticated cyberwarfare weapon discovered to date, was mostly written in Lua.

I’m quite biased towards lower level languages - I’ve spent the last 10 years writing highly optimised/scalable code in C and assembly language (in a different field, not game development). I’m the sort of guy that would take 20 times longer to write the code, and then offer a very sincere apology if I can’t make it handle over 400 hearthlings simply because I know it wouldn’t take much to handle 200 hearthlings. While I don’t expect this level of optimisation for any game; to me, LUA really is scraping the bottom of the barrel - it’s a toy for people that just couldn’t handle real programming. :wink:

2 Likes

What would your ideal solution be?

If you wrote it in a low level language, you’d have to do something special to make it moddable. Writing it in a high level language makes it easier to mod, but requires finer tuning for performance. It feels like I’m saying “they haven’t gotten to the tuning yet”, and you are saying “the game cannot be tuned”, and I am not sure on what basis you are saying that.

What would your ideal solution be?

Give me some time and I’ll put together a description for it and post it as a suggestion (in a different/new topic to avoid cluttering up this topic more).

If you wrote it in a low level language, you’d have to do something special to make it moddable.

Why? For some things (rendering, path finding, job control, liquid physics) performance is more important than making it moddable (and it’s worth the extra development time because it has a significant impact on performance). Modders would still be able to create new roles, new types of jobs, etc; they just wouldn’t be able to change the “which job gets done when” code.

Writing it in a high level language makes it easier to mod, but requires finer tuning for performance. It feels like I’m saying “they haven’t gotten to the tuning yet”, and you are saying “the game cannot be tuned”, and I am not sure on what basis you are saying that.

You can fine tune LUA as much as you like, it will never be comparable to the performance of the same code written in C, C++ or Java. LUA is fine for things that aren’t executed often enough to matter. It’s not fine for the 2nd largest performance bottleneck in games like Stonehearth (and not fine for what is currently the single largest performance bottleneck in Stonehearth).

I’m saying that to convert job control over to C++ is definitely technically possible, but would require significant changes throughout a lot of existing code (especially if it’s done properly), and therefore won’t ever be considered profitable and will never be done. I hope I’m proven wrong; but I’ve seen enough commercial software development over the years to be sceptical.

1 Like

Lua is pretty fast for a scripting language, but in general is not fast enough for high-performance code. For performance sensitive components, we write high-speed primitives in C++ and call them from lua. This includes pathfinding, rendering, ai scheduling, computational solid geometry, and object tracking.

The in-game performance bar shows a breakdown of the time spent in each area. If your game is slow, you can click on the bar and see what the rough bottleneck is. It will also help us if you post a screenshot so we can see the layout of your town and where the time is being spent.

1 Like

So wait, does that include the job scheduling? Is the job scheduling already in C++?

Job scheduling is in C++.

2 Likes

I had 8gig ddr3 in my system and doubled it too 16g ddr3. Now i have 36 citizens on speed mode 2 without any real slowdown issues.

1 Like

Oh weird. So I guess I’m kinda baffled at how much of a performance hit the extra villagers are costing. Though I guess I’ve never actually played a comparable game (Dwarf Fortress’ UI is terrifying to behold), so I I wouldn’t be able to guess how much of a jump in processing cost there would be from having all the same things going on in a fully 3D setting vice informed properties locked to grid positions.

Job scheduling is in C++.

Then I’m baffled.

Would you mind taking a look at this:

http://discourse.stonehearth.net/t/job-control-scheduling/23401?u=brendan

…and telling me if tell me if Stonehearth’s job control/scheduling is comparable?

Thanks :slight_smile:

For a game with 42 hearthlings, when running at 3x speed, with no building, mining or anything happening (mostly just hauling, farming and patrolling); it jumps around a lot (a static screenshot won’t tell the whole story) but it’s about 50% “lua”, 10% “luagc”, 20% “astar”, 10% “bfs” (and some smaller stuff).

If LUA isn’t being used for path finding or job scheduling, I’d be curious to know what it is doing that’s taking a massive amount of time (more than what should be the 2 largest performance problems combined).

Note: The path finding (astar, and I assume bfs too?) is perfectly understandable (path finding is always expensive).

Well, the I stands for intelligence. Hearthlings don’t have any understanding of who to attack best, where to hide, no real sense of self-preservation, they drop things they are carrying on the spot when they are hungry, even if the food is in the same direction they were going in anyways.

Bottom line. Hearthlings are increadibly dumb. So yes, of course they have an AI, but you said it was super developed compared to other games an I disagree. It is pretty barebones and doesn’t justify the amounts of performance it eats.

almost sounds like a lemming :slight_smile:

:astonished:

My PC starts to slow down at like 25 hearthlings at x1 speed, the performance bar always shows 50% astar 50% bfs, I honestly haven’t a clue what either of those mean or if there is anything I can do to help.

Those are both used for pathfinding. Can’t really do much to help that but keep your towns small and not very complex I would say.

I did not say it was super developed compared to other games. I myself disagree with that. I was asking you what would be AI for you, but you did answer that, so I’ll reply:

They currently choose their target by a combination of its remaining health and how threatening it is. I find them all to be concentrating on one target, which is really awesome (at least while they’re in melee). Sure, when there are ranged units, it’s different, but part of the whole point of enemy ranged units is to break up your army by kiting them and picking them off one by one. That’s up to you, the player, to deal with. Hint: traps, spiked arrows.

Gotta agree with that one. They don’t really think ahead when they run away. That said…

…I’d say that their sense of self-preservation is TOO real. When you’re inches away from death, facing something stronger than you that wants you dead, I don’t think you’ll always have time to find the best path to escape – your primary objective is to get away as far as possible as quickly as possible, and 9 times out of 10, their panicked flailing does accomplish this.

True, it’s a bit annoying. But I wouldn’t say that this little thing is equivalent to “no real AI”. It’s just under-optimized AI. See my other replies in this thread about tuning code.

Would you rather have 80 hearthlings who flawlessly, performantly do nothing? Or do you want things like buildings, ranged combat, mining, enemies, quests, classes?..

SH team is 6 people. You can’t do everything at once with that small a team. Especially one without any previous projects as a team – they’ve all gotta get to know each other too. SH right now is a small, unpolished version of a much grander vision. The alternative would be super-optimized hearthlings who WOULD do everything without lag… were there anything for them to do, since everyone spent their time pre-optimizing hearthlings.

What Avairian said, also avoid doing work far away from all your hearthlings. The further they have to walk (and the more of them have to find their way there), the more you’ll see those two flare up.

1 Like

Exactly. The player has to manage, that’s the very opposite of AI.

This is not about where they run when getting slaughtered… this is about seeing enemies approach and continuing to do their farming or running at them not waiting for back up. Those are two serious indicators of lack of AI.

And yes, I know most of things CAN be addressed or changed. We are talking about the game as it is though, and not some potential future.

You seem to completely miss the point. I didn’t say that the hearthlings are horrible and I neither said the Ai needs to improve or become worse. What I said, and you completely ignore by asking me things like if I want unmoved statues instead of hearthlings, is that the AI is NOT sophisticated enough to justify the bad performance it causes. That was my point from the very beginning and I think I have given more than enough examples where the AI is very lackluster, while still eating my PC when I have 30 idiots, while I can run new AAA titles at decent graphics and good performance. So stop defending something that doesn’t deserve being defendet. I love Stoneearth, but that doesn’t mean I cannot point at obvious problems.

Alright. Yes, it is a problem. An obvious one. They’re working on it, so I don’t know why you’re pointing at it. Relevant post by Albert:

We’re not in the “optimization” phase yet of the current batch of alphas. I reiterate my advice: leave for a few months, if slow hearthlings are upsetting you. I’ve said what I wanted on this subject.

I was, like other people in this thread, stressing that performance is a major concern and therefore hopefully showing the devs that I would love performance updates sooner rather than later :slight_smile:

2 Likes