Nice, except for one thing that is missing, big 20x20 stockpiles!
I heard in the stream that the game is fine with up to 64 entities in a 16x16 chunk, well, a 20x20 full stockpile will for sure occupies all squares resulting in 256 entities in 16x16 blocks.
In this video the stockpiles are small and not really full.
What I normally do is have one big 20x20 stockpile at the start, and when it gets full, I create a second one, and start branching items into different chests with filters. At this point the games starts to lag.
So I recommend to create this case in it to get another different lag reason. And also try to use bigger builds in it (default templates would be good) and scatter more items around the map.
I want to see chaos.
Every time I hear the explanation of how the AI works, with every hearthling (and creature) evaluating a list of possible tasks to do, I wonder if that isn’t a very inefficient way to do it. I guess it’s safe to assume that it’s a simplified explanation and highly optimised and so on, but I got to ask… wouldn’t it be more effective to have an AI that takes the role of a micro managing player? It would go through the list of things to do, sorted by priority and hand out work to the idle hearthling that is best suited for the job. It feels like that would lead to a task being evaluated fewer times. It might also make it easier to coordinate the hearthlings. For example: “Yes, you are idle and there is a log over there, but Alice is a lot closer and she will be done with her current task in 2 seconds, so let me find something else for you to do.”
Maybe it amounts to the same thing. I don’t know. I haven’t been digging in the code to try to find how it works, but maybe someone have some insights to share.
Yay performance optimization! maybe now my soap castle will finally be finished! 
I just commented in another post about AI performance - Hearthlings going idle and the game starting to slug up after Day 20 and 20 hearthlings - so this is the perfect place and time for focus on this … again you guys are spot on with your development priorities!
Like many other fans, I’d love to see some of those new systems and content you previously had planned for A15 but you’re so right - if performance can’t be maintained beyond 20 Hearthlings and maybe 5-8 hours of play on one save-game, then all those spiffy features aren’t going to be fully realized or balanced. We can’t even realistically test more right now.
But honestly, system performance has increased overall. I’m so happy about that - it’s solved a lot of my play problems in A14 … so onward and upward with fixing the AI issues next. I’m a player who looks forward to playing one save-game and world for long-term play … slowly building from a tiny village to a full sized Hearthling city! I’m also a Sims 2 and 3 fan and I fully know what happens when you overstuff a household or town with hundreds of individuals - all engaging in their own wants, needs and work queues - it can crush a lot of systems, even with the best processors and memory. I use a lot of Sims 3 mods that fixed a great many pathing and other issues EA never bothered to fix but still, there’s only so far you can push a complex AI running millions of processes for hundreds of things in the game. I guess that’s truly why Dwarf Fortress, the Granddaddy of all large-scale base building and management games is done entirely in ASCII … in 3D it’s crushing!
I will help as much as I can… playing long term and larger populations as you see fit. Right now I have only stayed at the default 20 Hearthlings and see the issues at that point. If you want us to start testing with more than that, just let me know!
YAAAAAY Crushinator!
It is horribly inefficient, compared to, say, a single repeat until false that does all the AI stuff and yields from time to time. But the AI wasn’t built with performance as its primary goal; it was built with modularity, reusability and extensibility in mind. All those things don’t go well with performance, but offer many advantages instead.
That’s kind of true, but also done at the same time. While there are AI actions that are evaluated on all hearthlings, most of them are inactive most of the time. For special tasks, such as harvesting things, there’s the task scheduler whose task it is to assign tasks to individual hearthlings.
In the end, it’s a design question. For example: For picking up items, instead of having each hearthling search for an item, you could rather have a BFS performed on the item and find the first available hearthling, then assign him the task. This could - depending on the engine details I’m not aware of - help performance a lot and there are quite a few scenarios that would profit from such an inversion of control.
I suppose these kind of things will be looked at much closer now. Even if there would be a performance gain by doing this inversion, it’s questionable whether it’s the right thing to do in regards to modding/the general architecture.
Thanks for the good reply. 
I agree. The positive things about the current way to do it is most likely worth the possible performance loss. Especially if it makes the modding easier.
I realise this was probably just a quick example from the top of your head, but considering that there’s usually more items than hearthlings and there’s no point (I think) in looking for a hearthling to assign the task to until there’s an available hearthling I don’t see how that would be an improvement. I admit that my memory of what a BFS does is a bit fuzzy at the moment though, so maybe that’s why.
Unless you predict when and where a hearthling will be available. That way you might be able to que up tasks for the hearthling that is actually able to get there the fastest instead of just considering the hearthlings that are currently idle.
Was that what you meant? Is there currently a prediction like that?
Sorry if I’m rambling. It’s late and I should sleep. 
Items don’t move, however, so they don’t need to re-calculate their path often. Their targets do, admittedly, but there’s way fewer of those than there is of potential items - i.e. there could be lots of optimization.
However, here’s a more detailed scenario which would work: Restocking. Unless it changed in the meantime, each storage (stockpile, chests probably too?) as scheduling a task in every hearthling to refill it. The task was basically “Search for any item that can be placed inside me, then go to it, pick it up, go to me, drop it off”.
This means for n hearthlings and m stockpiles, we had nm pathfindings going on, all the time - revalidated whenever an item was removed or added to the world. If we went for a different approach, it would look much better: Each stockpile would perform the search and, if an item has been found, looking for the nearest hearthling (if any). This means that we have usually m pathfinders running, unless we have found an item, in which case we’ll have a (hopefully shortlived) other PF that looks for a hearthling that could perform the task. Technically, because the list of hearthlings is known, and a list of available hearthlings (can_carry | not_doing_anything) is even smaller, we could maybe even head for an optimized pathfinding algorithm that is able to deal with multiple targets (for example, some cruel A with multiple nodes that can be the target). So we would have m big pathfinders, and then up to n smaller for each item that has been found.
Is that an improvement? Maybe. It depends heavily on how the engine is built, what kind of optimization it has. For example, is it better to have many pathfinders running than those with potentially long paths? Can I search for multiple targets in one go? Is a directed pathfinding (A* or similar) possible, or is it all some sort of BFS/Dijkstra?
I’ve been enjoying Alpha 14 a lot, and it’s so far been the most stable and efficient build I’ve seen. For example, I’ve actually got a town with 20 hearthlings in it now, which would have murdered my VM in a previous version long before it got to that point. I play in a VM so the game tends to slow up a lot sooner than a lot of other people’s, I expect.
There is one problem I’ve run into that I don’t think Crushinator is testing for, from the look at the video, and it relates directly to the new combat stuff we’ve gotten recently. After I dealt with the goblins and their wolves, the undead start spawning crypts. Well, these seem to spawn so far away since I’ve pushed the Fog of War out that it’s nearly impossible to send my troops out to destroy them and end the threat, and my workers never loot any of the goodies if I do manage to get my troops to kill everything. The pathfinder seems to still be struggling over long distances through heavily wooded areas. I feel that if I clear out some of the trees (which of course bogs down my workers more since they have trouble going so far to get items or do tasks far from home), then my footmen can travel more easily over long distances, but it’s still a struggle. I’ve also noticed they’re more willing to venture out far from home, but getting them to come back is an exercise in giving them many short-distance orders and basically micromanage their movements instead of dropping a banner in the middle of town to get them home.
Anyway, I’m looking forward to more optimizations. I already feel like 20 hearthlings is not enough now that there’s so many classes. I’m running a town with just 2 footmen, and when you add in 7 crafters, a trapper, a shepherd, you’re left with only 9 spots for your farmers and workers. It’s actually less than that, realistically, because I find without a second blacksmith it’s impossible to get any armor and weapons made in any reasonable period of time. You need one blacksmith churning out bars and another actually making stuff. I’d love a second weaver making thread and cloth too, but I just can’t give up any more farmer and worker slots if I want a town that’s well-fed and getting resources and items moved about. I can just forget about buildings, all my games end up being just beds under the open night sky because I never have enough workers that I can spare some for construction projects. Oh, and did I mention I’d feel a lot more comfortable if a large town had 4 footmen instead of 2? My two 6/6 body/spirit footmen are decked out in steel and fear no enemy, but there’s a lot of ground to cover in a big town and they can’t be everywhere at once to protect it.
So, I think for the number of classes we have now, and the amount of work I need to get done, I’d want 1 carpenter, 2 blacksmiths, 2 weavers, 1 cook, 1 potter, 1 mason, 1 herbalist, 1 trapper, 1 shepherd, then 6 farmers to feed them all and provide materials for the herbalist and reforestation, 4 footmen to defend them, and then no less than 5 workers just to mine, chop, and move stuff. That’s 26, and that’s just right now. That’s where I think the endgame has to be optimized, with the current enemy threats and current classes in game. If you add in new boss monsters and a bunch of new classes, that number just has to go up. I figure in the long-term combat might involve four footmen, two archers, and some kind of support class, so call it 29 as threats ramp up and we get some ranged units. There’s still a lot of unknown crafter classes as well in the tree, and figure that for every two new crafters added you need another worker and every four citizens in town needs another farmer.
Anyway, that’s my endgame math. Other people may have other ways of playing, but I’m hoping this game gets optimized well enough that we can all play the way we want to. The way things seem headed, I think a game of about 40 hearthlings is what I’ll end up needing.