Still not fixed after years: Materials are NOT handled threadsafely and WILL break production

Summary: Crafters claiming the same resource have always been a big issue in this game. Once you run low on a resource and have 2 crafters using it for crafting, they very frequently break each other’s crafting queue. The one not getting the item first will NOT produce anything anymore until you delete the order he was trying to execute in the first place.

Steps to reproduce:

  1. Have a low supply of e.g. stones
  2. Have mulitple crafters use them for their orders, such as a mason crafting benches and a cook crafting his furnace and whatnot.

Expected Results:
Items should be allocated to whoever first asked to use them up. The crafter going second should notice he’s beat and return to his queue, searching for another thing he can craft with leftover materials.

Actual Results:
Racecondition: First to reach the item gets the item. The second will immediately quit doing his job, his latest order will still be marked as being executed.

Notes: Put the requests in a list and make ONE thread decide on who gets which item. Iterating through is not the best way, but I don’t know which kind of multithreading technology you can use.

Attachments:

Version Number and Mods in use: No mods, latest from Steam. Problem has always been there since I first played the game.

System Information:

1 Like

TLDR: now crafters will continue crafting if they run out of ingredients in the middle of collecting them or if they can’t reach them. Hopefully this alleviates the last major pain point of the crafting system and we won’t have to babysit the crafting queues so often. :slightly_smiling_face:
This fix is available in the unstable branch.

Now sorry for the wall of text, but I wanted to clarify: if you read how does our AI system work: Docs for the updated AI system and take a look at the game files, you’ll notice that it’s impossible for two hearthlings to reserve the same item.

Crafters start an order because the inventory trackers say that we have enough ingredients for it and a workbench placed. It’s the player who has to make sure to have enough resources for all of the crafters, these should only worry about their own queue. Crafters are not smart enough to predict that they will run out of ingredients while collecting them (maybe someone else is also using them for building, crafting, etc. Too many edge cases).

The stuck crafter is idling/doing other things simply because they’re looking for their current ingredient in the world, and can’t find one (no more stone left in your inventory) OR can’t reach it (it’s in an unreachable place - also possible with workbenches). They haven’t reserved anything yet at this point. You can observe this in the AI inspector from debugtools. They can’t continue crafting until they find that ingredient (any wood log, any copper ore, whatever) in the world. They’re still trying to find it, it could appear at any moment. Once they find a path to it, then they can reserve it, and start walking to it to pick it up.

The collect ingredients orchestrator can’t be interrupted that easily (how do we know that the collection failed? What if you just ordered to chop a tree because you know you’re running short of wood? Or placed a ladder to some unreachable ores? The crafter could continue crafting their current recipe if they just waited for a minute). In the case of running out of ingredients, it’s a bit easier (though it can still have edge cases), but isn’t it better to fix both instances of the bug at the same time?

So the only workaround we could come up with is that if they can’t find an ingredient+find a path to it+reserve that ingredient in a while, mark that recipe as stuck and reset it (drop any ingredient that was already collected), so that the crafter continues working on another recipe. The downside: recipes marked as stuck will only be reconsidered after the crafter reaches the end of the order list (so when there are only stuck or uncraftable orders left). If we refreshed them earlier, crafter might end up picking the same order that has the unreachable ingredients again and again, and none of the orders below would get crafted.

Also, there might be false positives where the order will get reset when there are still ingredients, because there are edge cases where the timeout may trigger at the wrong moment (like, someone picked up the last of that ingredient to restock it, so crafter couldn’t find a path because it is in the backpack of that other worker, but as soon as that worker drops it the crafter would be able to find it, find a path, and reserve it. Or, crafter was about to find the ingredient but got hungry/sleepy and went to eat/sleep, the timeout hit right before they got to bed and the order got reset. Etc.). The timeout is long so that crafters have enough time to find the ingredients, and can be tuned in the constants.json file (might be too long for normal games, or too short for laggy savefiles). Hopefully these scenarios won’t happen that often anyway, and if the crafter runs out of ingredients they won’t reconsider the order again until it’s ready so that’s fine.

There’s also a delay before reconsidering stuck orders once we reach the end of the order list, to prevent a race with the thread getting suspended. New orders that you add when there are stuck orders in the list will be considered right away, from top to bottom.