Sitting at 22 villagers, may have started at around 20, I don’t remember exactly. I had a goblin camp for a long while before the freezes started. Very little mining or exploration done, a lot of buildings, some farms but not much. With your experiences as well, sakinoko, it may be villager related.
Also, it’s not just villagers that freeze, but the animations for lighting as well.