While working with tweaks on mods, I tend to just go as quick as possible between the Hearthling screen, the Starter-Pack screen, as well as the Map-screen.
I’ve never had it bug-out on me; I have no map-mods though - I only got TA and RC.
No idea what could cause the difference between us, I just wanted to provide my case. Maybe it is RAM-based or something similar (hardware)?