I would rather say that this would be realistic.
I’m working as an IT guy, but I could drop my job anytime and start learning something I’m intrested in at any time.
I wont lose my IT Skills Suddenly all at once and I will never lose them all.
BUT I will step down in use for IT Tasks, if I’m not keeping my skills ongoing and sharp with training.
Thats would maybe be another option.
A demoted “Expert” will learn a new job slower at the beginning, but while he gains mor EXP in his new job, he lose EXP in his old job until a certain point/level.
When he reaches that point he starts to get normal EXP for his new Job.
When he switches back, its the same thing but he starts at a “higher” level than a complete stranger to his old job.
Would balance the field pretty well.
As example why we need some kind of demoting (exept for the 15 Soldier thing :D) my City was joined by three carpenters… why would I need three of them? Maybe later in the game after some (ingame) month/years when the City goes big, but until than a second Farmer, Soldier, Trapper, Worker or something else might be much more of use.