crazy idea, but, bear with me here…
perhaps try rebuilding the gardens, the kitchens, and any other “living areas” and their attendant buildings and farms etc. all in one area? That dramatically cuts down on unnecessary walking between different parts of your town; and it often looks really nice to have some “veggie gardens” next to your kitchen buildings anyway.
Like, yeah there are other options you could look at; but at the end of the day the most effective one is usually the simplest/most straightforward too in a game like this. Keep things close together and everything just works better to start with.
But, if you do have some grand design in mind which requires a large dedicated farming space… my suggestion would be to provide your soldiers with housing closer to the farms, and move your kitchens near the farms too, then create a kind of barracks (by which I mean a building with a dining room, meeting/training space, and somewhere safe for citizens to hide during an attack with a rally banner there) guarding the path between the farm and the rest of the town. That at least gives your soldiers somewhere to eat and sleep which is mid-way between the places they’ll be patrolling rather than being at one end of the route. I’d also suggest taking a bit more control over the soldiers’ groupings – your footmen will run ahead of the other soldiers and get caught out, so it’s better to keep them in a separate group and when there’s any danger send them to a defend flag while your other soldiers engage the enemy. Then, send your footmen in as reinforcements – not only will they be safe from getting singled out, they’ll actually be really good at smashing through any attempt to surround your other soldiers.