I think the real problem is that only farmers can harvest.
-Allow other hearthlings to harvest things as well.
-Make growing crops a bit harder by having the farmers remove weeds and do trimmings and water them occasionally.
So only a farmer will know what plants need what, but anyone can walk into a field and see that the food is ripe and harvest it.
This makes getting your fields producing the only thing the farmer has to worry about.