Worldbuilding is fun, and you definitely need to think at least a little about how a setting Works for it to make any sense, but I sometimes get the impression that people on this website have forgotten the extent to which you can just kind of wing it.
You should probably have an explanation for why there are potatoes in your quasi-European setting, but you don't necessarily have to develop a five hundred year overview of the history of Fantasy Peru unless the story is actually set there. It's probably sufficient to say "imperial conquest that is a broad allegory for real world events" or "this world is filled with traveling bands of botanists" or "the world was a single supercontinent within living memory, and then some crazy shit happened, let me tell you about That" depending on the themes and ideas you are working with.
There just are.
JRR Tolkien/Jules Vernes worldbuilding vs. CS Lewis/HG Wells worldbuilding.
I'm not sure why if something is set in a fantasy world, there needs to be an explanation for why its foodways are different from the actual planet Earth in the early part of the second millennium CE. It's a different world, presumably a different planet or dimension, of course it isn't exactly the same, just like how the real historical Middle Ages didn't have dragons or elves. I don't see why you need to explain the flora if not the fauna, unless you're making some kind of point with your story where it's relevant, or you just want to have fun with seeing what you can do with food when you eliminate New World species.
