With boats, you can't look at a single design parameter, like jib foot length, as a decision on its own, you have to look at the influence on the whole boat.
For example, for an AC boat, jib foot length would be a function of mast base position, mast strength (how much compression can the mast handle which primarily comes from forestay tension, which you don't want to sag too much otherwise the jib won't set and you can't point), hull strength, hull beam (which affects the slot significantly), overall helm balance (which is affected by the underwater body, appendages and sail distribution, vessel trim, heel)...and the list goes on!!! Design decisions like that feed into other design decisions which feedback into the original design idea.
Most of these interactions are very difficult to compute so plenty of on-water training and development is required once a good baseline boat has been developed from experience, good rules of thumb and functional engineering/scientific analysis.
Yeah, you're right, it is a bit like alchemy.
One way to overcome this mess of a design problem is to use tools called
genetic algorithms with some velocity prediction programs and race prediction programs (with typical encountered weather as input) to evaluate potentially thousands of design modifications somewhat automatically.
It's a pretty cool concept, a mate of mine wrote a basic evolutionary program to see how two different coloured lines of pixels would interract. Funny to watch them grow/breed and die.
These tools are probably beyond the capabilities of mere mortals, we will just have to stick to experience, gut instinct and practice.
