Yes there is a magic formula. The texel rule states no part of the pole shall extend horizontally more than 800mm longer than the distance from your beams to the bow. This is the baseline that most cats are using today.

As far as height it has been argued that a safe distance between the tang and the mast head is anywhere from 1/3 to .65. If I remember correctly, Bill Roberts says 1/3 is the place where you can accidentally release the mainsheet with a spinnaker load and not break your mast. I can tell you that most of us do not heed this advice and use between 1/2 and .65. There is more risks involed with making mistakes when going higher.

Matt