Not a good answer but...

One method, not very scientific, is to use the breaking loads of the proposed rigging...prefer the rigging to break before the hull.

That brings up the issue of rigging selection though, which is where experience comes into it.

There are rules of thumb for design loads that have the vessel mass and righting moments as the primary inputs. I have no links for these.