Most of you have spoken against tethering yourself to the boat, but what if the tether also deployed a sea anchor?
Attach a line, say 25', to the boat, stow it in a bag on the tramp. Attach a sea anchor to the bitter end, then a short line from the sea anchor to your trapeze harness via a quick disconnect. If the sea anchor didn't slow the drift of the boat enough to get back aboard you could still disconnect and try swimming.