Jon,
If it is cold enough for you to be worried about it, it is cold enough that you should not be sailing alone. If you are sailing your own boat solo, you should at least be doing the buddy system and go out with another boat, so you can keep an eye on each other. In fact, that is always a good practice, even in warm weather and warm water -- especially when sailing singlehanded.