As long as the suit doesn't leak and you wear enough insulation under it, I would think you'd be fine. Lots of people wear drysuits for sports where they're frequently immersed- kayaking, windsurfing, kiteboarding. Personally, I feel a little safer in a wetsuit/steamer because there is no risk of a catastrophic failure if the suit is damaged and leaks. But that said, many people have gone over to drysuits in recent years and the good ones have become very dependable.
But why not just test your suit out in controlled conditions- pack on some fleece and go for a swim in shallow water close to shore and see for your self.

Also, a lot of people seem to go by the 100 degree rule- air and water temp should add up to 100 deg F before they'll go out. Personally, I know I've pushed it beyond that, but much farther and you're going to have trouble staying warm in any type of suit.

sm