I've used a number of life vests over the years and in the last couple of years have settled on the kayaking styles as being the most comfortable.

I'm currently using the Extrasport Retroglide Avenger:

But, it has a tendency to ride up - it's being pushed up by the high, stiff back support on my trapeze harness. I have to wear a rashie over everything to hold it down.

Alternatively, I also use a Lotus Sherman:


It sits higher, but the foam blocks are thicker, so when I wear a rashie over it, it makes me look like I've had . . . um . . an implant operation.

I've never had a problem swimming with either vest, but I did find that the minimal amount of flotation in the trapeze harness (in the butt) made me want to roll face-down if I was just floating in the water.