Hmmm. I've never had a problem swimming with my life vest (Magic Marine). It fits snugly, and doesn't ride up.

If you're used to swimming with your head out of the water (like surf swimming or open water triathaletes), it's relatively straightforward, although it takes more effort than if you just had a bathing suit on (more drag). The spray pants I wear probably cause the most drag...

The photo with the rasher over your life vest should also keep it from riding up if it (the life vest) fits properly.

How about a short piece of spectra to tie it to your harness (which has leg straps)? I'd use a quick release knot if for some reason you have to ditch the harness..