Kayaking PFD's are the best in my opinion. They either have a strop going between your big-toes (but a bit higher), or are a tightened at the sides after you put it on. This keeps them in place
We have been swimming plenty in our kayaking PFD's, and have not had any problems with them.