I think this was talked about some time ago. The topic being winglets, the term used for plane wings. Instead of on the end of a batten, why not the mast itself? The idea of a winglet is not one of added lift, rather it is of decreased drag.
One could make a winglet for a mast that even had some floatation in it, no? Most winglets are on the top surface only, even the airbus has a larger surface on the top, than on the bottom. I think it was conveyed that the winglet on a mast would indeed count towards sail area.
As far as vortex generators go, I don't know that you would get the same effect on a sail as you would a wing. The purpose is indeed for 'slow-flight', but, it is mostly to keep attachment over the aileron.
Might be curious to see this develop thru the years....