I think Luiz has hit on the biggest problem when it comes to winglets on sails or foils on sailboats. We are not operating in "smooth flow" all the time like an airplane is. With the up-down motion of a sailboat going through waves, there is a side loading on the winglet, from both sides, as it goes up and down, and I think that constant change of flow direction will make them quite a bit less efficient than on an airplane wing.

It may just increase drag without any real increase in lift.