yes, this is my problem.

Upwind when you get a gust you do not change to much, 2-3 degree max. But downwind sometimes I need to bear away 15-20 degree, or more. So no stable reference direction.