Something is asymmetric. Your movement or your boat.
If you have a small video camera like the GoPro Wide, you can screw it on the end of the boom pointing forward. One time left side, one time right side. You can analyse your moves afterwards.
Check your boat. Rudder alignment, is the tiller extension in the middle of the crossbar, is the main sheet really centred or slight offset, points the mast to the left or right, are your boards symmetric, the hulls, diamond wires, and so on?

Cheers,

Klaus