I have a sail of the same sailmaker and I'm sailing with an almost straight mast when cat rigged. If I don't do that then I get what you have. Other things to tune are traveller and the foot of the mainsail. Both seem to aggrevate this behaviour when set wrong.

Otherwise your rudders are mistaligned.

Don't compare your settings to sloop as both can be rather different.

I went down the same road with my redhead mainsail and now it sails much better without throwing up roostertails. Roosters tails are very draggy and can really hold you back. Everything and anything that adds drag to the boat when in cat rigged mode will kill speed especially upwind. Here being low drag is key when sailing an F16.

I've just finished a hole year of singlehanding my F16 in the races.

Wouter