There are a few thoughts to this:

Testing the TAIPAN 4.9 with and without the spinnaker (snuffer) it has been noticed to be slower up wind with a snuffer. However this was at the upper end of the wind range, and the testing had it's limitations.

A much more accurate test was done in the US on the Tornado's. Back when they where testing to figure out the best position to place the snuffer, there was extensive testing done to see whether the drag on the snuffer was a major disadvantage. The result was there was drag but it was so minimal they would be better focusing on other improvements.

Those "case studies'" aside the physical of the problem is:

The drag force is much higher at higher wind speeds. For the mathematically minded - the drag raises to the square power relative to wind velocity. This means that there is more drag in higher wind ranges, but then you have more power to over come this.

I would be very surprised if you're losses up wind where more than your gains down wind. If you are loosing out around the course, I would say it it something else apart from the snuffer.