Or the boat was carrying (alot) of ballast to get it up to specs c.q. F104 compliance.

The F104 class has no rules limiting the amount of ballast you can use as the true formula classes like the F18 have.

Of course this ballast will the first thing to be "forgotten" by private owners of these boats.

Pretty much the F104 class doesn't have much rules at all beyond the requirement that all boats must be rated between 1.035 and 1.045 when using the SCHRS handicap system. In fact, a F18 can be made to be a F104 by rather small modifications. In fact, the easiest route to do this is to fit the official small F18 sailplan and add about 10-15 kg of weight to the boat.

Wouter