Let's propose a hypothetical situation: Two identical F-16 boats, one 10 years old and one new. Both with new sails and rigging. What would make the older one less competitive?
The sailor!
Changes in hullform/rig
Construction methods
Seriously though, while listening to the wind, rain and enjoying that our current +5degC temperature stays outdoors..
A well built boat will be competitive at 10 years of age. A boat not designed for a 10+ year competitive life will be more flexible loosing speed. Pretty much all structures will flex and over time loose stiffness. If the flex becomes pronounced, you can even experience material fatigue and material failure.
Repairs will over time add weight and more flexibility.
Humidity goes into the sandwich, adding weight.
Store outdoors in wet weather, add weight.
Mount beams improperly and add flex.
Basically it can be summed up with increased flex, increased weight and design changes in hullform/rig/appendages/fittings/lines etc.
But the most important performance factor on all boats is the sailor anyway..