Here, I have a simple way to look at this and exclude the discussions about physics, fluid dynamics etc.

Look at the US Sailing Portsmouth numbers and compare these three boats; M20, CFR20 and I20. They have different rigs (uni vs. sloop) but carry similar sail areas both upwind and down wind (I know the M20 is 1' wider but it's still a good reference point). I probably should have pulled the Texel numbers and compared the I20 to the Eagle 20 Carbon but.... I did this....

Key
Boat - weight, SA up x SA Down, Portsmouth numbers

M20 - ~255lbs, 24m^2 x 25m^2, 58.0 61.0 59.0 57.5 55.0
CFR - ~265lbs, 24m^2 x 25m^2, 58.0 61.5 58.2 58.1 57.3
I20 - ~390lbs, 24.25m^2 x 24m^2, 59.3 62.0 60.2 58.5 57.5

Notice that in light/med conditions the light boats are rated faster. As wind speed goes up the gap closes for the I20 and the CFR but not the M20 (the beam starts paying off).

I think the conclusion should be obvious. Lighter is faster. (don't rake me over the coals a/b the sloop vs. uni, I know I know....)

One more anecdotal account. I have sailed with some upper upper level sailors. If weight didn't matter, I don't think they would be so anal about using small light lines and keeping the garbage off the boat.