They're racing! Think about it - the fastest sailors spend the least amount of time sailing.

Plus, the stamina thing kicks in. First few days are trying, middle days are agony, last few days seem easier. Again, this is from a shore crew point of view, but it seems like the sailors physically hit a wall, climb over, and the last part of the race isn't as hard on them. And I've never heard any of them say, "gee, that 14 hours of beating to weather in 8 knots with 6-foot, confused seas was FUN!"