Sailing isn't as much a spectator sport as it is a participant sport. I kind of like it that way...
You'll never get the average Joe 'Merican to appreciate this sport because most of the landmass is not coastal in nature, so there aren't as many places to sail (versus, say, basketball courts).
Also, sailing involves tactics as much as skill and technology. Regattas require figuring out place standings rather than "who's still standing", so our buddy Joe won't understand how GBR placed first in the "medal race" but didn't win the gold medal.
Joe likes 'em to go fast and crash often. IF Joe has seen sailing, it's probably been really slow Flying Scotts or cruising boats going around in a triangle at his local fishin' hole. Joe thinks multihulls are "hobies" and skiffs are "dem funny boats"
Media follows the money, and spectators follow the media.