Nick - do check out Fairwinds - you only need to clear the piers before you're in clear water - there's enough room to sail out and it's not too long of a paddle if you're becalmed or not confident of the room. They're near the mouth of the Magothy River, so you have the option of staying on the river or venturing out into the Bay. They have a website, do a Google you should find them.
But by all means come on out and check us out. Directions are on the website. First gun for the racing is supposed to be 6:00, usually it's a little later. People show up to rig starting around 4:00, some of us blast in at the last minute and try to get on the water on time... We end up at Pirate's Cove afterward, and you can see the start and finish from their pier.
Our only thing is that we try to reserve our spots for folks who will try to make at least half of the races. But once in you can use your boat whenever you want.
And you're right Nick, it would seem that the "sailing capitol" would have better access for people to go sailing. It would seem that it's the sailing capitol for those with the scratch to join an established club, buy waterfront, or live in a water-privileged community. If you want to show up, launch your boat and go sailing, well, sorry mister! Oh, and leave your extra hulls behind! Most people around the area believe you should only have one with a lot of lead attached.
