So what is right of way? Right of way does not give you the right to do anything you want. If you have the right of way you are the privileged boat. The privileged boat MUST maintain her course and speed. That means you can not iniate a tack in the middle of a channel and expect everyone to go around you!
The rules also state the any vessel that is "restricted in her ability to maneuver" has right of way. In a narrow channel this could very well be a large power vessel.

Also from the USCG "Navigation Rules should be regarded as a "code of conduct" and not a "bill of rights". They do not bestow rights or privileges, but impose the duty to either "give-way" or "stand-on", dependent on the circumstances. What is important is not so much what things are, i.e. sailing vessel, operational, etc., but how to avoid collisions, e.g. although under sail yet able to be propelled by machinery, obtaining an early warning by radar, etc. Understand, the Rules are in place to "prevent collisions" not to define nautical terms or to be subjected to strict interpretation."
So reality you don't have the right to anything, period.

I don't know much about your area, but here we are on a Bay that is actually part of a salt water tidel river. When I'm in my 41' power boat I have no choice but to stay in the channel, restricted by draft. But when I'm in either the whaler, hobie, or sunfish I stay away from the channel. Sure there are spots in the bay that are only 6" deep but I know where they are and stay away from them. The rest of the bay is plenty deep to sail or run the small boat. I feel I don't need to get into anyone's way in the channel. There is absolutely no reason for me to be there. If possible, stay out of the channel and it won't be a problem.

Many powerboaters may be indeed be idiots, but many more sailors are on a horse so high that they can't even see the water.