Matt, there is navigable water around the line and between the two marks that make up the imaginary start/finish line.

I was in this situation where we were both on starboard, heading to the weather mark. Leeward boat was maybe half a boat length in front. He came up and pinch me against a dingy tethered to an anchored yacht. I had no where to go and eased off my sails to fall off and prevent hitting any vessels.

I was told there was navigable waters around the yacht and dinghy so I had no rights for room.

I think this situation is similar...