Matt, there is navigable water around the line and between the two marks that make up the imaginary start/finish line.
I was in this situation where we were both on starboard, heading to the weather mark. Leeward boat was maybe half a boat length in front. He came up and pinch me against a dingy tethered to an anchored yacht. I had no where to go and eased off my sails to fall off and prevent hitting any vessels.
I was told there was navigable waters around the yacht and dinghy so I had no rights for room.
I think this situation is similar...
This falls under the room and oportunity. On approaching an obsticle that is long like an island or even a s/f line and they have to come up in time that you can tack away to take the water on the other side, it is still a normal course and no foul. If it is something short like a moored boat legaly they can wait much longer before heading you up because there is room. If they wait until the last minute and round you up so that you have sail away from your "normal" course then you have grounds to argue. If for no other reason than who ever did that is being an butt.