If you were on starboard tack and they on port then you have the right of way no matter what --- unless you are within 2 boat lengths of the mark at which point I think you have to give room to the port boat.
Absolutly positively not!
Rule 18 covers buoy room to which I think you refer.
18.1 Rule 18 does not apply:
18.1(b) while the boats are on opposite tacks either on a beat to windward or when the proper course for one of them (but not both) to round or pass the mark or obsturction is to tack.
ie: Anybody fool enough to rock up to the top mark on port tack expecting room better have their insurance paid up!

Tiger Mike