The way I read rule 18.5 is yes you have to give room. There is no reference to windward or leeward just inside overlap.

This might help or may muddy the water.

http://www.sailingworld.com/from-the-experts/rules/gambling-at-marks-and-obstructions-43458.html

It does look like one that could be be debated if navigable water comes into play, but in your example I'd say yeah, you need to give room for the obstruction.