I've been to a few regattas where the Sailing Instructions require a protesting boat to fly a flag, regardless of size. The issue has never come to a head, but I believe that an appeal would most likely declare the instruction as invalid. In order to be valid, the requirement would have to be present in the Notice Of Race, and would have to be correctly constructed - neither of which has been the case as far as I've seen.

If the requirement is not in the NOR, then the SI's cannot require you to carry equipment that you were not told to bring (a protest flag).

If the instruction is not correctly worded (referencing the number of the rule and stating the change), then it is not valid. I've written SI's that add additional requirements for a protest, and you have to be very careful exactly what you say. I even invalidated one of my own SI's once, when I was judging a protest.

Regards,
Eric