There has been talk of a rating system for race officials, but nobody's been able to figure out a way to not make it a popularity contest. Race Officials are asked to make difficult decisions that may not be popular at the time they are made. Have you ever known the person who lost a protest to be happy with the decision?
It seems simple to me. Both sides in a protest should be required to report on whether they think the ruling was fair/unbiased. A secret survey that is filled out after the ruling should be fine. The expected objection to this sort of idea is that everybody who wins a ruling will say the judge was "very fair" while everybody who looses will say "very unfair." Personally, I don't think that would be the case for every ruling in every situation. I suspect that some judges will be able to make a ruling and explain it convincingly enough that the looser will rate the judge higher. The upshot is that not every judge will have a 50% rating and a judge with a 55 or 60% rating would be highly sought after.
People who were not stakeholders in the ruling would not be allowed to rate the judge, that will mitigate the popularity contest aspects.
