Believe it or not, that's not that bad.

One way to fix it is to install plastic bushings in the rudder casting. They slip into the pin holes of the casting after you've drilled out the holes to accept them. They have a 1/16" lip to keep them from slipping down into the casting and provide a great bearing surface against the fiberglass and the gudgeons.