Well, file or scrape your edges a bit more. It has always been the trailing edges of the rudders in my experience. Make sure you get the tips too. I wouldn't go past the centerline of the tip though. You don't want to make them chip easy.

It took me three sessions with a rasp to do my first set because I didn't want to take off too much. After that I usually got it in one session.

cheers
Bill