Sonic

There's no need to attack your rudders with a rasp or file unless you only need say a 10mm adjustment.

They are best altered by removing your rudders by unbolting them and then drilling a second hole off centre in the rudder.

To find the spot. hold your rudder at the desired angle next to the rudder housing. IE parallel with the rudder box, in the "down" position as if your boat was in the water. Mark the spot with a marker through the bolt hole that the rudder pivots. [That's probably off centre by 3 - 5 cm]
Drill the rudder[s] at this position and put it all back together again, using the newly drilled holes. Then any minor adjustment can be made using your rasp etc

Let us all know here how you get on.

Good Luck