Try moving forward on your boat. It is probably more from the wash around your transom rather than your rudders.
Luke