I would agree with messing with rudders but since you used the word "prototype" I would ask if you have tried to sail the boat upwind without the rudders down? Is the front beam too far back on the hull? Does the main have too much roach? Are the boards in the right place? Good luck with your project!
Front beam and boards are in the right place, what do you mean for "too much roach" ?
Meaning the center of effort in the sail does not balance with the Center ofd lateral resistance on the boat and so you get weather or lee helm. More roach will give you more weather helm as more sail goes into the top aft portion of the sail. Pin head sails have little roach, the sail you show has a fair amount as it is a fairly "fat head" sail. Fat head sails have a LOT of roach. By adding more roach, you move the COE back.
A boat will exhibit weather / lee helm whet the boat is not balanced with the sails.
As I am sure you are aware this is because the center of effort / blanace of the sails does not balance where the center of lateral resistance of the hull is acting.
This can be for a number of reasons
1, COE of sail in in front or behind CLR of hull or visa versa
How do we fix this?
1.1 Rake the mast fore or aft to move the COE of sail
1.2, Change the shape of the sail by changing the camber position and thus COE. You get this to happen by adding more downhaul, but the sail also flattens. YOu are sailing with enough downhaul to remove most creases in the sail?????
1.3, Changing the shape of the sail by adding or removing cloth to move the COE
1.4, Moving the mast forward or backward on the platform (move beam(s))
1.5, Moving the position of the dagger/centerboards or skegs
1.6, changing the length of the hulls
1.7, Moving the position of the rudders further aft by adding gantrys (it's difficult to move them forward without removing hull!!)
2, Rudders are not correctly trimmed, with the tip in the incorrect fore/aft position
2.1 Adjust them via designed adjustments within the rudder system or get the files / wet and dry out or make the rudders bigger.
I would suggest that if your mast is vertical and you still have issues with weather helm you have the following possible problems
a, Dagger boards are in the wrong place - you state this is not the case
b, Mast (so front beam) is in the wrong place - you state this is not the case
c, Hulls are the wrong length
d, Sail is the wrong shape, too much roach maybe? (as stated above), too full too far back.
e, all of the above!
If you were to sail in "normal trim" and let go of the tiller, how long does it take to go head to in say 10 kts of wind?