...wait, just reread it... if I moved the jib up 6 inches it would just make the problem worse
Exactly. Try the opposite: can you somehow LOWER your foot attachment point? Maybe the problem is that your roller is too tall. Perhaps one of the new (thin) continuous rollers would solve the issue.
One different approach would work if your mast is too vertical and the boat is showing lee helm. In this case you could let the mast lean a couple additional degrees aft and solve both problems at once.