I've been in that situation so many times, and seen others having the same problem, my thinking is to avoid the committee end of the line all together.
This is often a viable alternative. I race in a fleet where many skippers try to win the committee boat end of the line. At one point, I simply gave up on playing bumper-boats and instead tried to get clear-air starts down the line. My starts (and finishes) improved markedly.
Regards,
Eric