It sounds like you are a reacher -- both downwind and upwind. You like to go fast. But speed will not get you to the mark fastest if you are sailing a longer course to get there. (It's just more fun to go fast.)
Reminds me of the ferry competition here at Put-in-Bay. The Jet Express is a big catamaran people ferry that goes up to 50 mph and brings people back and forth from Port Clinton to the island. They advertise that they are the "Fastest way to Put-in-Bay."
Interesting, and, in a way, misleading, advertising. They may be the fastest, but they are not the fastest way to get to the Bay.
The other ferry is the Miller Boat Line from Catawba Point to Put-in-Bay. It is a big, heavy, barge-type car ferry that goes much slower than the Jet Express, but it gets you to Put-in-Bay much faster because, even though it goes much slower, it is traveling a shorter distance.
So the fastest way to Put-in-Bay is on the slower boat.
Anyway, what you need to do is to follow and emulate the other boats that are going deeper than you downwind and the boats that are going higher than you upwind.