As an experiment try facing your boat directly into wind on the beach without any main sheet on, now tighten the downhaul and see what happens, you maybe surprised. Mine becomes like a cat on a hot roof, bouncing around all over the place.
Any idea why this is the case? Pulling the downhaul should flatten the sail, reducing the camber.
Because when the downhaul is completely off, there is no camber in the sail. The sail material is allowed to relax and align itself to the wind, especially along the leading edge. Once you pull on the downhaul, you are forcing shape into the sail and so the sail can never actually align itself to the wind, therefore it can never go fully into "neutral".
You crank on the downhaul to reduce sail power while underway, but whenever you want to park, you should let off the downhaul completely (so there is no tension along the leading edge of the sail).