If its windy enough that you're sticking it frequently, pull them up. When I'm singlehanded I leave them up the entire time. Seems to help going to weather to depower a bit as well, and I don't really notice any difference in height other than I'm not pinching as much.

If its not windy enough to fly a hull down wind, pull them up. But they need to be down going to weather.