You pull daggerboards up mainly when it's gusty. If your boat wants to lift a hull without accelerating if you have pulled well on the cunningham it's time to lift the boards.
As Karl says single handed you will have them up earlier than 2up.

Gill