It is complicated........
First consider letting out the kite and nothing else...
It flogs and you loose power but the whole rig becomes unstable. If you decelerate, the kite then starts to power up again (maybe).....
What happens if you oversheet the kite?
You loose power and so you may reduce heeling moment and thus prevent a capsise or pitchpole, but you then have tight sails and may be in trouble as the rig loads up as you might simple blow over sideways as the apparent wind moves aft as you slow down.
IMO, you need a mixture of the two.
Now, consider what happens when you free some mainsheet and do not change the kite. Letting some mainsail out you reduce heeling moment and drive (remember if you decelerate you may bring the sail back into the sweet spot and then power up again). BUT also what is happening at the top of the mast?.... Less sheet load means the mast tip moves forward (and maybe away from you). BUT the distance between the head and the tack of the kite has reduced and so the shape of the who kite has changed. This MAY also flatten the entry of the kite (most kites flatten as you ease tension in the luff).
Time on the water is key, but initially I would suggest dumping an arm full the kite until you get batter and can then combine oversheeting the kite with dumping small amounts of mainsail and then re-shet kite and mainsail and you will then also learn when you cannot over sheet the kite. It takes time to work it out.