Don`t sail with him again.
I crewed for a skipper on a 26ft keelboat who would drop the helm & run off to the foredeck to fix a problem.
Needless to say someone else had to run back & grab the tiller or the boat would choose it`s own route, with the spinnaker up this frequently turned into a chinese gybe situation.
If you want to stay on the boat, next time he does it, stop him & ask calmly "Now who`s doing YOUR job ?"

Cheers
Steve