I've had similar problems, best thing I came up with was to stick the tip of a screwdriver against the pin before you start releasing the handle, very dificult to do on your own and probably could be solved with a better tool.

You can get counter-sunk rivets