I had to do a repair on one over the summer and couldn't get the collar off without risking significant damage. Ended up drilling a small access hole for the pin to drop in/out.

I rotated it a bit beyond normal when drilling/lining it up so not likely to line up during normal use and then taped over it. Not quite as nice as driving out the pins in the sleeve, but it's worked rather nicely thus far with no issues.