I would take them off if you go in distance. Around the block won't hurt. But long trips will beat them to death no matter what the diameter of the pins. Wear is wear, they are still rubbing together and bouncing up and down. It makes a difference over time, not just a few times.

Doug