Been there, done that. I once forgot both plugs and only realized it when the boat started becoming REAL unstable. Luckily a willing speedboat towed us to shore.
Since then my plugs are attached to the rudder pins by means of a small length of string.