Been there, done that.
I once forgot both plugs and only realized it when the boat started becoming REAL unstable. Luckily a willing speedboat towed us to shore.

Since then my plugs are attached to the rudder pins by means of a small length of string.