Not neccesarily. You would both be responsible for a collision. The last rule is "if the give way vessel does not give way to the stand on vessel, the stand on vessel should give way so as to avoid a collision." You and the other skipper would likely be assigned equal blame. Your insurance would pay for your boat, his would pay for his.

Think about it, is it worth it to get hit by a motor boat
going 20 knots? Are you going to survive such a collision?

Also, I don't know where you are sailing, but if you are crossing shipping lanes certain areas (Puget Sound) has certain rules regarding what you can do in those lanes. Are the fishing boats restricted to a channel by draft? They would probably have right of way in such a situation.