Bending the material should actually harden it and make it stronger and stiffer.

A simple experiment is to bend a wire coat hanger and try to straighten it. It's very hard to do because the material at the bend has been work hardened.

I believe fatigue only occurs after it's been done a heap of times, ie. in the thousands of cycles.

Well, this is the case for steel, fairly sure it's the same for aluminium alloys.