If the wheel is going side to side it's probably because the cotter pins that locate the wheel hub are drilled too loosely. You should have cotter pin, large washer, aluminum wheel hub, bearings*, aluminum wheel hub, washer, cotter pin.

*The bearings are large needle bearings that are simply cut from a long stick of round delrin plastic. They're pretty darn durable and even if they wore out, they shouldn't cause play side to side.