... What I mean is if I checked it would I even notice?
Maybe not. I thought mine was pretty much right on. After installing the lift and adjusting the camber it wandered around a bit, so I took it in for an alignment. Turns out I had the camber right on, but the toe was way off. They fixed that and now the thing rolls down the road like a bumpy dream.
These things have such a short wheel base that any deviation from spec tolerances can make them a little squirrely.
I'd go have it aligned. Even if it doesn't solve your problem the $60 in alignment could save you $200 in tires.