Driver's side? Is that left or right?
I'm guessing by inner axle you're referring to the axle shaft that the left side CV axle is bolted to, if that's correct you're the first person I've heard complain about this, it's more common to hear complaints about the right side CVs failing.
Where did you add this 1/2 washer?
As far as I know there should be a snap ring on the inner end, and whilst I've never pulled one, it's the same setup on the right side and those are a pain to get out - have you checked to make sure there is one fitted? Be warned, it's not unusual for the ring to break when removing the right CV axle, and it's fairly easy to fish the pieces out with a magnetic pick up tool, the left side would need a longer tool and might not be so easy.