MY 99 GV used to do this when it was still pretty new. After driving in 5th gear for a while on the highway, I would try to go to 4th and it would stick. The dealership replaced the "shifter housing case" at around 16,000 miles and that seemed to help. Good luck.
Dave