Do you have a pin in your router table?  That may help you keep the piece stable as use the pattern bit.  Instead of just resting against the bearing, having 2 points can make it easier.  

--Nathan, TX. Hire the lazy man. He may not do as much work but that's because he will find a better way.