Active and passive control techniques have been devised over the years to mitigate the effect of vibrations on drill-string life with varying degrees of success. Here, it is proposed to design a robust trajectory tracking controller, which ultimately forces the rotary table and the drill-bit to move with the same speed (speed synchronization), hence reducing/eliminating torsional vibrations from the drill pipes. A model of the rotary drilling system, which includes torsional stick-slip, is first developed; then, an integral sliding mode control with time-varying exponent (ISMC-TVE) scheme is developed such that the bit motion tracks that of the rotary table to mitigate the effects of the induced vibrations. The ISMC-TVE is able to control the transient stage of the drill-string system’s response, maintain the system in the sliding state even under abrupt or existing external disturbances, and guarantee asymptotic stability of the rotary drilling system. The Lyapunov stability theorem is used here to analyze the performance of the closed-loop system, and the simulation results showed that the ISMC-TVE law is capable of accurately synchronizing the bit and rotary table speeds.