Abstract: Spinning vehicles achieve gyroscopic effects by rotating around their axis. However, transition significantly influences the stability of flight. In this work, a study of instability on the unstable modes in boundary layers over a rotating cone in hypersonic flows was presented with linear stability theory (LST). To quantify the effect of Mach number, Mach numbers 5~10 were considered, and a higher rotational velocity was considered for analysis of centrifugal mode. The influence of Mach number on different stability modes varies, which was confirmed with LST analysis. Modified Mack mode, cross-flow mode and centrifugal mode shift towards low frequency, low streamwise and circumferential wavenumbers. Also, maximum amplification rate of modified Mack mode, cross-flow mode and centrifugal mode decrease with Mach number. Meanwhile, oblique wave instinct of second mode is uncovered, and wave angle of second mode decreases with Mach number.