Abstract: Roughness-induced transition in hypersonic boundary layers has been a topic of interest for decades, due to its relevance to practical problems. Although the mechanism of this transition process has not been well understood, it is now clear that strong convective instability exists in the wake of the roughness. The motivation of the current paper is to study how the transition is triggered by the convective instability modes. First, the wake of the roughness element in a hypersonic boundary layer was obtained by CFD approach and 2D instability analysis on the obtained wake flow was performed. The results show that, high-growth-rate instability modes of inviscid nature are found at Reynolds numbers appreciably lower than the critical Reynolds number for the appearance of the unstable Tollmien-Schlichting (T-S) modes, and these modes can be either varicose (symmetric) or sinuous (anti-symmetric). Then, the evolution of the instability modes in the wake of the roughness element was numerically simulated, which confirmed the results of the 2D instability analysis and also showed the impact of the non-parallelism. Finally, through direct numerical simulation (DNS), the transition process triggered by these instability modes were investigated. It is confirmed that the convective instability modes play the leading role in boundary-layer transition. The scenario of transition is that, local turbulent spots firstly emerge in the vicinity of the peak of the eigen-function of the instability modes, and then spread into the whole flow field as convecting downstream. For the cases studied in this paper, the impact of the interaction between the varicose and sinuous modes on transition is not obvious, and the latter plays the dominant role in the transition process.