Abstract: Study on stability analysis of open cavity which is a device of scramjet for mixing enhancement and flame holding has significant value to academic and engineering application. Large eddy simulation method was used to simulate the scramjet open cavity flow. And the dynamic mode decomposition method (DMD) and the proper orthogonal decomposition(POD) were carried out on the stability analysis for self-sustained oscillation flow. The DMD method can accurately extract the oscillation frequency. And the results are in good agreement with Rossiter model and FFT analysis of pressure fluctuation. The order of leading role of the DMD modes which is corresponding the first three Rossiter modes in the flow is also consistent with the FFT analysis results. The Rossiter Ⅲ mode occupies the most leading role in the self-sustained oscillation flow. At the same time, the predictive ability of the DMD method for over third order Rossiter mode frequencies is better than that of FFT analysis method. As for the extracting low frequencies, DMD method has more advantages over Rossiter model. The DMD modes corresponding to the first six Rossiter modes converge slowly and show as the shear layer separated vortex and the vortex in middle region and downstream region. The separated vortex structure of the first three order unstable modes mainly focuses on the shear layer and the region near the trailing edge. For the POD method, a small number of modes contain most of the energy in the flow field. However, the resolution ratio of the mode frequency extracted by POD is not high enough. The lowest frequency extracted is corresponding with Rossiter three-order mode. Second frequencies exist in the mode, and the coupling between the main and the second frequencies results in a large difference of the mode form. In addition, the stability of the mode could not be decided by POD method compared with DMD method.