Abstract: Under the driving of explosive-emission cathodes, relativistic gyrotrons frequently suffer from virtual cathode phenomena induced by ultrahigh beam currents (＞300 A), where electron beams readily impact the inner conductor surfaces, accompanied by unintended excitations of cyclotron resonance and backward-wave oscillation (BWO) modes. This study systematically investigates the electromagnetic characteristics of an X-band coaxial gyrotron cavity driven by an intense relativistic electron beam (IREB), combining theoretical analysis with three-dimensional particle-in-cell (PIC) simulations. The results demonstrate that stable IREB transmission and TE₀₁ single-mode operation can be achieved through cavity geometry optimization and electron beam parameter matching. The cavity quality factor (Q_cav) plays a critical role in suppressing parasitic mode competition: TE₂₁-BWO modes are excited when Q_cav＜65, while TE₃₁ cyclotron resonance modes emerge when Q_cav＞90. Stable TE₀₁ single-mode oscillation with an output power of 35 MW (voltage: 300 kV, current: 500 A, transverse-to-longitudinal velocity ratio: 1.2) and efficiency of 34.4% are maintained within the Q_cav range of 65-90. Further studies reveal that the cavity exhibits significant robustness against electron beam velocity spread (Δβ＜25%), providing critical insights for high-power microwave source design.