Abstract: In order to realize the phase locking of GW-level high power microwave (HPM) driven by kW-level RF source, a relativistic klystron amplifier (RKA) with four resonant cavities is designed. However, the particle in cell (PIC) simulation shows that high order mode oscillation in the device destroys the phase locking of HPM seriously. And this high order mode excitation is closely related to the coupling between middle cavities. Combining with the structure of RKA, an equivalent circuit model is built to study the process of high order mode excitation (positive feedback process). Simulations show that positive feedback process of high order mode can be controlled by adding RF lossy material to the RKA structure. And this measure increases the threshold current of the high order mode excitation. We have got a phase-locking output microwave of 2.3 GW with an input power of 4 kW in simulation. Experiments were performed on the LTD accelerator, and the input microwave was provided by a solid-state microwave source. Under the conditions of an input microwave of 10 kW, the output power achieved 1.8 GW with a gain of 52.6 dB; and the relative phase difference between input and output RF signals was about ±10° in the locking duration of 90 ns.