In complex electromagnetic environments, due to the multipath propagation of signals and the impact of co-channel interference, direction-finding systems often receive coherent signals. The mutual coupling between antenna elements or gains inconsistency will cause the superimposed noise of each channel to become spatial colored noise. Due to the low signal-to-noise ratio (SNR) of signals or short transmission time, it is difficult to obtain sufficient high-quality signal samples. When using array direction finding systems for DOA estimation, it is difficult to achieve accurate DOA estimation under conditions of small samples, overlapping colored noise, and coherent incident signals.

This study aims to address how to solve the array direction-finding problems caused by radiation source coherence, aliased colored noise and small samples, which has become a research hotspot and challenge in the array signal processing field.

From the requirement of DOA estimation of narrowband signals, a DOA estimation method is proposed for small samples, overlapping colored noise, and coherent incident signals by using covariance matrix shrinkage estimation to improve the covariance estimation effect under small sample conditions, then using the covariance difference method to process the shrunk covariance matrix to suppress colored noise and signal coherence, and finally applying the MUSIC algorithm for DOA estimation.

Simulation experiments verify the effectiveness of the proposed method, providing an effective solution for solving DOA estimation problems in complex environments.

The proposed method offers an effective approach to array direction-finding in complex environments.