A Bayesian Neural Network for Robust Automatic Modulation Classification: Mitigating Adversarial Amplification

In recent years, the rapid advancement of wireless communication technologies, particularly in the development of sixth-generation networks, brought about challenges in spectrum efficiency, security, and reliability. machine learning (ML)-based automatic modulation classification (AMC) plays a critical role in addressing these challenges by enabling efficient signal classification in dynamic environments. However, such systems remain vulnerable to adversarial attacks, which can induce ML-based systems into making mistakes and, by doing so, compromise applications that rely on them. Accordingly, in this study, we propose a robust AMC framework based on Bayesian neural networks (BNNs) to mitigate the impact of adversarial attacks. Our approach uses a regularization term on the weight variance of the BNN to reduce the likelihood of extreme weight values, thereby enhancing model stability in adversarial settings. We also incorporate the Sinh-Arcsinh Gaussian distribution as a flexible prior to control skewness and tail behavior, thus improving the tradeoff between robustness and accuracy. Experimental evaluations against common glass-box adversarial attacks, such as fast gradient sign method, projected gradient descent (PGD), and automatic PGD (Auto-PGD), demonstrate that our proposed model outperforms conventional AMC models, achieving greater resilience in low perturbation-to-noise ratio conditions. Taken together, these findings highlight the potential of Bayesian methods in developing more secure and reliable intelligent wireless communication systems.