Deep reinforcement learning for autonomous and continual network intrusion detection

With the rapid evolution of cyber threats, traditional Network Intrusion Detection Systems (NIDS) face major challenges in effectively identifying both novel attack and benign traffic patterns. Machine Learning (ML)-based NIDS often struggle to adapt to evolving threats due to insufficient volumes of unseen data and are prone to catastrophic forgetting, where updates to the model lead to the loss of previously acquired knowledge. To address these issues, prior research has explored the use of Generative Adversarial Networks (GANs) to augment data for training on new patterns, and Reinforcement Learning (RL) to support adaptive detection and updating. However, GAN-based approaches typically require manual threat labeling and repeated fine-tuning, while RL-based systems often suffer from long adaptation times. Consequently, both approaches fall short of enabling autonomous continual adaptation to evolving network traffic, including benign behavior and zero-day attacks. To overcome these limitations, we propose GARDIAN, a novel, autonomous, end-to-end NIDS architecture that performs continual learning (CL) over NID classifiers, enabling ongoing fine-tuning for traffic detection and classification. At its core, GARDIAN integrates a deep RL agent that guides the ML-based NID classifier through the CL process by deciding when to adapt and how much retraining data to use from different sources. Supporting this process, the first layer acts as an emerging-pattern detection and labeling stage, where clustering is used to group and label novel traffic patterns. The second layer employs a conditional GAN (CGAN) to generate CL-related data for the detected patterns, helping support balanced retraining without catastrophic forgetting. Extensive experiments on four widely used NIDS datasets, including ablation studies, demonstrate that GARDIAN improves detection accuracy in the presence of evolving and previously unseen traffic patterns and outperforms existing solutions. The source code of our implementation is available on Github https://github.com/hanisami/nids_continual_learning