ANOMALY DETECTION IN ENCRYPTED NETWORK TRAFFIC USING DEEP LEARNING

Abstract

The increasing dominance of encrypted traffic in modern network communications poses significant challenges to cybersecurity monitoring, especially for traditional intrusion detection systems that rely on packet content inspection. This study addresses the problem of anomaly detection in encrypted traffic using deep learning approaches that analyze metadata without requiring decryption. A comprehensive experimental comparison of three architectures - Autoencoder, CNN+LSTM, and ET SSL (a contrastive self-supervised learning model) - was performed using three publicly available datasets: CIC-Darknet2020, UNSW-NB15, and QUIC-TLS, each representing diverse encrypted protocols and attack types. All datasets were preprocessed into flow-based formats with 75 standardized numerical features. The models were evaluated based on classification accuracy, F1 score, and false positive rate (FPR). The ET SSL model demonstrated the most consistent and superior performance, achieving up to 96.8% accuracy and 0.961 F1 score, with an FPR as low as 1.2%. CNN+LSTM achieved slightly lower but still competitive results, while the Autoencoder model exhibited limitations in adapting to high-level traffic obfuscation, especially in QUIC-based flows. Additionally, a hyperparameter sensitivity analysis was conducted to explore the influence of learning rate, time window size, and dropout regularization. The findings confirmed the critical role of adaptive configuration in optimizing model performance for specific deployment environments. For instance, lowering the learning rate improved accuracy but increased training time, while extending the temporal window improved F1 at the cost of computational overhead.

The empirical results substantiate the practical applicability of deep learning models for encrypted traffic monitoring without decryption. In particular, the ET SSL architecture stands out as a promising candidate for deployment in real-time threat detection systems due to its robustness, high generalization capability, and low false positive rate. Furthermore, its reliance on self-supervised learning allows for effective operation in scenarios with limited or no labeled data, making it especially suitable for detecting zero-day attacks. Future research directions include expanding the diversity of training datasets to reflect evolving encryption standards (e.g., Encrypted SNI, DoQ), integrating detection models into scalable, low-latency IDS/IPS environments, applying explainable AI (XAI) methods to increase trust and interpretability, and developing adversarially robust models. The presented findings serve as a foundation for the development of next-generation, adaptive, and context-aware cyber threat monitoring systems.