Literature Review – Machine Learning Approaches to Network Intrusion Detection

Introduction

Network intrusion detection systems (NIDS) monitor traffic for evidence of compromise and remain a foundational control in enterprise security architecture. The literature conventionally divides detection into two paradigms. Signature-based systems match traffic against curated patterns of known attacks; they generate few false alarms on threats they recognise but are structurally blind to novel or modified attacks. Anomaly-based systems instead model normal behaviour and flag deviations, promising detection of previously unseen intrusions at the cost of higher false-positive rates (Buczak & Guven, 2016). Machine learning (ML) research sits largely within the anomaly and hybrid traditions, and the past decade has produced an extensive body of work applying classical and deep models to network telemetry.

The operational stakes are especially visible in Australia. The Australian Signals Directorate received over 87,000 cybercrime reports in the 2023-24 financial year, an average of one report roughly every six minutes (Australian Signals Directorate [ASD], 2024). Reforms to the Security of Critical Infrastructure Act 2018 have introduced mandatory cyber incident reporting for critical infrastructure operators, and the notifiable data breaches scheme under the Privacy Act 1988 imposes comparable discipline across the wider economy. Detection capability is therefore no longer discretionary for Australian organisations, yet the Australian Cyber Security Centre’s Essential Eight, the country’s de facto security baseline, concentrates on preventative controls, leaving organisations to assemble detection maturity from a research literature whose operational readiness is contested (Australian Cyber Security Centre [ACSC], 2023).

This review synthesises ML-based NIDS research published between 2015 and 2024. It is organised thematically: classical ML and the benchmark dataset tradition, deep learning architectures, adversarial evasion and robustness, and deployment gaps considered with reference to Australian security operations. Figure 1 illustrates this structure. The review argues that the field’s central unresolved problem is not model accuracy but evaluation realism, and it closes by identifying the research gaps that follow from that diagnosis.

Classical Machine Learning and the Benchmark Dataset Tradition

The first wave of ML-based NIDS research applied classical algorithms to labelled traffic datasets. Buczak and Guven (2016), in the most widely cited survey of the period, catalogued applications of support vector machines (SVM), decision trees, random forests, naive Bayes and clustering methods, observing that studies routinely reported detection accuracies above 99 per cent. Random forests emerged as a durable baseline, combining strong performance on tabular flow features with modest training cost, and they continue to match more elaborate architectures in many comparative studies (Vinayakumar et al., 2019).

These results are inseparable from the datasets used to produce them. The field standardised early on the KDD Cup 99 corpus and its de-duplicated refinement NSL-KDD, both derived from traffic simulated in 1998; Ring et al. (2019) note that a large share of published evaluations still relied on this family two decades later, long after its protocols and attack mix had ceased to resemble operational networks. CICIDS2017 responded by generating benign behaviour from profiled users alongside more than a dozen contemporary attack types with labelled bidirectional flows, and it rapidly became the default modern benchmark (Sharafaldin et al., 2018). Yet a forensic re-analysis by Engelen et al. (2021) documented flawed attack generation, mislabelled flows and capture artefacts, concluding that a substantial share of the results reported on the dataset require re-examination.

The pattern emerging from this first theme is metric inflation: near-ceiling accuracy figures that reflect the idiosyncrasies and errors of a small number of synthetic benchmarks rather than genuine detection capability. That concern frames the remainder of the review.

Deep Learning Architectures

A second theme replaced hand-engineered features with representation learning. Vinayakumar et al. (2019) evaluated deep neural networks, including convolutional (CNN) and long short-term memory (LSTM) architectures, against classical baselines across several public datasets, arguing that deep models scale to high-dimensional inputs and adapt to evolving traffic without manual feature engineering. CNNs have been applied to flow statistics and raw packet bytes, while LSTMs exploit the sequential structure of connections to capture staged behaviour such as reconnaissance followed by exploitation. Reported gains over well-tuned ensembles were nonetheless inconsistent across datasets, a finding that recurs throughout the comparative literature.

Autoencoders anchor the unsupervised strand. Mirsky et al. (2018) presented Kitsune, an online ensemble of small autoencoders that learns a compact model of normal traffic and scores deviations by reconstruction error, running incrementally on hardware as constrained as a Raspberry Pi. Because labelled attack data are scarce in operational settings, unsupervised approaches of this kind are arguably better aligned with deployment reality than supervised classifiers trained on benchmark labels.

The costs of the deep learning turn are equally well documented: heavy demand for training data, opaque decision boundaries, and inference budgets that strain line-rate processing on high-throughput links. The steady growth of encrypted traffic further erodes payload-based features, pushing models toward flow metadata and timing behaviour. Deep learning has therefore broadened the design space of NIDS without resolving the evaluation problems inherited from the classical era.

Adversarial Evasion and Robustness

A third theme asks whether learned detectors withstand adaptive adversaries. Goodfellow et al. (2015) demonstrated that small, purposeful perturbations can flip the decisions of otherwise accurate neural networks. The insight transfers to intrusion detection with an important constraint: a perturbed flow must remain syntactically valid and still deliver the attack. Evasion in the problem space is therefore harder than gradient-based manipulation of feature vectors, and early studies that ignored the distinction overstated attacker power.

Apruzzese et al. (2022) reformulated the threat model around realistic capabilities, showing that even attackers with limited knowledge of the detector and no access to its training data can meaningfully degrade ML-based NIDS by manipulating attack traffic within functional bounds. Defences remain immature: adversarial training improves robustness at a measurable cost to accuracy on clean traffic, and no accepted benchmark exists for certifying the robustness of a network detector. For high-consequence environments such as critical infrastructure, that uncertainty is itself a deployment risk.

Deployment Gaps: From Benchmark to Security Operations

False-positive economics

Intrusion detection is a rare-event problem, and headline accuracy conceals its consequences. Arp et al. (2022), auditing thirty top-tier studies, identified recurrent pitfalls including sampling bias, inappropriate baselines and base-rate neglect. Their base-rate point is decisive for NIDS: a detector with a 1 per cent false-positive rate applied to one million benign flows per day produces ten thousand false alerts, overwhelming any plausible triage roster. Alert fatigue then degrades the very human judgement the system is meant to support, so precision at operational base rates, not accuracy on balanced benchmarks, is the binding constraint.

Concept drift

Traffic distributions shift as services, users and attacks evolve, violating the assumption that training and deployment data are identically distributed. Andresini et al. (2021) showed that a static detector’s performance decays measurably as traffic drifts and proposed INSOMNIA, which combines incremental model updating with selective labelling to recover most of the loss at feasible annotation cost. Benchmark evaluations that shuffle a single dataset and report cross-validated accuracy cannot observe this failure mode, which is arguably the dominant one in production networks.

Explainability and the Australian operational context

Security operations centre (SOC) analysts must convert alerts into evidenced escalation decisions, and opaque anomaly scores are difficult to action. In Australia this is a regulatory issue as much as an operational one. Critical infrastructure operators must report significant incidents within 12 hours under the Security of Critical Infrastructure Act 2018, and entities covered by the Privacy Act 1988 must assess and notify eligible data breaches; both obligations presuppose that a SOC can rapidly explain what a detector has found (ASD, 2024). The Essential Eight maturity model, the reference framework for Australian organisations, prioritises preventative and recovery controls such as patching, application control and multi-factor authentication, while network monitoring is addressed separately in ASD guidance; ML-based detection therefore occupies a capability layer that Australian maturity assessments largely presume rather than specify (ACSC, 2023). With many mid-sized Australian organisations outsourcing monitoring to managed providers, the analyst time consumed by false positives carries a direct contractual cost, making explainability and precision purchasing criteria rather than academic preferences.

The Dataset Realism Critique

Australia’s most influential contribution to the field addresses the data problem directly. UNSW-NB15 was generated at the Australian Centre for Cyber Security at UNSW Canberra using a hybrid testbed that blended contemporary normal traffic with nine synthesised attack families, explicitly to retire the KDD-era corpora (Moustafa & Slay, 2015). It marked a genuine advance in protocol diversity and labelling discipline, yet it is now approaching a decade old and remains a testbed product, with attack proportions and background behaviour chosen by its designers rather than observed in an enterprise.

The broader critique generalises this point. Ring et al. (2019) assessed 34 public NIDS datasets against criteria including recency, labelling accuracy and realistic traffic composition, and concluded that no available dataset satisfies them all. The labelling flaws that Engelen et al. (2021) exposed in CICIDS2017 show that even nominal ground truth can be unreliable, while cross-dataset evaluation remains rare, so the extent to which models learn dataset idiosyncrasies rather than attack behaviour is largely unmeasured. Privacy and commercial constraints prevent the release of operational Australian telemetry, which entrenches the problem: the traffic on which detectors would actually run is precisely the traffic on which they are never evaluated.

Synthesis and Research Gaps

Table 1 summarises the principal empirical studies reviewed. As shown in Table 1, the field’s own strongest work converges on a consistent conclusion: benchmark performance is necessary but insufficient evidence of operational value, because the benchmarks are synthetic, the base rates are unrealistic, the adversary is adaptive and the deployment environment drifts.

Table 1: Summary of key studies on machine learning for network intrusion detection.

Read together, the themes describe a discipline moving from accuracy maximisation toward operational credibility, but the movement is incomplete. Five gaps follow directly from this synthesis:

• Longitudinal and cross-dataset evaluation. Detectors are almost never tested across datasets or across time, so generalisation and drift resilience remain unquantified outside isolated studies such as Andresini et al. (2021).
• Cost-sensitive metrics. Evaluation should report precision at realistic base rates and model the analyst-hour cost of false positives, aligning research claims with SOC economics.
• Problem-space robustness benchmarks. Adversarial evaluation needs standardised, functionality-preserving attack suites so robustness claims are comparable across studies.
• Analyst-centred explainability. Explanation methods should be validated with practising analysts against triage and reporting tasks, not assessed solely by proxy metrics.
• Australian evaluation infrastructure. No public dataset reflects contemporary Australian enterprise or critical infrastructure telemetry; privacy-preserving data sharing, potentially brokered through ACSC partnership arrangements, would let detectors be evaluated on the traffic they are intended to protect.

Conclusion

This review has traced machine learning approaches to network intrusion detection across four themes: classical models bound to ageing benchmarks, deep architectures that broaden capability without resolving evaluation weaknesses, adversarial research exposing fragile robustness, and deployment realities of base rates, drift and explainability that separate laboratory performance from operational value. For Australian organisations, whose obligations under critical infrastructure and privacy law assume rapid and evidenced detection, the literature currently offers powerful components rather than deployable assurance. The most valuable contributions of the next research period are therefore unlikely to be new architectures. They are realistic evaluation infrastructure, cost-aware metrics and analyst-centred design, the foundations on which architectural progress could finally be measured.

References

Andresini, G., Pendlebury, F., Pierazzi, F., Loglisci, C., Appice, A., & Cavallaro, L. (2021). INSOMNIA: Towards concept-drift robustness in network intrusion detection. In Proceedings of the 14th ACM Workshop on Artificial Intelligence and Security (pp. 111-122). Association for Computing Machinery.

Apruzzese, G., Andreolini, M., Ferretti, L., Marchetti, M., & Colajanni, M. (2022). Modeling realistic adversarial attacks against network intrusion detection systems. Digital Threats: Research and Practice, 3(3), 1-19.

Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2022). Dos and don’ts of machine learning in computer security. In Proceedings of the 31st USENIX Security Symposium (pp. 3971-3988). USENIX Association.

Australian Cyber Security Centre. (2023). Essential Eight maturity model. Australian Signals Directorate.

Australian Signals Directorate. (2024). Annual cyber threat report 2023-24. Australian Government.

Buczak, A. L., & Guven, E. (2016). A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications Surveys & Tutorials, 18(2), 1153-1176.

Engelen, G., Rimmer, V., & Joosen, W. (2021). Troubleshooting an intrusion detection dataset: The CICIDS2017 case study. In 2021 IEEE Security and Privacy Workshops (pp. 7-12). IEEE.

Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations. ICLR.

Mirsky, Y., Doitshman, T., Elovici, Y., & Shabtai, A. (2018). Kitsune: An ensemble of autoencoders for online network intrusion detection. In Proceedings of the Network and Distributed System Security Symposium. Internet Society.

Moustafa, N., & Slay, J. (2015). UNSW-NB15: A comprehensive data set for network intrusion detection systems. In 2015 Military Communications and Information Systems Conference (MilCIS) (pp. 1-6). IEEE.

Ring, M., Wunderlich, S., Scheuring, D., Landes, D., & Hotho, A. (2019). A survey of network-based intrusion detection data sets. Computers & Security, 86, 147-167.

Sharafaldin, I., Habibi Lashkari, A., & Ghorbani, A. A. (2018). Toward generating a new intrusion detection dataset and intrusion traffic characterization. In Proceedings of the 4th International Conference on Information Systems Security and Privacy (pp. 108-116). SciTePress.

Vinayakumar, R., Alazab, M., Soman, K. P., Poornachandran, P., Al-Nemrat, A., & Venkatraman, S. (2019). Deep learning approach for intelligent intrusion detection system. IEEE Access, 7, 41525-41550.