Abstract
Traditional network anomaly detection involves developing models that rely on packet inspection. However, increasing network speeds and use of encrypted protocols make per-packet inspection unsuited for today’s networks. One method of overcoming this obstacle is aggregating packet header information and performing flow-based analysis where data flow patterns are examined rather than deep packet inspection. Many existing approaches are special purpose limited to detecting specific behavior. Also, the data reduction inherent in identifying anomalous flows hinders alert correlation. In this article, we propose and develop a dynamic anomaly detection approach for augmented network flows. We sketch network state during flow creation, enabling general-purpose threat detection. We describe an efficient flow augmentation approach based on the count-min sketch that provides per-flow-, per-node-, and per-network-level statistics parallel to flow record generation. We design and develop a support vector machine-based adaptive anomaly detection and correlation mechanism, which is capable of aggregating alerts without a priori alert classification and evolving models online. We further develop a lightweight evolving alert aggregation method and combine it with a confidence forwarding mechanism identifying a small percentage predictions for additional processing. We show effectiveness of our methods on both enterprise and backbone traces. Experimental results demonstrate its ability to maintain high accuracy without the need for offline training.
- 2014. LBNL/ICSI Enterprise Tracing Project. Retrieved from http://www.icir.org/enterprise-tracing/.Google Scholar
- D. Brauckhoff, X. Dimitropoulos, A. Wagner, and K. Salamatian. 2009. Anomaly extraction in backbone networks using association rules. IEEE/ACM Trans. Netw. 20, 6 (2009). Google Scholar
Digital Library
- N. Cascarano, A. Este, F. Gringoli, F. Risso, and L. Salgarelli. 2009. An experimental evaluation of the computational cost of a DPI traffic classifier. In Proc. of IEEE Global Telecommunications Conference (GLOBECOM). Google Scholar
Digital Library
- V. Chandola, A. Banerjee, and V. Kumar. 2009. Anomaly detection: A survey. Comput. Surv. 41, 3 (2009). Google Scholar
Digital Library
- K. Cho, K. Mitsuya, and A. Kato. 2000. Traffic data repository at the WIDE project. In Proc. USENIX Annual Technical Conference (ATC). Google Scholar
Digital Library
- C. Chung, P. Khatkar, T. Xing, J. Lee, and D. Huang. 2013. NICE: Network intrusion detection and countermeasure selection in virtual network systems. IEEE Trans. Depend. Sec. Comput. 10, 4 (2013). Google Scholar
Digital Library
- G. Cormode and S. Muthukrishnan. 2005. An improved data stream summary: The count-min sketch and its applications. J. Algor. 55, 1 (2005). Google Scholar
Digital Library
- H. Debar and A. Wespi. 2001. Aggregation and correlation of intrusion-detection alerts. In Proc. of International Symposium on Recent Advances in Intrusion Detection (RAID). Google Scholar
Digital Library
- G. Dewaele, K. Fukuda, P. Borgnat, P. Abry, and K. Cho. 2007. Extracting hidden anomalies using sketch and non gaussian multiresolution statistical detection procedure. In Proc. of ACM SIGCOMM Workshop on Large-Scale Attack Defense (LSAD). Google Scholar
Digital Library
- R. Fontugne and K. Fukuda. 2011. A hough-transform-based anomaly detector with an adaptive time interval. In Proc. of ACM International Symposium on Applied Computing (SAC). Google Scholar
Digital Library
- M. Gao, K. Zhang, and J. Lu. 2006b. Efficient packet matching for gigabit network intrusion detection using TCAMs. In Proc. of International Conference on Advanced Information Networking and Applications (AINA). Google Scholar
Digital Library
- Y. Gao, Z. Li, and Y. Chen. 2006a. A DOS resilient flow-level intrusion detection approach for high-speed networks. In Proc. of IEEE International Conference on Distributed Computing Systems (ICDCS). Google Scholar
Digital Library
- A. Hofmann and B. Sick. 2011. Online intrusion alert aggregation with generative data stream modeling. IEEE Trans. Depend. Sec. Comput. 8, 2 (2011). Google Scholar
Digital Library
- D. Ippoliti and X. Zhou. 2010. An adaptive growing hierarchical self organizing map for network intrusion detection. In Proc. of IEEE International Conference on Computer Communications and Networks (ICCCN).Google Scholar
- D. Ippoliti and X. Zhou. 2012. A-GHSOM: An adaptive growing hierarchical self organizing map for network anomaly detection. J. Parallel Distrib. Comput. 72, 12 (2012). Google Scholar
Digital Library
- D. Ippoliti and X. Zhou. 2014. Online adaptive anomaly detection for augmented network flows. In Proc. of IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS). Google Scholar
Digital Library
- Z. Ji and B. L. Lu. 2011. A support vector machine classifier with automatic confidence and its application to gender classification. Neurocomputing 74 (2011). Google Scholar
Digital Library
- Z. Ji, W. Y. Yang, S. Wu, and B. L. Lu. 2008. Encoding human knowledge for visual pattern recognition. In Proc. of International Symposium on Neural Networks (ISNN).Google Scholar
- Y. Kanda, K. Fukuda, and T. Sugawara. 2010. Evaluation of anomaly detection based on sketch and PCA. In Proc. of IEEE Global Telecommunications Conference (GLOBECOM).Google Scholar
- M. S. Kim, H. J. Kong, S. C. Hong, S. H. Chung, and J. Hong. 2004. A flow-based method for abnormal network traffic detection. In Proc. of IEEE/IFIP Network Operations and Management Symposium (NOMS).Google Scholar
- H. Lai, S. Cai, H. Huang, J. Xie, and H. Li. 2004. A parallel intrusion detection system for high-speed networks. In Proc. of International Conference Applied Cryptography and Network Security (ACNS).Google Scholar
- A. Lakhina, M. Crovella, and C. Diot. 2005. Mining anomalies using traffic feature distributions. ACM SIGCOMM Comput. Commun. Rev. 35, 4 (2005). Google Scholar
Digital Library
- X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, G. Iannaccone, and A. Lakhina. 2006. Detection and identification of network anomalies using sketch subspaces. In Proc. of ACM SIGCOMM Conference on Internet Measurement (IMC). Google Scholar
Digital Library
- I. Nischenko and E. M. Jordaan. 2006. Confidence of SVM predictions using a strangeness measure. In Proc. of International Joint Conference on Neural Networks (IJCNN).Google Scholar
- R. Perdisci, D. Ariu, P. Fogla, G. Giacinto, and W. Lee. 2009. McPAD: A multiple classifier system for accurate payload-based anomaly detection. Comput. Netw. 5, 6 (2009). Google Scholar
Digital Library
- J. Platt. 1998. Fast Training of Support Vector Machines using Sequential Minimal Optimization. B. Advances in Kernel Methods - Support Vector Learning, MIT Press. Google Scholar
Digital Library
- J. Platt. 1999. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, MIT Press.Google Scholar
- S. Ruping. 2004. A simple method for estimating conditional probabilities for SVMs. In Technical Report/University Dortmund.Google Scholar
- B. Scholkopf, J. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. 1999. Estimating the support of a high-dimensional distribution. In Technical Report, Microsoft Research, Redmond.Google Scholar
- F. Valeur, V. Giovanni, K. Christopher, and R. Kemmerer. 2004. Comprehensive approach to intrusion detection alert correlation. IEEE Trans. Depend. Sec. Comput. 1, 3 (2004). Google Scholar
Digital Library
- K. Wang and S. Stolfo. 2004. Anomalous payload-based network intrusion detection. In Proc. of International Symposium on Recent Advances in Intrusion Detection (RAID).Google Scholar
- K. Y. Whang, V. T. Vander-Zanden, and H. M. Taylor. 1990. A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15 (1990). Google Scholar
Digital Library
- A. White, S. Krishnan, M. Bailey, F. Monrose, and P. Parros. 2013. Clear and present data: Opaque traffic and its security implications for the future. In Proc. of Network and Distributed System Security Symposium (NDSS).Google Scholar
- Z. Yu, J. Tsai, and T. Weigert. 2008. An adaptive automatically tuning intrusion detection system. ACM Trans. Auton. Adapt. Syst. 3, 3 (2008). Google Scholar
Digital Library
- Z. Zhang and H. Shen. 2009. M-AID: An adaptive middleware built upon anomaly detectors for intrusion detection and rational response. ACM Trans. Auton. Adapt. Syst. 4, 4 (2009). Google Scholar
Digital Library
- D. Zhao, I. Traore, A. Ghorbani, B. Sayed, S. Saad, and W. Lu. 2012. Peer-to-peer botnet detection based on flow intervals. In Proc. of IFIP International Information Security and Privacy Conference (SEC).Google Scholar
- Q. Zhao, J. Xu, and A. Kumar. 2006. Detection of super sources and destinations in high-speed networks: Algorithms, analysis and evaluation. IEEE J. Select. Areas. Commun. 24, 10 (2006). Google Scholar
Digital Library
- J. Zheng and B. L. Lu. 2011. A support vector machine classifier with automatic confidence and its application to gender classification. Neurocomputing 74, 11 (2011). Google Scholar
Digital Library
Index Terms
Online Adaptive Anomaly Detection for Augmented Network Flows
Recommendations
Flow-Based Anomaly Detection Using Neural Network Optimized with GSA Algorithm
ICDCSW '13: Proceedings of the 2013 IEEE 33rd International Conference on Distributed Computing Systems WorkshopsReliable high-speed networks are essential to provide quality services to ever growing Internet applications. A Network Intrusion Detection System (NIDS) is an important tool to protect computer networks from attacks. Traditional packet-based NIDSs are ...
Online Adaptive Anomaly Detection for Augmented Network Flows
MASCOTS '14: Proceedings of the 2014 IEEE 22nd International Symposium on Modelling, Analysis & Simulation of Computer and Telecommunication SystemsTraditional network anomaly detection involves developing models that rely on packet inspection. Increasing network speeds and use of encrypted protocols make per-packet inspection unsuited for today's networks. One method of overcoming this obstacle is ...
Anomaly Detection Using LibSVM Training Tools
ISA '08: Proceedings of the 2008 International Conference on Information Security and Assurance (isa 2008)Intrusion detection is the means to identify the intrusive behaviors and provides useful information to intruded systems to respond fast and to avoid or reduce damages. In recent years, learning machine technology is often used as a detection method in ...






Comments