skip to main content
research-article

Online Adaptive Anomaly Detection for Augmented Network Flows

Published:20 September 2016Publication History
Skip Abstract Section

Abstract

Traditional network anomaly detection involves developing models that rely on packet inspection. However, increasing network speeds and use of encrypted protocols make per-packet inspection unsuited for today’s networks. One method of overcoming this obstacle is aggregating packet header information and performing flow-based analysis where data flow patterns are examined rather than deep packet inspection. Many existing approaches are special purpose limited to detecting specific behavior. Also, the data reduction inherent in identifying anomalous flows hinders alert correlation. In this article, we propose and develop a dynamic anomaly detection approach for augmented network flows. We sketch network state during flow creation, enabling general-purpose threat detection. We describe an efficient flow augmentation approach based on the count-min sketch that provides per-flow-, per-node-, and per-network-level statistics parallel to flow record generation. We design and develop a support vector machine-based adaptive anomaly detection and correlation mechanism, which is capable of aggregating alerts without a priori alert classification and evolving models online. We further develop a lightweight evolving alert aggregation method and combine it with a confidence forwarding mechanism identifying a small percentage predictions for additional processing. We show effectiveness of our methods on both enterprise and backbone traces. Experimental results demonstrate its ability to maintain high accuracy without the need for offline training.

References

  1. 2014. LBNL/ICSI Enterprise Tracing Project. Retrieved from http://www.icir.org/enterprise-tracing/.Google ScholarGoogle Scholar
  2. D. Brauckhoff, X. Dimitropoulos, A. Wagner, and K. Salamatian. 2009. Anomaly extraction in backbone networks using association rules. IEEE/ACM Trans. Netw. 20, 6 (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. N. Cascarano, A. Este, F. Gringoli, F. Risso, and L. Salgarelli. 2009. An experimental evaluation of the computational cost of a DPI traffic classifier. In Proc. of IEEE Global Telecommunications Conference (GLOBECOM). Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. V. Chandola, A. Banerjee, and V. Kumar. 2009. Anomaly detection: A survey. Comput. Surv. 41, 3 (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. K. Cho, K. Mitsuya, and A. Kato. 2000. Traffic data repository at the WIDE project. In Proc. USENIX Annual Technical Conference (ATC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. C. Chung, P. Khatkar, T. Xing, J. Lee, and D. Huang. 2013. NICE: Network intrusion detection and countermeasure selection in virtual network systems. IEEE Trans. Depend. Sec. Comput. 10, 4 (2013). Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. G. Cormode and S. Muthukrishnan. 2005. An improved data stream summary: The count-min sketch and its applications. J. Algor. 55, 1 (2005). Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. H. Debar and A. Wespi. 2001. Aggregation and correlation of intrusion-detection alerts. In Proc. of International Symposium on Recent Advances in Intrusion Detection (RAID). Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. G. Dewaele, K. Fukuda, P. Borgnat, P. Abry, and K. Cho. 2007. Extracting hidden anomalies using sketch and non gaussian multiresolution statistical detection procedure. In Proc. of ACM SIGCOMM Workshop on Large-Scale Attack Defense (LSAD). Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. R. Fontugne and K. Fukuda. 2011. A hough-transform-based anomaly detector with an adaptive time interval. In Proc. of ACM International Symposium on Applied Computing (SAC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. M. Gao, K. Zhang, and J. Lu. 2006b. Efficient packet matching for gigabit network intrusion detection using TCAMs. In Proc. of International Conference on Advanced Information Networking and Applications (AINA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Y. Gao, Z. Li, and Y. Chen. 2006a. A DOS resilient flow-level intrusion detection approach for high-speed networks. In Proc. of IEEE International Conference on Distributed Computing Systems (ICDCS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. A. Hofmann and B. Sick. 2011. Online intrusion alert aggregation with generative data stream modeling. IEEE Trans. Depend. Sec. Comput. 8, 2 (2011). Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. D. Ippoliti and X. Zhou. 2010. An adaptive growing hierarchical self organizing map for network intrusion detection. In Proc. of IEEE International Conference on Computer Communications and Networks (ICCCN).Google ScholarGoogle Scholar
  15. D. Ippoliti and X. Zhou. 2012. A-GHSOM: An adaptive growing hierarchical self organizing map for network anomaly detection. J. Parallel Distrib. Comput. 72, 12 (2012). Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. D. Ippoliti and X. Zhou. 2014. Online adaptive anomaly detection for augmented network flows. In Proc. of IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Z. Ji and B. L. Lu. 2011. A support vector machine classifier with automatic confidence and its application to gender classification. Neurocomputing 74 (2011). Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Z. Ji, W. Y. Yang, S. Wu, and B. L. Lu. 2008. Encoding human knowledge for visual pattern recognition. In Proc. of International Symposium on Neural Networks (ISNN).Google ScholarGoogle Scholar
  19. Y. Kanda, K. Fukuda, and T. Sugawara. 2010. Evaluation of anomaly detection based on sketch and PCA. In Proc. of IEEE Global Telecommunications Conference (GLOBECOM).Google ScholarGoogle Scholar
  20. M. S. Kim, H. J. Kong, S. C. Hong, S. H. Chung, and J. Hong. 2004. A flow-based method for abnormal network traffic detection. In Proc. of IEEE/IFIP Network Operations and Management Symposium (NOMS).Google ScholarGoogle Scholar
  21. H. Lai, S. Cai, H. Huang, J. Xie, and H. Li. 2004. A parallel intrusion detection system for high-speed networks. In Proc. of International Conference Applied Cryptography and Network Security (ACNS).Google ScholarGoogle Scholar
  22. A. Lakhina, M. Crovella, and C. Diot. 2005. Mining anomalies using traffic feature distributions. ACM SIGCOMM Comput. Commun. Rev. 35, 4 (2005). Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, G. Iannaccone, and A. Lakhina. 2006. Detection and identification of network anomalies using sketch subspaces. In Proc. of ACM SIGCOMM Conference on Internet Measurement (IMC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. I. Nischenko and E. M. Jordaan. 2006. Confidence of SVM predictions using a strangeness measure. In Proc. of International Joint Conference on Neural Networks (IJCNN).Google ScholarGoogle Scholar
  25. R. Perdisci, D. Ariu, P. Fogla, G. Giacinto, and W. Lee. 2009. McPAD: A multiple classifier system for accurate payload-based anomaly detection. Comput. Netw. 5, 6 (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. J. Platt. 1998. Fast Training of Support Vector Machines using Sequential Minimal Optimization. B. Advances in Kernel Methods - Support Vector Learning, MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. J. Platt. 1999. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, MIT Press.Google ScholarGoogle Scholar
  28. S. Ruping. 2004. A simple method for estimating conditional probabilities for SVMs. In Technical Report/University Dortmund.Google ScholarGoogle Scholar
  29. B. Scholkopf, J. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson. 1999. Estimating the support of a high-dimensional distribution. In Technical Report, Microsoft Research, Redmond.Google ScholarGoogle Scholar
  30. F. Valeur, V. Giovanni, K. Christopher, and R. Kemmerer. 2004. Comprehensive approach to intrusion detection alert correlation. IEEE Trans. Depend. Sec. Comput. 1, 3 (2004). Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. K. Wang and S. Stolfo. 2004. Anomalous payload-based network intrusion detection. In Proc. of International Symposium on Recent Advances in Intrusion Detection (RAID).Google ScholarGoogle Scholar
  32. K. Y. Whang, V. T. Vander-Zanden, and H. M. Taylor. 1990. A linear-time probabilistic counting algorithm for database applications. ACM Trans. Database Syst. 15 (1990). Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. A. White, S. Krishnan, M. Bailey, F. Monrose, and P. Parros. 2013. Clear and present data: Opaque traffic and its security implications for the future. In Proc. of Network and Distributed System Security Symposium (NDSS).Google ScholarGoogle Scholar
  34. Z. Yu, J. Tsai, and T. Weigert. 2008. An adaptive automatically tuning intrusion detection system. ACM Trans. Auton. Adapt. Syst. 3, 3 (2008). Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Z. Zhang and H. Shen. 2009. M-AID: An adaptive middleware built upon anomaly detectors for intrusion detection and rational response. ACM Trans. Auton. Adapt. Syst. 4, 4 (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. D. Zhao, I. Traore, A. Ghorbani, B. Sayed, S. Saad, and W. Lu. 2012. Peer-to-peer botnet detection based on flow intervals. In Proc. of IFIP International Information Security and Privacy Conference (SEC).Google ScholarGoogle Scholar
  37. Q. Zhao, J. Xu, and A. Kumar. 2006. Detection of super sources and destinations in high-speed networks: Algorithms, analysis and evaluation. IEEE J. Select. Areas. Commun. 24, 10 (2006). Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J. Zheng and B. L. Lu. 2011. A support vector machine classifier with automatic confidence and its application to gender classification. Neurocomputing 74, 11 (2011). Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Online Adaptive Anomaly Detection for Augmented Network Flows

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!