skip to main content
research-article
Public Access

Generalized Sketch Families for Network Traffic Measurement

Published:17 December 2019Publication History
Skip Abstract Section

Abstract

Traffic measurement provides critical information for network management, resource allocation, traffic engineering, and attack detection. Most prior art has been geared towards specific application needs with specific performance objectives. To support diverse requirements with efficient and future-proof implementation, this paper takes a new approach to establish common frameworks, each for a family of traffic measurement solutions that share the same implementation structure, providing a high level of generality, for both size and spread measurements and for all flows. The designs support many options of performance-overhead tradeoff with as few as one memory update per packet and as little space as several bits per flow on average. Such a family-based approach will unify implementation by removing redundancy from different measurement tasks and support reconfigurability in a plug-n-play manner. We demonstrate the connection and difference in the design of these traffic measurement families and perform experimental comparisons on hardware/software platforms to find their tradeoff, which provide practical guidance for which solutions to use under given performance goals.

References

  1. N. Bandi, D. Agrawal, and A. Abbadi. 2007. Fast Algorithms for Heavy Distinct Hitters using Associative Memories . Proc. of IEEE International Conference on Distributed Computing Systems(ICDCS) (June 2007).Google ScholarGoogle Scholar
  2. Z. Bar-yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. 2002. Counting Distinct Elements in a Data Stream . Proc. of RANDOM: Workshop on Randomization and Approximation (2002).Google ScholarGoogle Scholar
  3. R. B. Basat, G. Einziger, R. Friedman, M. C. Luizelli, and E. Waisbard. 2017. Constant Time Updates in Hierarchical Heavy Hitters . Proc. of ACM SIGCOMM (2017).Google ScholarGoogle Scholar
  4. V. Braverman and R. Ostrovsky. 2010. Zero-one frequency laws . in Proc. of STOC (2010).Google ScholarGoogle Scholar
  5. J. Cao, Y. Jin, A. Chen, T. Bu, and Z. Zhang. 2009. Identifying High Cardinality Internet Hosts . Proc. of IEEE INFOCOM (April 2009).Google ScholarGoogle Scholar
  6. M. Charikar, K. Chen, and M. Farach-Colton. 2002. Finding Frequent Items in Data Streams . Proc. of International Colloquium on Automata, Languages, and Programming (ICALP) (July 2002).Google ScholarGoogle ScholarCross RefCross Ref
  7. S. Chen and Y. Tang. 2004. Slowing Down Internet Worms . Proc. of IEEE ICDCS (March 2004).Google ScholarGoogle Scholar
  8. Cisco. Online. Cisco IOS NetFlow . http://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/index.htmlGoogle ScholarGoogle Scholar
  9. S. Cohen and Y. Matias. 2003. Spectral Bloom Filters . Proc. of ACM SIGMOD (June 2003).Google ScholarGoogle Scholar
  10. J. Considine, M. Hadjieleftheriou, F. Li, J. Byers, and G. Kollios. 2009. Robust approximate aggregation in sensor data management systems. ACM Transactions on Database Systems (TODS) , Vol. 34, 1 (2009), 6.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. G. Cormode. 2011. Sketch techniques for approximate query processing. Foundations and Trends in Databases. NOW publishers (2011).Google ScholarGoogle Scholar
  12. G. Cormode and S. Muthukrishnan. 2004. An Improved Data Stream Summary: the Count-Min Sketch and Its Applications . Proc. of LATIN (2004).Google ScholarGoogle Scholar
  13. G. Cormode and S. Muthukrishnan. 2005. Space Efficient Mining of Multigraph Streams . Proc. of ACM PODS (June 2005).Google ScholarGoogle Scholar
  14. E. Demaine, A. Lopez-Ortiz, and J. Ian-Munro. 2002. Frequency Estimation of Internet Pacet Streams with Limited Space . Proc. of Annual European Symposium on Algorithms (ESA) (September 2002).Google ScholarGoogle Scholar
  15. X. Dimitropoulos, P. Hurley, and A. Kind. 2008. Probabilistic Lossy Counting: An Efficient Algorithm for Finding Heavy Hitters . ACM SIGCOMM Computer Communication Review , Vol. 38, 1 (2008), 7--16.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. N. Duffield, C. Lund, and M. Thorup. 2003. Estimating Flow Distributions from Sampled Flow Statistics . Proc. of ACM SIGCOMM (October 2003).Google ScholarGoogle Scholar
  17. M. Durand and P. Flajolet. 2003. Loglog Counting of Large Cardinalities . ESA: European Symposia on Algorithms (2003), 605--617.Google ScholarGoogle Scholar
  18. C. Estan, K. Keys, D. Moore, and G. Varghese. 2004. Building a better netflow . Proc. of ACM SIGCOMM (2004).Google ScholarGoogle Scholar
  19. C. Estan and G. Varghese. 2002. New Directions in Traffic Measurement and Accounting . Proc. of ACM SIGCOMM (August 2002).Google ScholarGoogle Scholar
  20. C. Estan, G. Varghese, and M. Fish. 2006. Bitmap Algorithms for Counting Active Flows on High-Speed Links . IEEE/ACM Trans. on Networking , Vol. 14, 5 (October 2006).Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. P. Flajolet, E. Fusy, O. Gandouet, and F. Meunier. 2007. HyperLogLog: The Analysis of a Near-optimal Cardinality Estimation Algorithm. Proc. of AOFA (2007), 127--146.Google ScholarGoogle Scholar
  22. P. Flajolet and G. N. Martin. 1985. Probabilistic Counting Algorithms for Database Applications. J. Comput. System Sci. , Vol. 31 (September 1985), 182--209. Issue 2.Google ScholarGoogle Scholar
  23. G. Gu, R. Perdisci, J. Zhang, and W. Lee. 2008a. BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-independent Botnet Detection . Proc. of ACM Conference on Security Symposium (2008).Google ScholarGoogle Scholar
  24. G. Gu, J. Zhang, and W. Lee. 2008b. BotSniffer: Detecting Botnet Command and Control Channels in Network Traffic . Proc. of Network and Distributed System Security Symposium (2008).Google ScholarGoogle Scholar
  25. F. Hao, M. Kodialam, and T. V. Lakshman. 2004. ACCEL-RATE: A Faster Mechanism for Memory Efficient Per-flow Traffic Estimation . Proc. of ACM SIGMETRICS/Performance (June 2004).Google ScholarGoogle Scholar
  26. S. Heule, M. Nunkesser, and A. Hall. 2013. HyperLogLog in Practice: Algorithmic Engineering of a State-of-The-Art Cardinality Estimation Algorithm . Proc. of EDBT (2013).Google ScholarGoogle Scholar
  27. Q. Huang, X. Jin, P. P. C. Lee, R. Li, L. Tang, Y. Chen, and G. Zhang. 2017. SketchVisor: Robust Network Measurement for Software Packet Processing . Proc. of ACM SIGCOMM (2017).Google ScholarGoogle Scholar
  28. Q. Huang, P. P. C. Lee, and Y. Bao. 2018. SketchLearn: Relieving User Burdens in Approximate Measurement with Automated Statistical Inference . Proc. of ACM SIGCOMM (August 2018), 576 -- 590.Google ScholarGoogle Scholar
  29. A. Kumar, M. Sung, J. Xu, and J. Wang. 2004. Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Size Distribution . Proc. of ACM SIGMETRICS (June 2004).Google ScholarGoogle Scholar
  30. A. Kumar, J. Xu, J. Wang, O. Spatschek, and L. Li. 2004, A journal version was published in IEEE JSAC, 24(12):2327--2339, December 2006. Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement . Proc. of IEEE INFOCOM (March 2004, A journal version was published in IEEE JSAC, 24(12):2327--2339, December 2006).Google ScholarGoogle Scholar
  31. A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. 2006. Data Streaming Algorithms for Estimating Entropy of Network Traffic . Proc. of SIGMETRICS/Performance (2006).Google ScholarGoogle Scholar
  32. T. Li, S. Chen, and Y. Ling. 2011. Fast and Compact Per-Flow Traffic Measurement through Randomized Counter Sharing . IEEE INFOCOM (2011).Google ScholarGoogle Scholar
  33. Y. Li, R. Miao, C. Kim, and M. Yu. 2016. FlowRadar: A Better NetFlow for Data Centers . in Proc. of USENIX NSDI (2016).Google ScholarGoogle Scholar
  34. Z. Liu, A. Manousis, G. Vorsanger, V. Sekar, and V. Braverman . 2016. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon . Proc. of ACM Sigcomm (2016).Google ScholarGoogle Scholar
  35. Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani. 2008. Counter Braids: A Novel Counter Architecture for Per-Flow Measurement . Proc. of ACM SIGMETRICS (June 2008).Google ScholarGoogle Scholar
  36. Y. Lu and B. Prabhakar. 2009. Robust Counting Via Counter Braids: An Error-Resilient Network Measurement Architecture . Proc. of IEEE INFOCOM (April 2009).Google ScholarGoogle Scholar
  37. P. Mahajan, S. M. Bellovin, S. Floyd, J. Ioannidis, V. Paxson, and S. Shenker. 2002. Controlling High Bandwidth Aggregates in the Network . Computer Communications Review , Vol. 32, 3 (July 2002), 62--73.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J. Mai, C. N. Chuan, A. Sridharan, T. Ye, and H. Zang. 2006. Is sampled data sufficient for anomaly detection? Proc. of ACM IMC (2006).Google ScholarGoogle Scholar
  39. G. Manku and R. Motwani. 2002. Approximate Frequency Counts over Data Streams . Proc. of VLDB (August 2002).Google ScholarGoogle Scholar
  40. D. Moore, C. Shannon, G. M. Voelker, and S. Savage. 2003. Internet Quarantine: Requirements for Containing Self-Propagating Code . Proc. of IEEE INFOCOM (April 2003).Google ScholarGoogle Scholar
  41. D. Moore, G. Voelker, and S. Savage. 2001. Inferring Internet Denial of Service Activity . Proc. of USENIX Security Symposium'2001 (August 2001).Google ScholarGoogle Scholar
  42. M. Moshref, M. Yu, R. Govindan, and A. Vahdat. 2014. DREAM: Dynamic Resource Allocation for Software-defined Measurement . in Proc. of ACM SIGCOMM (2014).Google ScholarGoogle Scholar
  43. UF Networklab. Online. The Source Codes of Generalized Sketch Families . https://github.com/mcynever/GeneralizedSketchFamiliesGoogle ScholarGoogle Scholar
  44. Nvidia. Online. Nvidia cuda c programming guide, version 10.0 . http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.htmlGoogle ScholarGoogle Scholar
  45. OVS. Online. Open vSwitch . https://www.openvswitch.org/Google ScholarGoogle Scholar
  46. K. Park and H. Lee. 2001. On the Effectiveness of Route-Based Packet Filtering for Distributed DoS Attack Prevention in Power-Law Internets . Proc. of ACM SIGCOMM'2001 (August 2001).Google ScholarGoogle Scholar
  47. D. Plonka. 2000. FlowScan: A Network Traffic Flow Reporting and Visualization Tool . Proc. of USENIX LISA (2000).Google ScholarGoogle Scholar
  48. M. Roesch. 1999. Snort -- Lightweight Intrusion Detection for Networks . Proc. of 13th Systems Administration Conference, USENIX (1999).Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. R. Schweller, A. Gupta, E. Parsons, and Y. Chen. 2004. Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams . Proc. of IMC (2004).Google ScholarGoogle Scholar
  50. S. Staniford, V. Paxson, and N. Weaver. 2002. How to 0wn the Internet in Your Spare Time . Proc. of USENIX Security Symposium (August 2002).Google ScholarGoogle Scholar
  51. UCSD. 2015. CAIDA UCSD Anonymized 2015 Internet Traces on Jan. 17 . http://www.caida.org/data/passive/passive_2015_dataset.xml .Google ScholarGoogle Scholar
  52. S. Venkatataman, D. Song, P. Gibbons, and A. Blum. 2005. New Streaming Algorithms for Fast Detection of Superspreaders . Proc. of NDSS (February 2005).Google ScholarGoogle Scholar
  53. H. Wang, D. Zhang, and K. G. Shin. 2002. SYN-dog: Sniffing SYN Flooding Sources . Proc. of 22nd International Conference on Distributed Computing Systems (ICDCS'02) (July 2002).Google ScholarGoogle ScholarCross RefCross Ref
  54. K. Whang, B. T. Vander-Zanden, and H. M. Taylor. 1990. A Linear-time Probabilistic Counting Algorithm for Database Applications . ACM Transactions on Database Systems , Vol. 15, 2 (1990), 208--229.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Wikipedia. Online. Chebyshev inequality . https://en.wikipedia.org/wiki/Chebyshev%27s_inequalityGoogle ScholarGoogle Scholar
  56. Q. Xiao, S. Chen, M. Chen, and Y. Ying. 2015. Hyper-Compact Virtual Estimators for Big Network Data Based on Register Sharing . in Proc. of ACM SIGMETRICS (2015).Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Q. Xiao, S. Chen, Y. Zhou, M. Chen, J. Luo, T. Li, and Y. Ling. 2017. Cardinality Estimation for Elephant Flows: A Compact Solution based on Virtual Register Sharing . IEEE/ACM Transactions on Networking (2017).Google ScholarGoogle Scholar
  58. T. Yang, J. Jiang, P. Liu, Q. Huang, J. Gong, Y. Zhou, R. Miao, X. Li, and S. Uhlig. 2018. Elastic Sketch: Adaptive and Fast Network-wide Measurements . Proc. of ACM SIGCOMM (August 2018).Google ScholarGoogle Scholar
  59. M. Yoon, T. Li, S. Chen, and J. Peir. 2009. Fit a Spread Estimator in Small Memory . Proc. of IEEE INFOCOM (April 2009).Google ScholarGoogle Scholar
  60. M. Yoon, T. Li, S. Chen, and J. Peir. 2011. Fit a Compact Spread Estimator in Small High-Speed Memory . IEEE/ACM Transactions on Networking , Vol. 19, 5 (October 2011), 1253--1264.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. M. Yu, L. Jose, and R. Miao. 2013. Software Defined Traffic Measurement with OpenSketch . Proc. of USENIX Symposium on Networked Systems Design and Implementation (2013).Google ScholarGoogle Scholar
  62. Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund. 2004. Online Identification of Hierarchical Heavy Hitters: Algorithms, Evaluation, and Application . Proc. of ACM SIGCOMM IMC (October 2004).Google ScholarGoogle Scholar
  63. Y. Zhou, Y. Zhou, M. Chen, and S. Chen. 2017. Persistent Spread Measurement for Big Network Data Based on Register Intersection . Proc. of ACM SIGMETRICS (2017).Google ScholarGoogle Scholar

Index Terms

  1. Generalized Sketch Families for Network Traffic Measurement

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!