Abstract
Traffic measurement provides critical information for network management, resource allocation, traffic engineering, and attack detection. Most prior art has been geared towards specific application needs with specific performance objectives. To support diverse requirements with efficient and future-proof implementation, this paper takes a new approach to establish common frameworks, each for a family of traffic measurement solutions that share the same implementation structure, providing a high level of generality, for both size and spread measurements and for all flows. The designs support many options of performance-overhead tradeoff with as few as one memory update per packet and as little space as several bits per flow on average. Such a family-based approach will unify implementation by removing redundancy from different measurement tasks and support reconfigurability in a plug-n-play manner. We demonstrate the connection and difference in the design of these traffic measurement families and perform experimental comparisons on hardware/software platforms to find their tradeoff, which provide practical guidance for which solutions to use under given performance goals.
- N. Bandi, D. Agrawal, and A. Abbadi. 2007. Fast Algorithms for Heavy Distinct Hitters using Associative Memories . Proc. of IEEE International Conference on Distributed Computing Systems(ICDCS) (June 2007).Google Scholar
- Z. Bar-yossef, T. S. Jayram, R. Kumar, D. Sivakumar, and L. Trevisan. 2002. Counting Distinct Elements in a Data Stream . Proc. of RANDOM: Workshop on Randomization and Approximation (2002).Google Scholar
- R. B. Basat, G. Einziger, R. Friedman, M. C. Luizelli, and E. Waisbard. 2017. Constant Time Updates in Hierarchical Heavy Hitters . Proc. of ACM SIGCOMM (2017).Google Scholar
- V. Braverman and R. Ostrovsky. 2010. Zero-one frequency laws . in Proc. of STOC (2010).Google Scholar
- J. Cao, Y. Jin, A. Chen, T. Bu, and Z. Zhang. 2009. Identifying High Cardinality Internet Hosts . Proc. of IEEE INFOCOM (April 2009).Google Scholar
- M. Charikar, K. Chen, and M. Farach-Colton. 2002. Finding Frequent Items in Data Streams . Proc. of International Colloquium on Automata, Languages, and Programming (ICALP) (July 2002).Google Scholar
Cross Ref
- S. Chen and Y. Tang. 2004. Slowing Down Internet Worms . Proc. of IEEE ICDCS (March 2004).Google Scholar
- Cisco. Online. Cisco IOS NetFlow . http://www.cisco.com/c/en/us/products/ios-nx-os-software/ios-netflow/index.htmlGoogle Scholar
- S. Cohen and Y. Matias. 2003. Spectral Bloom Filters . Proc. of ACM SIGMOD (June 2003).Google Scholar
- J. Considine, M. Hadjieleftheriou, F. Li, J. Byers, and G. Kollios. 2009. Robust approximate aggregation in sensor data management systems. ACM Transactions on Database Systems (TODS) , Vol. 34, 1 (2009), 6.Google Scholar
Digital Library
- G. Cormode. 2011. Sketch techniques for approximate query processing. Foundations and Trends in Databases. NOW publishers (2011).Google Scholar
- G. Cormode and S. Muthukrishnan. 2004. An Improved Data Stream Summary: the Count-Min Sketch and Its Applications . Proc. of LATIN (2004).Google Scholar
- G. Cormode and S. Muthukrishnan. 2005. Space Efficient Mining of Multigraph Streams . Proc. of ACM PODS (June 2005).Google Scholar
- E. Demaine, A. Lopez-Ortiz, and J. Ian-Munro. 2002. Frequency Estimation of Internet Pacet Streams with Limited Space . Proc. of Annual European Symposium on Algorithms (ESA) (September 2002).Google Scholar
- X. Dimitropoulos, P. Hurley, and A. Kind. 2008. Probabilistic Lossy Counting: An Efficient Algorithm for Finding Heavy Hitters . ACM SIGCOMM Computer Communication Review , Vol. 38, 1 (2008), 7--16.Google Scholar
Digital Library
- N. Duffield, C. Lund, and M. Thorup. 2003. Estimating Flow Distributions from Sampled Flow Statistics . Proc. of ACM SIGCOMM (October 2003).Google Scholar
- M. Durand and P. Flajolet. 2003. Loglog Counting of Large Cardinalities . ESA: European Symposia on Algorithms (2003), 605--617.Google Scholar
- C. Estan, K. Keys, D. Moore, and G. Varghese. 2004. Building a better netflow . Proc. of ACM SIGCOMM (2004).Google Scholar
- C. Estan and G. Varghese. 2002. New Directions in Traffic Measurement and Accounting . Proc. of ACM SIGCOMM (August 2002).Google Scholar
- C. Estan, G. Varghese, and M. Fish. 2006. Bitmap Algorithms for Counting Active Flows on High-Speed Links . IEEE/ACM Trans. on Networking , Vol. 14, 5 (October 2006).Google Scholar
Digital Library
- P. Flajolet, E. Fusy, O. Gandouet, and F. Meunier. 2007. HyperLogLog: The Analysis of a Near-optimal Cardinality Estimation Algorithm. Proc. of AOFA (2007), 127--146.Google Scholar
- P. Flajolet and G. N. Martin. 1985. Probabilistic Counting Algorithms for Database Applications. J. Comput. System Sci. , Vol. 31 (September 1985), 182--209. Issue 2.Google Scholar
- G. Gu, R. Perdisci, J. Zhang, and W. Lee. 2008a. BotMiner: Clustering Analysis of Network Traffic for Protocol- and Structure-independent Botnet Detection . Proc. of ACM Conference on Security Symposium (2008).Google Scholar
- G. Gu, J. Zhang, and W. Lee. 2008b. BotSniffer: Detecting Botnet Command and Control Channels in Network Traffic . Proc. of Network and Distributed System Security Symposium (2008).Google Scholar
- F. Hao, M. Kodialam, and T. V. Lakshman. 2004. ACCEL-RATE: A Faster Mechanism for Memory Efficient Per-flow Traffic Estimation . Proc. of ACM SIGMETRICS/Performance (June 2004).Google Scholar
- S. Heule, M. Nunkesser, and A. Hall. 2013. HyperLogLog in Practice: Algorithmic Engineering of a State-of-The-Art Cardinality Estimation Algorithm . Proc. of EDBT (2013).Google Scholar
- Q. Huang, X. Jin, P. P. C. Lee, R. Li, L. Tang, Y. Chen, and G. Zhang. 2017. SketchVisor: Robust Network Measurement for Software Packet Processing . Proc. of ACM SIGCOMM (2017).Google Scholar
- Q. Huang, P. P. C. Lee, and Y. Bao. 2018. SketchLearn: Relieving User Burdens in Approximate Measurement with Automated Statistical Inference . Proc. of ACM SIGCOMM (August 2018), 576 -- 590.Google Scholar
- A. Kumar, M. Sung, J. Xu, and J. Wang. 2004. Data Streaming Algorithms for Efficient and Accurate Estimation of Flow Size Distribution . Proc. of ACM SIGMETRICS (June 2004).Google Scholar
- A. Kumar, J. Xu, J. Wang, O. Spatschek, and L. Li. 2004, A journal version was published in IEEE JSAC, 24(12):2327--2339, December 2006. Space-Code Bloom Filter for Efficient Per-Flow Traffic Measurement . Proc. of IEEE INFOCOM (March 2004, A journal version was published in IEEE JSAC, 24(12):2327--2339, December 2006).Google Scholar
- A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang. 2006. Data Streaming Algorithms for Estimating Entropy of Network Traffic . Proc. of SIGMETRICS/Performance (2006).Google Scholar
- T. Li, S. Chen, and Y. Ling. 2011. Fast and Compact Per-Flow Traffic Measurement through Randomized Counter Sharing . IEEE INFOCOM (2011).Google Scholar
- Y. Li, R. Miao, C. Kim, and M. Yu. 2016. FlowRadar: A Better NetFlow for Data Centers . in Proc. of USENIX NSDI (2016).Google Scholar
- Z. Liu, A. Manousis, G. Vorsanger, V. Sekar, and V. Braverman . 2016. One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon . Proc. of ACM Sigcomm (2016).Google Scholar
- Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani. 2008. Counter Braids: A Novel Counter Architecture for Per-Flow Measurement . Proc. of ACM SIGMETRICS (June 2008).Google Scholar
- Y. Lu and B. Prabhakar. 2009. Robust Counting Via Counter Braids: An Error-Resilient Network Measurement Architecture . Proc. of IEEE INFOCOM (April 2009).Google Scholar
- P. Mahajan, S. M. Bellovin, S. Floyd, J. Ioannidis, V. Paxson, and S. Shenker. 2002. Controlling High Bandwidth Aggregates in the Network . Computer Communications Review , Vol. 32, 3 (July 2002), 62--73.Google Scholar
Digital Library
- J. Mai, C. N. Chuan, A. Sridharan, T. Ye, and H. Zang. 2006. Is sampled data sufficient for anomaly detection? Proc. of ACM IMC (2006).Google Scholar
- G. Manku and R. Motwani. 2002. Approximate Frequency Counts over Data Streams . Proc. of VLDB (August 2002).Google Scholar
- D. Moore, C. Shannon, G. M. Voelker, and S. Savage. 2003. Internet Quarantine: Requirements for Containing Self-Propagating Code . Proc. of IEEE INFOCOM (April 2003).Google Scholar
- D. Moore, G. Voelker, and S. Savage. 2001. Inferring Internet Denial of Service Activity . Proc. of USENIX Security Symposium'2001 (August 2001).Google Scholar
- M. Moshref, M. Yu, R. Govindan, and A. Vahdat. 2014. DREAM: Dynamic Resource Allocation for Software-defined Measurement . in Proc. of ACM SIGCOMM (2014).Google Scholar
- UF Networklab. Online. The Source Codes of Generalized Sketch Families . https://github.com/mcynever/GeneralizedSketchFamiliesGoogle Scholar
- Nvidia. Online. Nvidia cuda c programming guide, version 10.0 . http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.htmlGoogle Scholar
- OVS. Online. Open vSwitch . https://www.openvswitch.org/Google Scholar
- K. Park and H. Lee. 2001. On the Effectiveness of Route-Based Packet Filtering for Distributed DoS Attack Prevention in Power-Law Internets . Proc. of ACM SIGCOMM'2001 (August 2001).Google Scholar
- D. Plonka. 2000. FlowScan: A Network Traffic Flow Reporting and Visualization Tool . Proc. of USENIX LISA (2000).Google Scholar
- M. Roesch. 1999. Snort -- Lightweight Intrusion Detection for Networks . Proc. of 13th Systems Administration Conference, USENIX (1999).Google Scholar
Digital Library
- R. Schweller, A. Gupta, E. Parsons, and Y. Chen. 2004. Reversible Sketches for Efficient and Accurate Change Detection over Network Data Streams . Proc. of IMC (2004).Google Scholar
- S. Staniford, V. Paxson, and N. Weaver. 2002. How to 0wn the Internet in Your Spare Time . Proc. of USENIX Security Symposium (August 2002).Google Scholar
- UCSD. 2015. CAIDA UCSD Anonymized 2015 Internet Traces on Jan. 17 . http://www.caida.org/data/passive/passive_2015_dataset.xml .Google Scholar
- S. Venkatataman, D. Song, P. Gibbons, and A. Blum. 2005. New Streaming Algorithms for Fast Detection of Superspreaders . Proc. of NDSS (February 2005).Google Scholar
- H. Wang, D. Zhang, and K. G. Shin. 2002. SYN-dog: Sniffing SYN Flooding Sources . Proc. of 22nd International Conference on Distributed Computing Systems (ICDCS'02) (July 2002).Google Scholar
Cross Ref
- K. Whang, B. T. Vander-Zanden, and H. M. Taylor. 1990. A Linear-time Probabilistic Counting Algorithm for Database Applications . ACM Transactions on Database Systems , Vol. 15, 2 (1990), 208--229.Google Scholar
Digital Library
- Wikipedia. Online. Chebyshev inequality . https://en.wikipedia.org/wiki/Chebyshev%27s_inequalityGoogle Scholar
- Q. Xiao, S. Chen, M. Chen, and Y. Ying. 2015. Hyper-Compact Virtual Estimators for Big Network Data Based on Register Sharing . in Proc. of ACM SIGMETRICS (2015).Google Scholar
Digital Library
- Q. Xiao, S. Chen, Y. Zhou, M. Chen, J. Luo, T. Li, and Y. Ling. 2017. Cardinality Estimation for Elephant Flows: A Compact Solution based on Virtual Register Sharing . IEEE/ACM Transactions on Networking (2017).Google Scholar
- T. Yang, J. Jiang, P. Liu, Q. Huang, J. Gong, Y. Zhou, R. Miao, X. Li, and S. Uhlig. 2018. Elastic Sketch: Adaptive and Fast Network-wide Measurements . Proc. of ACM SIGCOMM (August 2018).Google Scholar
- M. Yoon, T. Li, S. Chen, and J. Peir. 2009. Fit a Spread Estimator in Small Memory . Proc. of IEEE INFOCOM (April 2009).Google Scholar
- M. Yoon, T. Li, S. Chen, and J. Peir. 2011. Fit a Compact Spread Estimator in Small High-Speed Memory . IEEE/ACM Transactions on Networking , Vol. 19, 5 (October 2011), 1253--1264.Google Scholar
Digital Library
- M. Yu, L. Jose, and R. Miao. 2013. Software Defined Traffic Measurement with OpenSketch . Proc. of USENIX Symposium on Networked Systems Design and Implementation (2013).Google Scholar
- Y. Zhang, S. Singh, S. Sen, N. Duffield, and C. Lund. 2004. Online Identification of Hierarchical Heavy Hitters: Algorithms, Evaluation, and Application . Proc. of ACM SIGCOMM IMC (October 2004).Google Scholar
- Y. Zhou, Y. Zhou, M. Chen, and S. Chen. 2017. Persistent Spread Measurement for Big Network Data Based on Register Intersection . Proc. of ACM SIGMETRICS (2017).Google Scholar
Index Terms
Generalized Sketch Families for Network Traffic Measurement
Recommendations
Generalized Sketch Families for Network Traffic Measurement
Traffic measurement provides critical information for network management, resource allocation, traffic engineering, and attack detection. Most prior art has been geared towards specific application needs with specific performance objectives. To support ...
Generalized Sketch Families for Network Traffic Measurement
SIGMETRICS '20: Abstracts of the 2020 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer SystemsTraffic measurement provides critical information for network management, resource allocation, traffic engineering, and attack detection. Most prior art has been geared towards specific application needs with specific performance objectives. To support ...
Applications of sketches in network traffic measurement: A survey
AbstractAccurate and timely network traffic measurement is essential for network status monitoring, network fault analysis, network intrusion detection, and network security management. With the rapid development of the network, massive ...
Highlights- Main traffic measurement tasks and advantages of applying sketches.
- Criteria to ...






Comments