Abstract
We describe a general framework for adding the values of two approximate counters to produce a new approximate counter value whose expected estimated value is equal to the sum of the expected estimated values of the given approximate counters. (To the best of our knowledge, this is the first published description of any algorithm for adding two approximate counters.) We then work out implementation details for five different kinds of approximate counter and provide optimized pseudocode. For three of them, we present proofs that the variance of a counter value produced by adding two counter values in this way is bounded, and in fact is no worse, or not much worse, than the variance of the value of a single counter to which the same total number of increment operations have been applied. Addition of approximate counters is useful in massively parallel divide-and-conquer algorithms that use a distributed representation for large arrays of counters. We describe two machine-learning algorithms for topic modeling that use millions of integer counters, and confirm that replacing the integer counters with approximate counters is effective, speeding up a GPU-based implementation by over 65% and a CPU-based by nearly 50%, as well as reducing memory requirements, without degrading their statistical effectiveness.
- A. C. Callahan. Random rounding: Some principles and applications. In ICASSP '76: IEEE Intl. Conf. Acoustics, Speech, and Signal Processing, volume 1, pages 501--504, Apr 1976.Google Scholar
Cross Ref
- Miklós Csűrös. Approximate counting with a floating-point counter. In COCOON '10: Proc. 16th Annual International Conf. Computing and Combinatorics, pages 358--367, Berlin, Heidelberg, 2010. Springer-Verlag. Google Scholar
Digital Library
- Andrej Cvetkovski. An algorithm for approximate counting using limited memory resources. In SIGMETRICS '07: Proc. 2007 ACM SIGMETRICS International Conf. Measurement and Modeling of Computer Systems, pages 181--190, New York, 2007. ACM. Google Scholar
Digital Library
- Dave Dice, Yossi Lev, and Mark Moir. Scalable statistics counters. In PPoPP '13: Proc. 18th ACM SIGPLAN Symp. Principles and Practice of Parallel Programming, pages 307--308, New York, 2013. ACM. Google Scholar
Digital Library
- Philippe Flajolet. Approximate counting: A detailed analysis. BIT Numerical Mathematics, 25(1):113--134, 1985. Google Scholar
Digital Library
- George E. Forsythe. Reprint of a note on rounding-off errors. SIAM Review, 1(1):66--67, 1959. dx.doi.org/10.1137/1001011.Google Scholar
Digital Library
- Scott A. Mitchell and David M. Day. Flexible approximate counting. In IDEAS '11: Proc. 15th Symp. International Database Engineering & Applications, pages 233--239, New York, 2011. ACM. Google Scholar
Digital Library
- Robert Morris. Counting large numbers of events in small registers. Commun. ACM, 21(10):840--842, October 1978. Google Scholar
Digital Library
- D. S. Parker, B. Pierce, and P. R. Eggert. Monte Carlo arithmetic: How to gamble with floating point and win. Computing in Science Engineering, 2(4):58--68, July 2000. Google Scholar
Digital Library
- D. Stott Parker. Monte Carlo arithmetic: Exploiting randomness in floating-point arithmetic. Technical Report 970002, Computer Science Department, UCLA, Los Angeles, CA, March 1997. http://fmdb.cs.ucla.edu/Treports/970002.pdf.Google Scholar
- Marc Snir, Steve Otto, Steven Huss-Lederman, David Walker, and Jack Dongarra. MPI---The Complete Reference: Volume 1, The MPI Core. MIT Press, Cambridge, Massachusetts, second edition, 1998. Google Scholar
Digital Library
- Rade Stanojevic. Small active counters. In 26th IEEE International Conf. Computer Communications (INFOCOM 2007), pages 2153--2161, May 2007.Google Scholar
Digital Library
- Jean-Baptiste Tristan, Joseph Tassarotti, and Guy L. Steele Jr. Efficient training of LDA on a GPU by Mean-For-Mode Gibbs sampling. In ICML 2015: 32nd International Conf. Machine Learning, volume 37, July 2015. Volume 37 of the Journal in Machine Learning Research: Workshop and Conference Proceedings.Google Scholar
- A. J. Walker. Fast generation of uniformly distributed pseudorandom numbers with floating-point representation. Electronics Letters, 10(25):533--534, December 1974.Google Scholar
Cross Ref
- Manzil Zaheer, Michael Wick, Jean-Baptiste Tristan, Alex Smola, and Guy L. Steele Jr. Exponential stochastic cellular automata for massively parallel inference. In LearningSys: Workshop on Machine Learning Systems at Neural Information Processing Systems (NIPS), December 2015. http://learningsys.org/papers/LearningSys_2015_paper_11.pdf.Google Scholar
- Manzil Zaheer, Michael Wick, Jean-Baptiste Tristan, Alex Smola, and Guy L. Steele Jr. Exponential stochastic cellular automata for massively parallel inference. In AISTATS: 19th Intl. Conf. Artificial Intelligence and Statistics, May 2016. To appear.Google Scholar
Index Terms
Adding approximate counters
Recommendations
Adding Approximate Counters
Special Issue: Invited papers from PPoPP 2016, Part 1We describe a general framework for adding the values of two approximate counters to produce a new approximate counter value whose expected estimated value is equal to the sum of the expected estimated values of the given approximate counters. (To the ...
Adding approximate counters
PPoPP '16: Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel ProgrammingWe describe a general framework for adding the values of two approximate counters to produce a new approximate counter value whose expected estimated value is equal to the sum of the expected estimated values of the given approximate counters. (To the ...
Threshold counters with increments and decrements
A threshold counter is a shared data structure that assumes integer values. It provides two operations: Increment changes the current counter value from v to v+1, while Read returns the value [v/w], where v is the current counter value and w is a fixed ...






Comments