Parallel random numbers: as easy as 1, 2, 3

ABSTRACT

Most pseudorandom number generators (PRNGs) scale poorly to massively parallel high-performance computation because they are designed as sequentially dependent state transformations. We demonstrate that independent, keyed transformations of counters produce a large alternative class of PRNGs with excellent statistical properties (long period, no discernable structure or correlation). These counter-based PRNGs are ideally suited to modern multi-core CPUs, GPUs, clusters, and special-purpose hardware because they vectorize and parallelize well, and require little or no memory for state. We introduce several counter-based PRNGs: some based on cryptographic standards (AES, Threefish) and some completely new (Philox). All our PRNGs pass rigorous statistical tests (including TestU01's BigCrush) and produce at least 2⁶⁴ unique parallel streams of random numbers, each with period 2¹²⁸ or more. In addition to essentially unlimited parallel scalability, our PRNGs offer excellent single-chip performance: Philox is faster than the CURAND library on a single NVIDIA GPU.

References

1. M. Bellare and P. Rogaway. Pseudorandom functions. In Introduction to Modern Cryptography. UCSD CSE 207 Online Course Notes, 2011. Chap 3. http://cseweb.ucsd.edu/~mihir/cse207/w-prf.pdf.
2. D. J. Bernstein and P. Schwabe. New AES software speed records. In D. R. Chowdhury and V. Rijmen, editors, Progress in Cryptology - INDOCRYPT 2008, volume 5365 of Lecture Notes in Computer Science, pages 322--336, Berlin, 2008. Springer-Verlag.
3. R. P. Brent. Some long-period random number generators using shifts and xors. ANZIAM Journal, 48:C188--C202, 2007.
4. R. G. Brown. Dieharder: A random number test suite. http://phy.duke.edu/~rgb/General/dieharder.php.
5. D. S. Cerutti, R. Duke, P. L. Freddolino, H. Fan, and T. P. Lybrand. Vulnerability in popular molecular dynamics packages concerning Langevin and Andersen dynamcs. J. Chem. Theory, 4:1669--1680, 2008.
6. P. Coddington. Random number generators for parallel computers. Technical Report 13, Northeast Parallel Architecture Center, 1997.
7. A. De Matteis and S. Pagnutti. Parallelization of random number generators and long-range correlations. Numer. Math., 53:595--608, August 1988.
8. A. De Matteis and S. Pagnutti. Long-range correlations in linear and non-linear random number generators. Parallel Computing, 14:207--210, 1990.
9. M. Dworkin. Recommendation for block cipher modes of operation, methods and techniques. NIST Special Publication 800-38A. National Institute of Standards and Technology (NIST), 2001.
10. H. Feistel. Cryptography and computer privacy. Scientific American, 228(5):15--23, 1973.
11. N. Ferguson, S. Lucks, B. Schneier, B. Whiting, M. Bellare, T. Kohno, J. Callas, and J. Walker. The Skein hash function family. http://www.schneier.com/skein.pdf, 2010.
12. A. M. Ferrenberg, D. P. Landau, and Y. J. Wong. Monte Carlo simulations: Hidden errors from "good" random number generators. Phy. Rev. Lett., 69:3382--3384, 1992.
13. G. C. Fox, M. A. Johnson, G. A. Lyzenga, S. W. Otto, J. K. Salmon, and D. W. Walker. Solving Problems on Concurrent Processors; Volume 1: General Techniques and Regular Problems. Prentice-Hall, 1988.
14. S. Gueron. Intel Advanced Encryption Standard (AES) instructions set. Technical report, Intel, 2010.
15. P. Hellekalek. Don't trust parallel Monte Carlo! In Proc. 12 ^th Workshop on Parallel and Distributed Simulation, PADS '98, pages 82--89, Washington, D. C., 1998. IEEE Computer Society.
16. P. Hellekalek. Good random number generators are (not so) easy to find. Math. Comput. Simul., 46:485--505, June 1998.
17. P. Hellekalek and S. Wegenkittl. Empirical evidence concerning AES. ACM Trans. Model. Comput. Simul., 13:322--333, October 2003.
18. Intel. Vector Statistical Library (VSL) performance data. http://software.intel.com/sites/products/documentation/hpc/mkl/vsl/vsl_performance_data.htm.
19. M. H. Kalos and P. A. Whitlock. Monte Carlo Methods. Wiley-VCH, 2nd edition, 2008.
20. D. E. Knuth. The Art of Computer Programming, Volume 2 (3rd ed.): Seminumerical Algorithms. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1997.
21. P. L'Ecuyer. Combined multiple recursive random number generators. Operations Research, 44(5):816--822, 1996.
22. P. L'Ecuyer. Random number generation. In J. E. Gentle, W. Haerdle, and Y. Mori, editors, Handbook of Computational Statistics, pages 35--70. Springer-Verlag, Berlin, 2004. Chapter II.2.
23. P. L'Ecuyer, F. Blouin, and R. Couture. A search for good multiple recursive random number generators. ACM Trans. Model. Comput. Simul., 3(2):87--98, 1993.
24. P. L'Ecuyer and R. Simard. TestU01: A C library for empirical testing of random number generators. ACM Trans. Math. Softw., 33, August 2007.
25. D. H. Lehmer. Mathematical methods in large-scale computing units. In Proc. 2nd Symp. on Large-Scale Digital Calculating Machinery, pages 141--146. Harvard University Press, 1949.
26. G. Marsaglia. DIEHARD: A battery of tests of randomness. http://stat.fsu.edu/~geo/diehard.html.
27. G. Marsaglia. Xorshift RNGs. J. Stat. Soft., 8:1--6, 2003.
28. M. Mascagni and A. Srinivasan. Algorithm 806: SPRNG: A scalable library for pseudorandom number generation. ACM Transactions on Mathematical Software, 26:436--461, 2000.
29. M. Matsumoto and T. Nishimura. Mersenne twister: A 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul., 8:3--30, January 1998.
30. M. Matsumoto, I. Wada, A. Kuramoto, and H. Ashihara. Common defects in initialization of pseudorandom number generators. ACM Trans. Model. Comput. Simul., 17, September 2007.
31. E. H. Mckinney. Generalized birthday problem. The American Mathematical Monthly, 73(4):385--387, April 1966.
32. National Bureau of Standards. Data Encryption Standard. FIPS PUB 46--3, 1977.
33. National Institute of Standards and Technology. Cryptographic hash algorithm competition website. http://csrc.nist.gov/groups/ST/hash/sha-3/index.html.
34. National Institute of Standards and Technology. Advanced Encryption Standard (AES). FIPS PUB 197, 2001.
35. F. Panneton, P. L'Ecuyer, and M. Matsumoto. Improved long-period generators based on linear recurrences modulo 2. ACM Trans. Math. Softw., 32:1--16, March 2006.
36. S. K. Park and K. W. Miller. Random number generators: good ones are hard to find. Commun. ACM, 31:1192--1201, October 1988.
37. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical Recipes in C. Cambridge University Press, Cambridge, 2^nd edition, 1992.
38. A. Rukhin, J. Soto, J. Nechvatal, M. Smid, E. Barker, M. Levinson, M. Vangel, D. Banks, A. Heckert, J. Dray, and S. Vo. A statistical test suite for random and pseudorandom number generators for cryptographic applications. Special Publication 800-22 Revision 1a, NIST, April 2010.
39. C. E. Shannon. Communication theory of secrecy systems. Bell System Technical Journal, 28:656--715, 1949.
40. D. E. Shaw, R. O. Dror, J. K. Salmon, J. P. Grossman, K. M. Mackenzie, J. A. Bank, C. Young, M. M. Deneroff, B. Batson, K. J. Bowers, E. Chow, M. P. Eastwood, D. J. Ierardi, J. L. Klepeis, J. S. Kuskin, R. H. Larson, K. Lindorff-Larsen, P. Maragakis, M. A. Moraes, S. Piana, Y. Shan, and B. Towles. Millisecond-scale molecular dynamics simulations on Anton. In Proc. Conf. on High Performance Computing, Networking, Storage and Analysis, SC09, pages 39:1--39:11, New York, NY, 2009. ACM.
41. D. E. Shaw, P. Maragakis, K. Lindorff-Larsen, S. Piana, R. O. Dror, M. P. Eastwood, J. A. Bank, J. M. Jumper, J. K. Salmon, Y. Shan, and W. Wriggers. Atomic-level characterization of the structural dynamics of proteins. Science, 330:341--346, 2010.
42. D. J. Sindhikara, S. Kim, A. F. Voter, and A. E. Roitberg. Bad seeds sprout perilous dynamics: Stochastic thermostat induced trajectory synchronization in biomolecules. J. Chem. Theory and Comp., 5(6):1624--1631, 2009.
43. J. L. Smith. The design of Lucifer, a cryptographic device for data communications. IBM Research Report RC3326, IBM T. J. Watson Research Center, Yorktown Heights, NY 10598, USA, Apr. 1971.
44. A. Sorkin. Lucifer, a crytographic algorithm. Cryptologia, 8:22--35, 1984.
45. S. Tzeng and L.-Y. Wei. Parallel white noise generation on a GPU via cryptographic hash. In Proc. 2008 Symp. on Interactive 3D graphics and games, I3D '08, pages 79--87, New York, NY, 2008. ACM.
46. S. Ulam, R. Richtmeyer, and J. von Neumann. Statistical methods in neutron diffusion. Technical Report LAMS-551, Los Alamos Scientific Laboratory, April 1947.
47. J. von Neuman. Various techniques used in connection with random digits. In A. Householder, G. Forsythe, and H. Germond, editors, Monte Carlo Method, Applied Math Series, Volume 11, pages 36--38. National Bureau of Standards, 1951.
48. F. Zafar, M. Olano, and A. Curtis. GPU random numbers via the Tiny Encryption Algorithm. In Proc. Conf. High Performance Graphics, HPG '10, pages 133--141, Aire-la-Ville, Switzerland, 2010. Eurographics Association.

Index Terms

• Computer systems organization

  • Dependable and fault-tolerant systems and networks

• General and reference

  • Cross-computing tools and techniques

    • Performance

• Mathematics of computing

  • Probability and statistics

    • Probabilistic reasoning algorithms

      • Random number generation

• Networks

  • Network performance evaluation

• Theory of computation

  • Models of computation

    • Concurrency

      • Parallel computing models