skip to main content
research-article

Compressed matrix multiplication

Published:22 August 2013Publication History
Skip Abstract Section

Abstract

We present a simple algorithm that approximates the product of n-by-n real matrices A and B. Let ‖AB‖F denote the Frobenius norm of AB, and b be a parameter determining the time/accuracy trade-off. Given 2-wise independent hash functions h1,h2: [n]→ [b], and s1,s2: [n]→ {−1,+1} the algorithm works by first “compressing” the matrix product into the polynomial p(x) = ∑k=1n \left(∑i=1n Aik s1(i) xh1(i)\right) \left(∑j=1n Bkj s2(j) xh2(j)\right). Using the fast Fourier transform to compute polynomial multiplication, we can compute c0,…,cb−1 such that ∑i ci xi = (p(x) mod xb) + (p(x) div xb) in time Õ(n2+ nb). An unbiased estimator of (AB)ij with variance at most ‖ABF2/b can then be computed as: Cij = s1(i)s2(j)c(h1(i)+h2(j)) mod b. Our approach also leads to an algorithm for computing AB exactly, with high probability, in time Õ(N + nb) in the case where A and B have at most N nonzero entries, and AB has at most b nonzero entries.

Skip Supplemental Material Section

Supplemental Material

References

  1. Alon, N., Gibbons, P. B., Matias, Y., and Szegedy, M. 2002. Tracking join and self-join sizes in limited storage. J. Comput. Syst. Sci 64, 3, 719--747.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Alon, N., Matias, Y., and Szegedy, M. 1999. The space complexity of approximating the frequency moments. J. Comput. Syst. Sci 58, 1, 137--147. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Braverman, V., Chung, K.-M., Liu, Z., Mitzenmacher, M., and Ostrovsky, R. 2010. AMS without 4-wise independence on product domains. In Proceedings of the 27th International Symposium on Theoretical Aspects of Computer Science (STACS). Leibniz International Proceedings in Informatics (LIPIcs) Series, vol. 5. 119--130.Google ScholarGoogle Scholar
  4. Carter, J. L. and Wegman, M. N. 1979. Universal classes of hash functions. J. Comput. Syst. Sci. 18, 2, 143--154.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Charikar, M., Chen, K., and Farach-Colton, M. 2004. Finding frequent items in data streams. Theoret. Comput. Sci. 312, 1, 3--15. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Clarkson, K. L. and Woodruff, D. P. 2009. Numerical linear algebra in the streaming model. In Proceedings of Symposium on Theory of Computing (STOC). ACM, 205--214. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Cohen, E. 1998. Structure prediction and computation of sparse matrix products. J. Combin. Optimat. 2, 4, 307--332.Google ScholarGoogle ScholarCross RefCross Ref
  8. Cohen, E. and Lewis, D. D. 1999. Approximating matrix multiplication for pattern recognition tasks. J. Algor. 30, 2, 211--252. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Cooley, J. W. and Tukey, J. W. 1965. An algorithm for the machine calculation of complex Fourier series. Math. Computat. 19, 90, 297--301.Google ScholarGoogle ScholarCross RefCross Ref
  10. Coppersmith, D. and Winograd, S. 1990. Matrix multiplication via arithmetic progressions. J. Symbol. Computat. 9, 3, 251--280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Dietzfelbinger, M., Gil, J., Matias, Y., and Pippenger, N. 1992. Polynomial hash functions are reliable (extended abstract). In Proceedings of the 19th International Colloquium on Automata, Languages and Programming. Lecture Notes in Computer Science Series, vol. 623, Springer, 235--246. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Drineas, P., Kannan, R., and Mahoney, M. W. 2006. Fast Monte Carlo algorithms for matrices I: Approximating matrix multiplication. SIAM J. Comput. 36, 1, 132--157. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Gilbert, A. and Indyk, P. 2010. Sparse recovery using sparse matrices. Proc. IEEE 98, 6, 937--947.Google ScholarGoogle ScholarCross RefCross Ref
  14. Gilbert, A. C., Li, Y., Porat, E., and Strauss, M. J. 2010. Approximate sparse recovery: optimizing time and measurements. In Proceedings of the 42nd ACM Symposium on Theory of Computing (STOC). ACM, 475--484. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Goodman, L. A. 1960. On the exact variance of products. J. Amer. Statis. Assoc. 55, 292, 708--713.Google ScholarGoogle ScholarCross RefCross Ref
  16. Indyk, P. and McGregor, A. 2008. Declaring independence via the sketching of sketches. In Proceedings of the 19th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA). SIAM, 737--745. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Iwen, M. A. and Spencer, C. V. 2009. A note on compressed sensing and the complexity of matrix multiplication. Inf. Process. Lett. 109, 10, 468--471. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Kutzkov, K. 2013. Deterministic algorithms for skewed matrix products. In Proceedings of the 30th International Symposium on Theoretical Aspects of Computer Science (STACS). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, vol. 20, 466--477.Google ScholarGoogle Scholar
  19. Le Gall, F. 2012. Faster algorithms for rectangular matrix multiplication. In Proceedings of the 53rd Annual IEEE Symposium on Foundations of Computer Science (FOCS). IEEE, 514--523. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Lingas, A. 2009. A fast output-sensitive algorithm for Boolean matrix multiplication. In Proceedings of 17th European Symposium on Algorithms (ESA). Lecture Notes in Computer Science Series, vol. 5757, Springer, 408--419.Google ScholarGoogle ScholarCross RefCross Ref
  21. Mahoney, M. W. 2012. Algorithmic and statistical perspectives on large-scale data analysis. In Combinatorial Scientific Computing. Chapman and Hall/CRC Press.Google ScholarGoogle Scholar
  22. Motwani, R. and Raghavan, P. 1995. Randomized Algorithms. Cambridge University Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Pagh, R. 2012. Compressed matrix multiplication. In Proceedings of Innovations in Theoretical Computer Science (ITCS). ACM, 442--451. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Paturi, R., Rajasekaran, S., and Reif, J. 1995. The light bulb problem. Inf. Computat. 117, 2, 187--192. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Pǎtraşcu, M. and Thorup, M. 2011. The power of simple tabulation hashing. In Proceedings of the 43rd Annual ACM Symposium on Theory of Computing (STOC). ACM, 1--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Resen Amossen, R., Campagna, A., and Pagh, R. 2010. Better size estimation for sparse matrix products. In Proceedings of 13th International Workshop on Approximation, Randomization, and Combinatorial Optimization (APPROX-RANDOM). Lecture Notes in Computer Science Series, vol. 6302, Springer, 406--419. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Resen Amossen, R. and Pagh, R. 2009. Faster join-projects and sparse matrix multiplications. In Proceedings of 12th International Conference on Database Theory (ICDT). ACM International Conference Proceeding Series Series, vol. 361. ACM, 121--126. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Sarlós, T. 2006. Improved approximation algorithms for large matrices via random projections. In Proceedings of the IEEE Symposium on Foundations of Computer Science (FOCS). IEEE Computer Society, 143--152. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Sipser, M. and Spielman, D. A. 1996. Expander codes. IEEE Trans. Inf. Theory 42, 6, 1710--1722. Google ScholarGoogle ScholarCross RefCross Ref
  30. Valiant, G. 2012. Finding correlations in subquadratic time, with applications to learning parities and juntas. In Proceedings of Symposium on Foundations of Computer Science (FOCS). IEEE Computer Society, 11--20. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Vassilevska Williams, V. 2012. Multiplying matrices faster than Coppersmith-Winograd. In Proceedings of the 44th Symposium on Theory of Computing Conference (STOC). ACM, 887--898. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yuster, R. and Zwick, U. 2005. Fast sparse matrix multiplication. ACM Trans. Algor. 1, 1, 2--13. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Compressed matrix multiplication

            Recommendations

            Reviews

            James Harold Davenport

            This paper is concerned with approximating a matrix product, that is, finding the large entries ( AB ){ i , j } [and the corresponding indices ( i , j )] from A and B without computing AB first. Two main applications are cited explicitly, and a third implicitly: (1) computing a covariance matrix from samples, where most of the variables are independent, and hence the sampled covariances are mostly small (and uninteresting); (2) JPEG-style transformation of column vectors to a basis in which they are (approximately) sparse; and (3) when AB is genuinely sparse, that is, most of the entries are not merely small but actually zero. Let A and B be n × n matrices (the paper does consider nonsquare matrices as well), with b large (or nonzero) entries. In the case of a sparse product, let be the number of entries in AB that are not trivially zero, that is, they are either nonzero, or zero because two or more nonzero summands cancel. Let || A || F denote the Frobenius norm of the matrix A , that is, the square root of the sum of the squares of the entries. The aim, then, is to compute a matrix C such that || C - AB || F is small. Without actually computing AB , the author regards AB as being a sparse signal to the approximated result. The methods are probabilistic, “compressing” A and B , via hash functions, into polynomials, using fast Fourier transform methods to multiply the polynomials, and then using the author's algorithm FindSignificantEntries to reconstruct the location of the large elements of AB . This algorithm is not easy to follow: the reader should be warned at the outset that S is a multiset, not a set, but iteration over S is over distinct members of S . I understand the logic in terms of , but confess to not following the reduction to b . The results are somewhat technical, since they are all probabilistic in terms of the hash functions, but a typical result would be an algorithm to compute a sparse product, with high probability, in time ~ O ( N + nb ), where N bounds the number of nonzeros in A and B . There is no experimental analysis, which is rather a pity, as (1) the paper is full of ~ O - and o -expressions, which makes practical appreciation hard; (2) the key reduction is to fast Fourier transforms, which are generally implemented efficiently on modern hardware; and (3) it is claimed (page 9:8) that the method only makes a single pass over A and B (assumed stored in column-major and row-major order, respectively), which is a significant improvement on straightforward multiplication. Online Computing Reviews Service

            Soubhik Chakraborty

            There has been a lot of research on matrix multiplication algorithms in the recent past. For insight, one should read the second volume of Knuth's wonderful set of books on algorithms [1]. In this paper, the author presents a novel algorithm that finds an approximation of the product of two n × n real matrices. The idea is to compress the matrix product into a polynomial and then use the Fourier transform. The algorithm takes the time O( N + nb ) where N is the maximum number of nonzero entries and b is a parameter depicting the time-accuracy tradeoff. The paper is interesting. Researchers and postgraduate students in computer science will definitely find it useful. Note that Amir Schoor's algorithm [2] on sparse matrices is also fast for dense matrices as (1) it is faster to work with rows only than both rows and columns, and (2) the statistical bound for this algorithm is O( n × n ) as the n × n exact number of comparisons dominate over the O( d 1 d 2 n 3) multiplications on average. d 1 and d 2 are the densities of pre- and post-factor matrices. It happens because when comparisons and multiplications are collectively taken (by weight-based analysis, for example, the time of an operation can be taken as its weight), the comparisons outweigh the “lighter” multiplications. It might be worth trying Pagh's strategy on Schoor's algorithm. Online Computing Reviews Service

            Access critical reviews of Computing literature here

            Become a reviewer for Computing Reviews.

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            • Published in

              cover image ACM Transactions on Computation Theory
              ACM Transactions on Computation Theory  Volume 5, Issue 3
              Special issue on innovations in theoretical computer science 2012
              August 2013
              94 pages
              ISSN:1942-3454
              EISSN:1942-3462
              DOI:10.1145/2493252
              Issue’s Table of Contents

              Copyright © 2013 ACM

              Publisher

              Association for Computing Machinery

              New York, NY, United States

              Publication History

              • Published: 22 August 2013
              • Revised: 1 March 2013
              • Accepted: 1 March 2013
              • Received: 1 August 2012
              Published in toct Volume 5, Issue 3

              Permissions

              Request permissions about this article.

              Request Permissions

              Check for updates

              Qualifiers

              • research-article
              • Research
              • Refereed

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!