skip to main content
research-article

Adding concurrency in python using a commercial processor's hardware transactional memory support

Published:29 April 2010Publication History
Skip Abstract Section

Abstract

This paper reports on our experiences of using a commercial processor's best-effort hardware transactional memory to improve concurrency in CPython, the reference Python implementation. CPython protects its data structures using a single global lock, which inhibits parallelism when running multiple threads.

We modified the CPython interpreter to use besteffort hardware transactions available in Sun's Rock processor, and fall back on the single global lock when unable to commit in hardware. The modifications were minimal; however, we had to restructure some of CPython's shared data structures to handle false conflicts arising from CPython's management of the shared data. Our results show that the modified CPython interpreter can run small, simple, workloads and scale almost linearly, while improving the concurrency of more complex workloads.

References

  1. TIOBE programming community index for May. TIOBE Software BV, 2010.Google ScholarGoogle Scholar
  2. D. Beazley. Inside the python GIL (slides). Presented at the Python Concurrency Workshop, 2009.Google ScholarGoogle Scholar
  3. C. Blundell, A. Raghavan, and M. M. K. Martin. RETCON: Transactional repair without replay. In the proceedings of the 37th annual international symposium on Computer architecture. ACM, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. S. Chaudhry, R. Cypher, M. Ekman, M. Karlsson, A. Landin, S. Yip, H. Zeffer, and M. Tremblay. Rock: A high-performance SPARC CMT processor. IEEE Micro, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. D. Dice, Y. Lev, M. Moir, and D. Nussbaum. Early experience with a commercial hardware transactional memory implementation. In the proceeding of the 14th international conference on Architectural support for programming languages and operating systems. ACM, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. Dice, Y. Lev, M. Moir, D. Nussbaum, and M. Olszewski. Early experience with a commercial hardware transactional memory implementation. TR-2009-180, Sun Microsystems Laboratories, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. F. Ellen, Y. Lev, V. Luchangco, and M. Moir. SNZI: scalable nonzero indicators. In the proceedings of the twenty-sixth annual ACM symposium on Principles of distributed computing. ACM, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. D. Geer. Chip makers turn to multicore processors. IEEE Computer, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. M. Herlihy and J. E. B. Moss. Transactional memory: architectural support for lock-free data structures. In the proceedings of the 20th annual international symposium on Computer architecture. ACM, 1993. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. J. Juneau, J. Baker, F. Wierzbicki, L. S. Munoz, and V. Ng. The Definitive Guide to Jython: Python for the Java Platform. Apress, 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. J. R. Larus and R. Rajwar. Transactional Memory. Synthesis Lectures on Computer Architecture. Morgan and Claypool Publishers, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. M. Moir. Hybrid transactional memory. Sun Microsystems Laboratories, 2005.Google ScholarGoogle Scholar
  13. R. Rajwar and J. R. Goodman. Speculative lock elision: enabling highly concurrent multithreaded execution. In the proceedings of the 34th annual ACM/IEEE international symposium on Microarchitecture. IEEE Computer Society, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. H. E. Ramadan, C. J. Rossbach, and E. Witchel. Dependence-aware transactional memory for increased concurrency. In the proceedings of the 41st annual IEEE/ACM International Symposium on Microarchitecture. IEEE Computer Society, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. A. Rigo and S. Pedroni. PyPy's approach to virtual machine construction. In the companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. N. Riley and C. Zilles. Hardware tansactional memory support for lightweight dynamic language evolution. In the companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications. ACM, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. G. Stein. Free threading. Python Software Foundation, 2001. http://mail.python.org/pipermail/python-dev/2001-August/017099.html.Google ScholarGoogle Scholar
  18. G. van Rossum. The Python Language Reference: Release 2.6.4. Python Software Foundation, 2009.Google ScholarGoogle Scholar
  19. G. van Rossum. The Python/C API: Release 2.6.4. Python Software Foundation, 2009.Google ScholarGoogle Scholar
  20. G. van Rossum. Extending and Embedding Python: Release 2.6.4. Python Software Foundation, 2009.Google ScholarGoogle Scholar
  21. G. van Rossum. The Python Standard Library: Release 2.6.4. Python Software Foundation, 2009.Google ScholarGoogle Scholar
  22. R. P. Weicker. Dhrystone: a synthetic systems programming benchmark. Communications of the ACM, 1984. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Adding concurrency in python using a commercial processor's hardware transactional memory support

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM SIGARCH Computer Architecture News
          ACM SIGARCH Computer Architecture News  Volume 38, Issue 5
          December 2010
          35 pages
          ISSN:0163-5964
          DOI:10.1145/1978907
          Issue’s Table of Contents

          Copyright © 2010 Author

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 29 April 2010

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!