ACM Digital Library home
ACM home
Advanced Search
10.1145/1508293.1508305acmconferencesArticle/Chapter ViewAbstractPublication PagesveeConference Proceedingsconference-collections
research-article

Demystifying magic: high-level low-level programming

VEE '09: Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environmentsMarch 2009 Pages 81–90https://doi.org/10.1145/1508293.1508305
Published:11 March 2009Publication History
  • 45citation
  • 1,077
  • Downloads
Metrics
Total Citations45
Total Downloads1,077
Last 12 Months74
Last 6 weeks1

ABSTRACT

The power of high-level languages lies in their abstraction over hardware and software complexity, leading to greater security, better reliability, and lower development costs. However, opaque abstractions are often show-stoppers for systems programmers, forcing them to either break the abstraction, or more often, simply give up and use a different language. This paper addresses the challenge of opening up a high-level language to allow practical low-level programming without forsaking integrity or performance.

The contribution of this paper is three-fold: 1) we draw together common threads in a diverse literature, 2) we identify a framework for extending high-level languages for low-level programming, and 3) we show the power of this approach through concrete case studies. Our framework leverages just three core ideas: extending semantics via intrinsic methods, extending types via unboxing and architectural-width primitives, and controlling semantics via scoped semantic regimes. We develop these ideas through the context of a rich literature and substantial practical experience. We show that they provide the power necessary to implement substantial artifacts such as a high-performance virtual machine, while preserving the software engineering benefits of the host language.

The time has come for high-level low-level programming to be taken more seriously: 1) more projects now use high-level languages for systems programming, 2) increasing architectural heterogeneity and parallelism heighten the need for abstraction, and 3) a new generation of high-level languages are under development and ripe to be influenced.

References

  1. A.-R. Adl-Tabatabai, J. Bharadwaj, D.-Y. Chen, A. Ghuloum, V. Menon, B. Murphy, M. Serrano, and T. Shpeisman. The StarJIT compiler: a dynamic compiler for managed runtime environments. Intel Technology Journal, 7(1):19--31, Feb. 2003.Google ScholarGoogle Scholar
  2. V. Agarwal, M. S. Hrishikesh, S. W. Keckler, and D. Burger. Clock rate versus IPC: the end of the road for conventional microarchitectures. In ISCA'00: Proceedings of the 27th Annual International Symposium on Computer Architecture, pages 248--259, New York, NY, USA, 2000. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. M. Aiken, M. Fähndrich, C. Hawblitzel, G. Hunt, and J. Larus. Deconstructing process isolation. In MSPC'06: Proceedings of the 2006 Workshop on Memory System Performance and Correctness, pages 1--10, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. E. Allen, D. Chase, J. Hallett, V. Luchangco, J.-W. Maesse, S. Ryu, G. L. S. Jr., and S. Tobin-Hochstadt. The Fortress language specification, version 1.0. Technical report, Sun Microsystems, Mar. 2008.Google ScholarGoogle Scholar
  5. B. Alpern, C. R. Attanasio, A. Cocchi, D. Lieber, S. Smith, T. Ngo, J. J. Barton, S. F. Hummel, J. C. Sheperd, and M. Mergen. Implementing Jalapeño in Java. In OOPSLA'99: Proceedings of the 14th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 314--324, New York, NY, USA, 1999. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. B. Alpern, D. Attanasio, J. J. Barton, M. G. Burke, P. Cheng, J.-D. Choi, A. Cocchi, S. J. Fink, D. Grove, M. Hind, S. F. Hummel, D. Lieber, V. Litvinov, M. Mergen, T. Ngo, J. R. Russell, V. Sarkar, M. J. Serrano, J. Shepherd, S. Smith, V. C. Sreedhar, H. Srinivasan, and J. Whaley. The Jalapeño virtual machine. IBM System Journal, 39(1), Feb. 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Apache. DRLVM -- Dynamic Runtime Layer Virtual Machine. http://harmony.apache.org/subcomponents/drlvm/.Google ScholarGoogle Scholar
  8. B. Bershad, S. M. Blackburn, H. Boehm, M. Cierniak, C. Click, D. Frampton, D. Gregg, D. Grove, X. Li, B. Mathiske, and G. Skinner. First Moxie brainstorming meeting, Dec. 2005. http://moxie.sf.net/.Google ScholarGoogle Scholar
  9. B. N. Bershad, C. Chambers, S. Eggers, C. Maeda, D. McNamee, P. Pardyak, S. Savage, and E. G. Sirer. SPIN: an extensible microkernel for application-specific operating system services. In EW 6: Proceedings of the 6th ACM SIGOPS European Workshop, pages 68--71, New York, NY, USA, 1994. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. B. N. Bershad, S. Savage, P. Pardyak, E. G. Sirer, M. E. Fiuczynski, D. Becker, C. Chambers, and S. Eggers. Extensibility, safety and performance in the SPIN operating system. In SOSP'95: Proceedings of the 15th ACM Symposium on Operating Systems Principles, pages 267--283, New York, NY, USA, 1995. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. S. M. Blackburn, P. Cheng, and K. S. McKinley. Oil and water? High performance garbage collection in Java with MMTk. In ICSE'04: Proceedings of the 26th International Conference on Software Engineering, pages 137--146, Washington, DC, USA, 2004. IEEE Computer Society. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. M. Blackburn, S. I. Salishev, M. Danilov, O. A. Mokhovikov, A. A. Nashatyrev, P. A. Novodvorsky, V. I. Bogdanov, X. F. Li, and D. Ushakov. The Moxie JVM experience. Technical Report TRCS-08-01, Australian National University, Department of Computer Science, Jan. 2008.Google ScholarGoogle Scholar
  13. H.-J. Boehm. Space efficient conservative garbage collection. In PLDI'93: Proceedings of the 1993 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 197--206, New York, NY, USA, 1993. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. H.-J. Boehm. Threads cannot be implemented as a library. In PLDI'05: Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 261--268, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. H.-J. Boehm and M. Weiser. Garbage collection in an uncooperative environment. Software: Practice and Experience, 18(9):807--820, 1988. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. L. Cardelli, J. Donahue, L. Glassman, I. Jordan, B. Kalsow, and G. Nelson. Modula-3 report (revised). Technical Report 52, DEC SRC, Nov. 1989.Google ScholarGoogle Scholar
  17. B. L. Chamberlain, D. Callahan, and H. P. Zima. Parallel programmability and the Chapel language. International Journal of High Performance Computing Applications, 21(3):291--312, Aug. 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. P. Charles, C. Grothoff, V. Saraswat, C. Donawa, A. Kielstra, K. Ebcioglu, C. von Praun, and V. Sarkar. X10: an object-oriented approach to non-uniform cluster computing. In OOPSLA'05: Proceedings of the 20th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 519--538, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. M. Cierniak, M. Eng, N. Glew, B. Lewis, and J. Stichnoth. The Open Runtime Platform: a flexible high-performance managed runtime environment. Concurrency and Computation: Practice and Experience, 17(5--6):617--637, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. ECMA. C# Language Specification, ECMA-334. http://www.ecma-international.org/publications/standards/Ecma-334.htm, June 2006. (ISO/IEC 23270:2006).Google ScholarGoogle Scholar
  21. ECMA. Common Language Infrastructure (CLI), ECMA-335. http://www.ecma-international.org/publications/standards/Ecma-335.htm, June 2006. (ISO/IEC 23271:2006).Google ScholarGoogle Scholar
  22. M. Fähndrich, M. Aiken, C. Hawblitzel, O. Hodson, G. Hunt, J. R. Larus, and S. Levi. Language support for fast and reliable messagebased communication in Singularity OS. In EuroSys'06: Proceedings of the 1st ACM SIGOPS/EuroSys European Conference on Computer Systems 2006, pages 177--190, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. P. Ferrara, F. Logozzo, and M. Fähndrich. Safer unsafe code for .NET. In OOPSLA'08: Proceedings of the 23rd ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 329--346, New York, NY, USA, 2008. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. M. E. Fiuczynski, W. C. Hsieh, E. G. Sirer, P. Pardyak, and B. N. Bershad. Low-level systems programming with Modula-3. In Threads: A Modula-3 Newsletter, number 3, Fall 1997.Google ScholarGoogle Scholar
  25. C. Flack, T. Hosking, and J. Vitek. Idioms in OVM. Technical Report CSD-TR-03-017, Purdue University, 2003.Google ScholarGoogle Scholar
  26. J. G. Fletcher. No! High level languages should not be used to write systems software. In ACM 75: Proceedings of the 1975 Annual Conference, pages 209--211, New York, NY, USA, 1975. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. J. G. Fletcher, C. S. Badger, G. L. Boer, and G. G. Marshall. On the appropriate language for system programming. SIGPLAN Notices, 7(7):28--30, 1972. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. D. J. Frailey. Should high level languages be used to write systems software? In ACM 75: Proceedings of the 1975 Annual Conference, page 205, New York, NY, USA, 1975. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. R. Garner. JMTK: a portable memory management toolkit. Honours thesis, Australian National University, Dec. 2003.Google ScholarGoogle Scholar
  30. R. Garner, S. M. Blackburn, and D. Frampton. Effective prefetch for mark-sweep garbage collection. In ISMM'07: Proceedings of the 6th International Symposium on Memory Management, pages 43--54, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. D. Gay, R. Ennals, and E. Brewer. Safe manual memory management. In ISMM'07: Proceedings of the 6th International Symposium on Memory Management, pages 2--14, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. N. Glew, S. Triantafyllis, M. Clerniak, M. Eng, B. Lewis, and J. Stichnoth. LIL: an architecture-neutral language for virtualmachine stubs. In VM'04: Proceedings of the 3rd Virtual Machine Research & Technology Symposium, page 9, Berkeley, CA, USA, 2004. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. J. Gosling, B. Joy, G. Steel, and G. Bracha. The Java Language Specification, Third Edition. Prentice Hall, 3rd edition, June 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. D. Grossman. Type-safe multithreading in Cyclone. In TLDI'03: Proceedings of the 2003 ACM SIGPLAN International Workshop on Types in Languages Design and Implementation, pages 13--25, New York, NY, USA, 2003. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. T. Hallgren, M. P. Jones, R. Leslie, and A. Tolmach. A principled approach to operating system construction in Haskell. In ICFP'05: Proceedings of the 10th ACM SIGPLAN International Conference on Functional Programming, pages 116--128, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. T. Hart and M. Levin. The new compiler. Technical report, Massachusetts Institute of Technology, Cambridge, MA, USA, 1962. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. M. Hirzel and R. Grimm. Jeannie: granting Java Native Interface developers their wishes. In OOPSLA'07: Proceedings of the 22nd ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 19--38, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. J. J. Horning. Yes! High level languages should be used to write systems software. In ACM 75: Proceedings of the 1975 Annual Conference, pages 206--208, New York, NY, USA, 1975. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. S. S. Huang, A. Hormati, D. F. Bacon, and R. Rabbah. Liquid Metal: object-oriented programming across the hardware/software boundary. In ECOOP'08: Proceedings of the 22nd European Conference on Object-Oriented Programming, pages 76--103, Berlin, Heidelberg, 2008. Springer-Verlag. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. G. Hunt, J. Larus, M. Abadi, M. Aiken, P. Barham, M. Fähndrich, C. Hawblitzel, O. Hodson, S. Levi, N. Murphy, B. Steensgaard, D. Tarditi, T. Wobber, and B. Zill. An overview of the Singularity project. Technical Report MSR-TR-2005-135, Microsoft Research, 2005.Google ScholarGoogle Scholar
  41. G. C. Hunt and J. R. Larus. Singularity: rethinking the software stack. SIGOPS Operating Systems Review, 41(2):37--49, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. T. Jim, J. G. Morrisett, D. Grossman, M. W. Hicks, J. Cheney, and Y. Wang. Cyclone: a safe dialect of C. In ATEC'02: Proceedings of the General Track of the USENIX Annual Technical Conference, pages 275--288, Berkeley, CA, USA, 2002. USENIX Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. K. Kennedy. Generation of high performance domain-specific languages from component libraries. http://www.cs.rice.edu/~ken/Presentations/TelescopeOSU.pdf.Google ScholarGoogle Scholar
  44. B. W. Kernighan and D. M. Ritchie. The C Programming Language. Prentice Hall, Upper Saddle River, NJ, USA, 1988. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. S. Kuksenko. Suggestion: Let's write some small and hot native (kernel) methods on vmmagics. http://www.mail-archive.com/[email protected]/msg07606.html, May 2007.Google ScholarGoogle Scholar
  46. D. Lea. Low-level memory fences. http://gee.cs.oswego.edu/dl/concurrent/dist/docs/java/util/concurrent/atomic/Fences.html.Google ScholarGoogle Scholar
  47. S. Liang. The Java Native Interface: Programmer's Guide and Specification. The Java Series. Prentice Hall, June 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. J.-W. Maessen, V. Sarkar, and D. Grove. Program analysis for safety guarantees in a Java virtual machine written in Java. In PASTE'01: Proceedings of the 2001 ACM SIGPLAN-SIGSOFT Workshop on Program Analysis for Software Tools and Engineering, pages 62--65, New York, NY, USA, 2001. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. S. Microsystems. Maxine Research Project. http://research.sun.com/projects/maxine.Google ScholarGoogle Scholar
  50. J. E. B. Moss, T. Palmer, T. Richards, I. Edward K. Walters, and C. C. Weems. CISL: a class-based machine description language for co-generation of compilers and simulators. International Journal of Parallel Programming, 33(2):231--246, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. N. Nethercote and J. Seward. Valgrind: a framework for heavyweight dynamic binary instrumentation. In PLDI'07: Proceedings of the 2007 ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 89--100, New York, NY, USA, 2007. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. E. Prangsma. Why Java is practical for modern operating systems. In Libre Software Meeting, 2005. Presentation only. See www.jnode.org.Google ScholarGoogle Scholar
  53. M. Richards. BCPL: a tool for compiler writing and system programming. In AFIPS'69 (Spring): Proceedings of the May 14-16, 1969, Spring Joint Computer Conference, pages 557--566, New York, NY, USA, 1969. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. A. Rigo and S. Pedroni. PyPy's approach to virtual machine construction. In OOPSLA Companion'06: Companion to the 21st ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 944--953, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. D. M. Ritchie. The development of the C language. In HOPL-II: The second ACM SIGPLAN Conference on the History of Programming Languages, pages 201--208, New York, NY, USA, 1993. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. S. Ritchie. Systems programming in Java. IEEE Micro, 17(3):30--35, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. J. Shapiro. Programming language challenges in systems codes: Why systems programmers still use C, and what to do about it. In PLOS'06: Proceedings of the 3rd Workshop on Programming Languages and Operating Systems, page 9, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. D. Simon, C. Cifuentes, D. Cleal, J. Daniels, and D. White. Java on the bare metal of wireless sensor devices: the Squawk Java virtual machine. In VEE'06: Proceedings of the 2nd International Conference on Virtual Execution Environments, pages 78--88, New York, NY, USA, 2006. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. L. Stepanian, A. D. Brown, A. Kielstra, G. Koblents, and K. Stoodley. Inlining Java native calls at runtime. In VEE'05: Proceedings of the 1st ACM/USENIX International Conference on Virtual Execution Environments, pages 121--131, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. B. Stroustrup. The C++ Programming Language. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1986. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. B. Stroustrup. A history of C++: 1979-1991. In HOPL-II: The second ACM SIGPLAN Conference on the History of Programming Languages, pages 271--297, New York, NY, USA, 1993. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. D. Stutz, T. Neward, and G. Shilling. Shared Source CLI Essentials. O'Reilly, 2003. A. Tridgell. Using talloc in Samba4. Technical report, Samba Team, 2004. http://samba.org/ftp/unpacked/talloc/talloc_guide.txt. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. D. Ungar, A. Spitz, and A. Ausch. Constructing a metacircular virtual machine in an exploratory programming environment. In OOPSLA Companion'05: Companion to the 20th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, pages 11--20, New York, NY, USA, 2005. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. K. Venstermans, L. Eeckhout, and K. De Bosschere. 64-bit versus 32-bit virtual machines for Java. Software: Practice and Experience, 36(1):1--26, 1 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. J. R. Vernooij. SAMBA developers guide. http://www.samba.org/samba/docs/Samba-Developers-Guide.pdf, Apr. 2008.Google ScholarGoogle Scholar
  66. J. Whaley. Joeq: a virtual machine and compiler infrastructure. In IVME'03: Proceedings of the 2003 Workshop on Interpreters, Virtual Machines and Emulators, pages 58--66, New York, NY, USA, 2003. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. W. A.Wulf, D. B. Russell, and A. N. Habermann. BLISS: a language for systems programming. Communcations of the ACM, 14(12):780--790, 1971. Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. H. Yamauchi and M.Wolczko. Writing Solaris device drivers in Java. In PLOS'06: Proceedings of the 3rd Workshop on Programming Languages and Operating Systems, page 3, New York, NY, USA, Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Demystifying magic: high-level low-level programming

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      Get this Publication

      • Published in

        cover image ACM Conferences
        VEE '09: Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments
        March 2009
        148 pages
        ISBN:9781605583754
        DOI:10.1145/1508293

        Copyright © 2009 ACM

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 11 March 2009

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Author Tags

        Qualifiers

        • research-article

        Acceptance Rates

        Overall Acceptance Rate 18 of 57 submissions, 32%

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      View Table of Contents
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!