skip to main content
research-article

Predictable Shared Cache Management for Multi-Core Real-Time Virtualization

Published:06 December 2017Publication History
Skip Abstract Section

Abstract

Real-time virtualization has gained much attention for the consolidation of multiple real-time systems onto a single hardware platform while ensuring timing predictability. However, a shared last-level cache (LLC) on modern multi-core platforms can easily hamper the timing predictability of real-time virtualization due to the resulting temporal interference among consolidated workloads. Since such interference caused by the LLC is highly variable and may have not even existed in legacy systems to be consolidated, it poses a significant challenge for real-time virtualization. In this article, we propose a predictable shared cache management framework for multi-core real-time virtualization. Our framework introduces two hypervisor-level techniques, vLLC and vColoring, that enable the cache allocation of individual tasks running in a virtual machine (VM), which is not achievable by the current state of the art. Our framework also provides a cache management scheme that determines cache allocation to tasks, designs VMs in a cache-aware manner, and minimizes the aggregated utilization of VMs to be consolidated. As a proof of concept, we implemented vLLC and vColoring in the KVM hypervisor running on x86 and ARM multi-core platforms. Experimental results with three different guest OSs (i.e., Linux/RK, vanilla Linux, and MS Windows Embedded) show that our techniques can effectively control the cache allocation of tasks in VMs. Our cache management scheme yields a significant utilization benefit compared to other approaches while satisfying timing constraints.

References

  1. Sebastian Altmeyer, Roeland Douma, Will Lunniss, and Robert I. Davis. 2014. Evaluation of cache partitioning for hard real-time systems. In Euromicro Conference on Real-Time Systems (ECRTS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. ARM. 2016. ARM Cortex-A15 Reference Manual. (2016). Retrieved Dec. 14, 2016 from http://arm.com.Google ScholarGoogle Scholar
  3. Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the art of virtualization. ACM Operating Systems Review 37, 5 (2003), 164--177. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Swagato Basumallick and Kelvin Nilsen. 1994. Cache issues in real-time systems. In ACM Workshop on Language, Compiler, and Tools for Real-Time Systems.Google ScholarGoogle Scholar
  5. Fabrice Bellard. 2005. QEMU, a fast and portable dynamic translator. In USENIX Annual Technical Conference (USENIX ATC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Guillem Bernat and Alan Burns. 1999. New results on fixed priority aperiodic servers. In IEEE Real-Time Systems Symposium (RTSS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Ravi Bhargava, Benjamin Serebrin, Francesco Spadini, and Srilatha Manne. 2008. Accelerating two-dimensional page walks for virtualized systems. ACM Operating Systems Review 42, 2 (2008), 26--35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC benchmark suite: Characterization and architectural implications. In International Conference on Parallel Architectures and Compilation Techniques (PACT). Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Bach D. Bui, Marco Caccamo, Lui Sha, and Joseph Martinez. 2008. Impact of cache partitioning on multi-tasking real time embedded systems. In IEEE Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. José V. Busquets-Mataix, Juan José Serrano, and Andy Wellings. 1997. Hybrid instruction cache partitioning for preemptive real-time systems. In Euromicro Workshop on Real-Time Systems (ECRTS).Google ScholarGoogle ScholarCross RefCross Ref
  11. Tommaso Cucinotta, Gaetano Anastasi, and Luca Abeni. 2009. Respecting temporal constraints in virtualised services. In IEEE International Computer Software and Applications Conference (COMPSAC). Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Robert I. Davis and Alan Burns. 2005. Hierarchical fixed priority pre-emptive scheduling. In IEEE Real-Time Systems Symposium (RTSS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. General Dynamics. 2016. OKL4 Microvisor. (2016). Retrieved Dec. 14, 2016 from https://gdmissionsystems.com/cyber/products/trusted-computing-cross-domain/microvisor-products/.Google ScholarGoogle Scholar
  14. Per Hammarlund. 2013. 4th generation Intel core processor, codenamed haswell. In Hot Chips: A Symposium on High Performance Chips (HC25).Google ScholarGoogle ScholarCross RefCross Ref
  15. Hardkernel. 2016. ODROID. (2016). Retrieved Dec. 19, 2016 from http://www.hardkernel.com.Google ScholarGoogle Scholar
  16. Intel. 2016. Intel 64 and IA-32 Developer’s Manual. (2016). Retrieved Dec. 14, 2016 from http://intel.com.Google ScholarGoogle Scholar
  17. Xinxin Jin, Haogang Chen, Xiaolin Wang, Zhenlin Wang, Xiang Wen, Yingwei Luo, and Xiaoming Li. 2009. A simple cache partitioning approach in a virtualized environment. In IEEE Symposium on Parallel and Distributed Processing with Applications.Google ScholarGoogle ScholarCross RefCross Ref
  18. Mathai Joseph and Paritosh K. Pandya. 1986. Finding response times in a real-time system. Computer Journal 29, 5 (1986), 390--395.Google ScholarGoogle ScholarCross RefCross Ref
  19. Daehoon Kim, Hwanju Kim, and Jaehyuk Huh. 2014. vCache: Providing a transparent view of the LLC in virtualized environments. Computer Architecture Letters 13, 2 (2014), 109--112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Hyoseung Kim, Dionisio de Niz, Björn Andersson, Mark Klein, Onur Mutlu, and Ragunathan Rajkumar. 2016. Bounding and reducing memory interference in COTS-based multi-core systems. Real-Time Systems 52, 3 (2016), 356--395. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Hyoseung Kim, Arvind Kandhalu, and Ragunathan (Raj) Rajkumar. 2013. A coordinated approach for practical OS-level cache management in multi-core real-time systems. In Euromicro Conference on Real-Time Systems (ECRTS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Hyoseung Kim and Ragunathan Rajkumar. 2012. Shared-page management for improving the temporal isolation of memory reservations in resource kernels. In IEEE Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Hyoseung Kim and Ragunathan Rajkumar. 2014. Memory reservation and shared page management for real-time systems. Journal of Systems Architecture (JSA) 60, 2 (2014), 165--178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Hyoseung Kim and Ragunathan Rajkumar. 2016. Real-time cache management for multi-core virtualization. In ACM International Conference on Embedded Software (EMSOFT). Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Hyoseung Kim, Shige Wang, and Ragunathan Rajkumar. 2014. vMPCP: A synchronization framework for multi-core virtual machines. In IEEE Real-Time Systems Symposium (RTSS).Google ScholarGoogle ScholarCross RefCross Ref
  26. Hyoseung Kim, Shige Wang, and Ragunathan Rajkumar. 2015. Responsive and enforced interrupt handling for real-time system virtualization. In IEEE Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Jan Kiszka. 2009. Towards Linux as a real-time hypervisor. In Real-Time Linux Workshop (RTLWS).Google ScholarGoogle Scholar
  28. Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. 2007. KVM: The Linux virtual machine monitor. In Linux Symposium, Vol. 1. 225--230.Google ScholarGoogle Scholar
  29. Karthik Lakshmanan, Dionisio de Niz, and Ragunathan Rajkumar. 2009. Coordinated task scheduling, allocation and synchronization on multiprocessors. In IEEE Real-Time Systems Symposium (RTSS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Oded Lempel. 2011. 2nd generation Intel core processor family: Intel core i7, i5 and i3. In Hot Chips: A Symposium on High Performance Chips (HC23).Google ScholarGoogle ScholarCross RefCross Ref
  31. Hennadiy Leontyev and James H. Anderson. 2009. A hierarchical multiprocessor bandwidth reservation scheme with timing guarantees. Real-Time Systems 43, 1 (2009), 60--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Ye Li, Richard West, and Eric Missimer. 2014. A virtualized separation kernel for mixed criticality systems. In ACM Conference on Virtual Execution Environments (VEE). Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Jochen Liedtke, Hermann Hartig, and Michael Hohmuth. 1997. OS-Controlled cache predictability for real-time systems. In IEEE Real-Time Technology and Applications Symposium (RTAS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Jiang Lin, Qingda Lu, Xiaoning Ding, Zhao Zhang, Xiaodong Zhang, and P. Sadayappan. 2008. Gaining insights into multicore cache partitioning: Bridging the gap between simulation and real systems. In IEEE Symposium on High-Performance Computer Architecture (HPCA).Google ScholarGoogle Scholar
  35. C. L. Liu and James W. Layland. 1973. Scheduling algorithms for multiprogramming in a hard-real-time environment. Journal of the ACM 20, 1 (1973), 46--61. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Will Lunniss, Sebastian Altmeyer, Giuseppe Lipari, and Robert I. Davis. 2014. Accounting for cache related pre-emption delays in hierarchical scheduling. In International Conference on Real-Time and Network Systems (RTNS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Ruhui Ma, Wei Ye, Alei Liang, Haibing Guan, and Jian Li. 2013. Cache isolation for virtualization of mixed general-purpose and real-time systems. Journal of Systems Architecture 59, 10 (2013), 1405--1413. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Renato Mancuso, Roman Dudko, Emiliano Betti, Marco Cesati, Marco Caccamo, and Rodolfo Pellizzoni. 2013. Real-time cache management framework for multi-core architectures. In IEEE Real-Time Technology and Applications Symposium (RTAS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Shuichi Oikawa and Ragunathan Rajkumar. 1998. Linux/RK: A portable resource kernel in Linux. In IEEE Real-Time Systems Symposium (RTSS) Work-In-Progress.Google ScholarGoogle Scholar
  40. Marco Paolieri, Eduardo Quiñones, Francisco J Cazorla, Robert I. Davis, and Mateo Valero. 2011. IA: An interference aware allocation algorithm for multicore hard real-time systems. In IEEE Real-Time Technology and Applications Symposium (RTAS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. QNX. 2017. QNX Hypervisor. (2017). Retrieved Apr. 14, 2017 from http://www.qnx.com.Google ScholarGoogle Scholar
  42. Wind River. 2016. VxWorks. (2016). Retrieved Dec. 14, 2016 from http://www.windriver.com.Google ScholarGoogle Scholar
  43. Saowanee Saewong, Mark H. Klein, Ragunathan Rajkumar, and John P. Lehoczky. 2002. Analysis of hierarchical fixed-priority scheduling. In Euromicro Conference on Real-Time Systems (ECRTS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Filip Sebek. 2001. Cache memories and real-time systems. Technical Report. Mälardalen University. Retrieved Dec. 14, 2016 from http://www.es.mdh.se/publications/290-.Google ScholarGoogle Scholar
  45. Lui Sha, John P. Lehoczky, and Ragunathan Rajkumar. 1986. Solutions for some practical problems in prioritized preemptive scheduling. In IEEE Real-Time Systems Symposium (RTSS).Google ScholarGoogle Scholar
  46. Jicheng Shi, Xiang Song, Haibo Chen, and Binyu Zang. 2011. Limiting cache-based side-channel in multi-tenant cloud using dynamic page coloring. In Dependable Systems and Networks Workshops (DSN-W). Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Insik Shin, Arvind Easwaran, and Insup Lee. 2008. Hierarchical scheduling framework for virtual clustering of multiprocessors. In Euromicro Conference on Real-Time Systems (ECRTS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Insik Shin and Insup Lee. 2008. Compositional real-time scheduling framework with periodic model. ACM Transactions on Embedded Computing Systems (TECS) 7, 3 (2008), 30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Brinkley Sprunt, Lui Sha, and John Lehoczky. 1989. Aperiodic task scheduling for hard-real-time systems. Real-Time Systems 1, 1 (1989), 27--60.Google ScholarGoogle ScholarCross RefCross Ref
  50. Udo Steinberg and Bernhard Kauer. 2010. NOVA: A microhypervisor-based secure virtualization architecture. In ACM European Conference on Computer Systems (EuroSys). Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Jay K. Strosnider, John P. Lehoczky, and Lui Sha. 1995. The deferrable server algorithm for enhanced aperiodic responsiveness in hard real-time environments. IEEE Transactions on Computing 44, 1 (1995), 73--91. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Noriaki Suzuki, Hyoseung Kim, Dionisio de Niz, Bjorn Andersson, Lutz Wrage, Mark Klein, and Ragunathan Rajkumar. 2013. Coordinated bank and cache coloring for temporal protection of memory accesses. In IEEE International Conference on Embedded Software and Systems (ICESS).Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Dominique Thiebaut, Joel L. Wolf, and Harold S. Stone. 1992. Synthetic traces for trace-driven simulation of cache memories. IEEE Transactions on Computers 41, 4 (1992). Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. Xiaolin Wang, Jiarui Zang, Zhenlin Wang, Yingwei Luo, and Xiaoming Li. 2011. Selective hardware/software memory virtualization. ACM SIGPLAN Notices 46, 7 (2011), 217--226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Bryan C. Ward, Jonathan L. Herman, Christopher J. Kenna, and James H. Anderson. 2013. Making shared caches more predictable on multicore platforms. In Euromicro Conference on Real-Time Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. Sisu Xi, Justin Wilson, Chenyang Lu, and Christopher Gill. 2011. RT-Xen: Towards real-time hypervisor scheduling in Xen. In ACM International Conference on Embedded Software (EMSOFT). Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Sisu Xi, Meng Xu, Chenyang Lu, Linh TX Phan, Christopher Gill, Oleg Sokolsky, and Insup Lee. 2014. Real-time multi-core virtual machine scheduling in xen. In ACM International Conference on Embedded Software (EMSOFT). Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Meng Xu, Linh TX Phan, Hyon-Young Choi, and Insup Lee. 2017. vCAT: Dynamic cache management using CAT virtualization. In IEEE Real-Time Technology and Applications Symposium (RTAS).Google ScholarGoogle ScholarCross RefCross Ref
  59. Meng Xu, Linh TX Phan, Insup Lee, Oleg Sokolsky, Sisu Xi, Chenyang Lu, and Christopher Gill. 2013. Cache-aware compositional analysis of real-time multicore virtualization platforms. In IEEE Real-Time Systems Symposium (RTSS). Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Jungwoo Yang, Hyungseok Kim, Sangwon Park, Changki Hong, and Insik Shin. 2011. Implementation of compositional scheduling framework on virtualization. ACM SIGBED Review 8, 1 (2011), 30--37. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Ying Ye, Richard West, Zhuoqun Cheng, and Ye Li. 2014. COLORIS: A dynamic cache partitioning system using page coloring. In International Conference on Parallel Architectures and Compilation Techniques. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Heechul Yun, Renato Mancuso, Zheng-Pei Wu, and Rodolfo Pellizzoni. 2014. PALLOC: DRAM bank-aware memory allocator for performance isolation on multicore platforms. In IEEE Real-Time Technology and Applications Symposium (RTAS).Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Predictable Shared Cache Management for Multi-Core Real-Time Virtualization

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!