Abstract
Real-time virtualization has gained much attention for the consolidation of multiple real-time systems onto a single hardware platform while ensuring timing predictability. However, a shared last-level cache (LLC) on modern multi-core platforms can easily hamper the timing predictability of real-time virtualization due to the resulting temporal interference among consolidated workloads. Since such interference caused by the LLC is highly variable and may have not even existed in legacy systems to be consolidated, it poses a significant challenge for real-time virtualization. In this article, we propose a predictable shared cache management framework for multi-core real-time virtualization. Our framework introduces two hypervisor-level techniques, vLLC and vColoring, that enable the cache allocation of individual tasks running in a virtual machine (VM), which is not achievable by the current state of the art. Our framework also provides a cache management scheme that determines cache allocation to tasks, designs VMs in a cache-aware manner, and minimizes the aggregated utilization of VMs to be consolidated. As a proof of concept, we implemented vLLC and vColoring in the KVM hypervisor running on x86 and ARM multi-core platforms. Experimental results with three different guest OSs (i.e., Linux/RK, vanilla Linux, and MS Windows Embedded) show that our techniques can effectively control the cache allocation of tasks in VMs. Our cache management scheme yields a significant utilization benefit compared to other approaches while satisfying timing constraints.
- Sebastian Altmeyer, Roeland Douma, Will Lunniss, and Robert I. Davis. 2014. Evaluation of cache partitioning for hard real-time systems. In Euromicro Conference on Real-Time Systems (ECRTS). Google Scholar
Digital Library
- ARM. 2016. ARM Cortex-A15 Reference Manual. (2016). Retrieved Dec. 14, 2016 from http://arm.com.Google Scholar
- Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. 2003. Xen and the art of virtualization. ACM Operating Systems Review 37, 5 (2003), 164--177. Google Scholar
Digital Library
- Swagato Basumallick and Kelvin Nilsen. 1994. Cache issues in real-time systems. In ACM Workshop on Language, Compiler, and Tools for Real-Time Systems.Google Scholar
- Fabrice Bellard. 2005. QEMU, a fast and portable dynamic translator. In USENIX Annual Technical Conference (USENIX ATC). Google Scholar
Digital Library
- Guillem Bernat and Alan Burns. 1999. New results on fixed priority aperiodic servers. In IEEE Real-Time Systems Symposium (RTSS). Google Scholar
Digital Library
- Ravi Bhargava, Benjamin Serebrin, Francesco Spadini, and Srilatha Manne. 2008. Accelerating two-dimensional page walks for virtualized systems. ACM Operating Systems Review 42, 2 (2008), 26--35. Google Scholar
Digital Library
- Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008. The PARSEC benchmark suite: Characterization and architectural implications. In International Conference on Parallel Architectures and Compilation Techniques (PACT). Google Scholar
Digital Library
- Bach D. Bui, Marco Caccamo, Lui Sha, and Joseph Martinez. 2008. Impact of cache partitioning on multi-tasking real time embedded systems. In IEEE Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). Google Scholar
Digital Library
- José V. Busquets-Mataix, Juan José Serrano, and Andy Wellings. 1997. Hybrid instruction cache partitioning for preemptive real-time systems. In Euromicro Workshop on Real-Time Systems (ECRTS).Google Scholar
Cross Ref
- Tommaso Cucinotta, Gaetano Anastasi, and Luca Abeni. 2009. Respecting temporal constraints in virtualised services. In IEEE International Computer Software and Applications Conference (COMPSAC). Google Scholar
Digital Library
- Robert I. Davis and Alan Burns. 2005. Hierarchical fixed priority pre-emptive scheduling. In IEEE Real-Time Systems Symposium (RTSS). Google Scholar
Digital Library
- General Dynamics. 2016. OKL4 Microvisor. (2016). Retrieved Dec. 14, 2016 from https://gdmissionsystems.com/cyber/products/trusted-computing-cross-domain/microvisor-products/.Google Scholar
- Per Hammarlund. 2013. 4th generation Intel core processor, codenamed haswell. In Hot Chips: A Symposium on High Performance Chips (HC25).Google Scholar
Cross Ref
- Hardkernel. 2016. ODROID. (2016). Retrieved Dec. 19, 2016 from http://www.hardkernel.com.Google Scholar
- Intel. 2016. Intel 64 and IA-32 Developer’s Manual. (2016). Retrieved Dec. 14, 2016 from http://intel.com.Google Scholar
- Xinxin Jin, Haogang Chen, Xiaolin Wang, Zhenlin Wang, Xiang Wen, Yingwei Luo, and Xiaoming Li. 2009. A simple cache partitioning approach in a virtualized environment. In IEEE Symposium on Parallel and Distributed Processing with Applications.Google Scholar
Cross Ref
- Mathai Joseph and Paritosh K. Pandya. 1986. Finding response times in a real-time system. Computer Journal 29, 5 (1986), 390--395.Google Scholar
Cross Ref
- Daehoon Kim, Hwanju Kim, and Jaehyuk Huh. 2014. vCache: Providing a transparent view of the LLC in virtualized environments. Computer Architecture Letters 13, 2 (2014), 109--112. Google Scholar
Digital Library
- Hyoseung Kim, Dionisio de Niz, Björn Andersson, Mark Klein, Onur Mutlu, and Ragunathan Rajkumar. 2016. Bounding and reducing memory interference in COTS-based multi-core systems. Real-Time Systems 52, 3 (2016), 356--395. Google Scholar
Digital Library
- Hyoseung Kim, Arvind Kandhalu, and Ragunathan (Raj) Rajkumar. 2013. A coordinated approach for practical OS-level cache management in multi-core real-time systems. In Euromicro Conference on Real-Time Systems (ECRTS). Google Scholar
Digital Library
- Hyoseung Kim and Ragunathan Rajkumar. 2012. Shared-page management for improving the temporal isolation of memory reservations in resource kernels. In IEEE Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). Google Scholar
Digital Library
- Hyoseung Kim and Ragunathan Rajkumar. 2014. Memory reservation and shared page management for real-time systems. Journal of Systems Architecture (JSA) 60, 2 (2014), 165--178. Google Scholar
Digital Library
- Hyoseung Kim and Ragunathan Rajkumar. 2016. Real-time cache management for multi-core virtualization. In ACM International Conference on Embedded Software (EMSOFT). Google Scholar
Digital Library
- Hyoseung Kim, Shige Wang, and Ragunathan Rajkumar. 2014. vMPCP: A synchronization framework for multi-core virtual machines. In IEEE Real-Time Systems Symposium (RTSS).Google Scholar
Cross Ref
- Hyoseung Kim, Shige Wang, and Ragunathan Rajkumar. 2015. Responsive and enforced interrupt handling for real-time system virtualization. In IEEE Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA). Google Scholar
Digital Library
- Jan Kiszka. 2009. Towards Linux as a real-time hypervisor. In Real-Time Linux Workshop (RTLWS).Google Scholar
- Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. 2007. KVM: The Linux virtual machine monitor. In Linux Symposium, Vol. 1. 225--230.Google Scholar
- Karthik Lakshmanan, Dionisio de Niz, and Ragunathan Rajkumar. 2009. Coordinated task scheduling, allocation and synchronization on multiprocessors. In IEEE Real-Time Systems Symposium (RTSS). Google Scholar
Digital Library
- Oded Lempel. 2011. 2nd generation Intel core processor family: Intel core i7, i5 and i3. In Hot Chips: A Symposium on High Performance Chips (HC23).Google Scholar
Cross Ref
- Hennadiy Leontyev and James H. Anderson. 2009. A hierarchical multiprocessor bandwidth reservation scheme with timing guarantees. Real-Time Systems 43, 1 (2009), 60--92. Google Scholar
Digital Library
- Ye Li, Richard West, and Eric Missimer. 2014. A virtualized separation kernel for mixed criticality systems. In ACM Conference on Virtual Execution Environments (VEE). Google Scholar
Digital Library
- Jochen Liedtke, Hermann Hartig, and Michael Hohmuth. 1997. OS-Controlled cache predictability for real-time systems. In IEEE Real-Time Technology and Applications Symposium (RTAS). Google Scholar
Digital Library
- Jiang Lin, Qingda Lu, Xiaoning Ding, Zhao Zhang, Xiaodong Zhang, and P. Sadayappan. 2008. Gaining insights into multicore cache partitioning: Bridging the gap between simulation and real systems. In IEEE Symposium on High-Performance Computer Architecture (HPCA).Google Scholar
- C. L. Liu and James W. Layland. 1973. Scheduling algorithms for multiprogramming in a hard-real-time environment. Journal of the ACM 20, 1 (1973), 46--61. Google Scholar
Digital Library
- Will Lunniss, Sebastian Altmeyer, Giuseppe Lipari, and Robert I. Davis. 2014. Accounting for cache related pre-emption delays in hierarchical scheduling. In International Conference on Real-Time and Network Systems (RTNS). Google Scholar
Digital Library
- Ruhui Ma, Wei Ye, Alei Liang, Haibing Guan, and Jian Li. 2013. Cache isolation for virtualization of mixed general-purpose and real-time systems. Journal of Systems Architecture 59, 10 (2013), 1405--1413. Google Scholar
Digital Library
- Renato Mancuso, Roman Dudko, Emiliano Betti, Marco Cesati, Marco Caccamo, and Rodolfo Pellizzoni. 2013. Real-time cache management framework for multi-core architectures. In IEEE Real-Time Technology and Applications Symposium (RTAS). Google Scholar
Digital Library
- Shuichi Oikawa and Ragunathan Rajkumar. 1998. Linux/RK: A portable resource kernel in Linux. In IEEE Real-Time Systems Symposium (RTSS) Work-In-Progress.Google Scholar
- Marco Paolieri, Eduardo Quiñones, Francisco J Cazorla, Robert I. Davis, and Mateo Valero. 2011. IA: An interference aware allocation algorithm for multicore hard real-time systems. In IEEE Real-Time Technology and Applications Symposium (RTAS). Google Scholar
Digital Library
- QNX. 2017. QNX Hypervisor. (2017). Retrieved Apr. 14, 2017 from http://www.qnx.com.Google Scholar
- Wind River. 2016. VxWorks. (2016). Retrieved Dec. 14, 2016 from http://www.windriver.com.Google Scholar
- Saowanee Saewong, Mark H. Klein, Ragunathan Rajkumar, and John P. Lehoczky. 2002. Analysis of hierarchical fixed-priority scheduling. In Euromicro Conference on Real-Time Systems (ECRTS). Google Scholar
Digital Library
- Filip Sebek. 2001. Cache memories and real-time systems. Technical Report. Mälardalen University. Retrieved Dec. 14, 2016 from http://www.es.mdh.se/publications/290-.Google Scholar
- Lui Sha, John P. Lehoczky, and Ragunathan Rajkumar. 1986. Solutions for some practical problems in prioritized preemptive scheduling. In IEEE Real-Time Systems Symposium (RTSS).Google Scholar
- Jicheng Shi, Xiang Song, Haibo Chen, and Binyu Zang. 2011. Limiting cache-based side-channel in multi-tenant cloud using dynamic page coloring. In Dependable Systems and Networks Workshops (DSN-W). Google Scholar
Digital Library
- Insik Shin, Arvind Easwaran, and Insup Lee. 2008. Hierarchical scheduling framework for virtual clustering of multiprocessors. In Euromicro Conference on Real-Time Systems (ECRTS). Google Scholar
Digital Library
- Insik Shin and Insup Lee. 2008. Compositional real-time scheduling framework with periodic model. ACM Transactions on Embedded Computing Systems (TECS) 7, 3 (2008), 30. Google Scholar
Digital Library
- Brinkley Sprunt, Lui Sha, and John Lehoczky. 1989. Aperiodic task scheduling for hard-real-time systems. Real-Time Systems 1, 1 (1989), 27--60.Google Scholar
Cross Ref
- Udo Steinberg and Bernhard Kauer. 2010. NOVA: A microhypervisor-based secure virtualization architecture. In ACM European Conference on Computer Systems (EuroSys). Google Scholar
Digital Library
- Jay K. Strosnider, John P. Lehoczky, and Lui Sha. 1995. The deferrable server algorithm for enhanced aperiodic responsiveness in hard real-time environments. IEEE Transactions on Computing 44, 1 (1995), 73--91. Google Scholar
Digital Library
- Noriaki Suzuki, Hyoseung Kim, Dionisio de Niz, Bjorn Andersson, Lutz Wrage, Mark Klein, and Ragunathan Rajkumar. 2013. Coordinated bank and cache coloring for temporal protection of memory accesses. In IEEE International Conference on Embedded Software and Systems (ICESS).Google Scholar
Digital Library
- Dominique Thiebaut, Joel L. Wolf, and Harold S. Stone. 1992. Synthetic traces for trace-driven simulation of cache memories. IEEE Transactions on Computers 41, 4 (1992). Google Scholar
Digital Library
- Xiaolin Wang, Jiarui Zang, Zhenlin Wang, Yingwei Luo, and Xiaoming Li. 2011. Selective hardware/software memory virtualization. ACM SIGPLAN Notices 46, 7 (2011), 217--226. Google Scholar
Digital Library
- Bryan C. Ward, Jonathan L. Herman, Christopher J. Kenna, and James H. Anderson. 2013. Making shared caches more predictable on multicore platforms. In Euromicro Conference on Real-Time Systems. Google Scholar
Digital Library
- Sisu Xi, Justin Wilson, Chenyang Lu, and Christopher Gill. 2011. RT-Xen: Towards real-time hypervisor scheduling in Xen. In ACM International Conference on Embedded Software (EMSOFT). Google Scholar
Digital Library
- Sisu Xi, Meng Xu, Chenyang Lu, Linh TX Phan, Christopher Gill, Oleg Sokolsky, and Insup Lee. 2014. Real-time multi-core virtual machine scheduling in xen. In ACM International Conference on Embedded Software (EMSOFT). Google Scholar
Digital Library
- Meng Xu, Linh TX Phan, Hyon-Young Choi, and Insup Lee. 2017. vCAT: Dynamic cache management using CAT virtualization. In IEEE Real-Time Technology and Applications Symposium (RTAS).Google Scholar
Cross Ref
- Meng Xu, Linh TX Phan, Insup Lee, Oleg Sokolsky, Sisu Xi, Chenyang Lu, and Christopher Gill. 2013. Cache-aware compositional analysis of real-time multicore virtualization platforms. In IEEE Real-Time Systems Symposium (RTSS). Google Scholar
Digital Library
- Jungwoo Yang, Hyungseok Kim, Sangwon Park, Changki Hong, and Insik Shin. 2011. Implementation of compositional scheduling framework on virtualization. ACM SIGBED Review 8, 1 (2011), 30--37. Google Scholar
Digital Library
- Ying Ye, Richard West, Zhuoqun Cheng, and Ye Li. 2014. COLORIS: A dynamic cache partitioning system using page coloring. In International Conference on Parallel Architectures and Compilation Techniques. Google Scholar
Digital Library
- Heechul Yun, Renato Mancuso, Zheng-Pei Wu, and Rodolfo Pellizzoni. 2014. PALLOC: DRAM bank-aware memory allocator for performance isolation on multicore platforms. In IEEE Real-Time Technology and Applications Symposium (RTAS).Google Scholar
Cross Ref
Index Terms
Predictable Shared Cache Management for Multi-Core Real-Time Virtualization
Recommendations
Real-time cache management for multi-core virtualization
EMSOFT '16: Proceedings of the 13th International Conference on Embedded SoftwareReal-time virtualization techniques have been investigated with the primary goal of consolidating multiple real-time systems onto a single hardware platform while ensuring timing predictability. However, a shared last-level cache (LLC) on recent multi-...
Predictable task migration for locked caches in multi-core systems
LCTES '10Locking cache lines in hard real-time systems is a common means of achieving predictability of cache access behavior and tightening as well as reducing worst case execution time, especially in a multitasking environment. However, cache locking poses a ...
Push-assisted migration of real-time tasks in multi-core processors
LCTES '09Multicores are becoming ubiquitous, not only in general-purpose but also embedded computing. This trend is a reflexion of contemporary embedded applications posing steadily increasing demands in processing power. On such platforms, prediction of timing ...






Comments