Abstract
Embedded devices are ubiquitous in areas of industrial and environmental monitoring, health and safety, and consumer appliances. A common use case is data collection, processing, and performing actions based on data analysis. Although many Internet of Things (IoT) applications use the embedded device simply for data collection, there are benefits to having more data processing done closer to data collection to reduce network transmissions and power usage and provide faster response. This work implements and evaluates algorithms for sorting data on embedded devices with specific focus on the smallest memory devices. In devices with less than 4 KB of available RAM, the standard external merge sort algorithm has limited application as it requires a minimum of three memory buffers and is not flash-aware. The contribution is a memory-optimized external sorting algorithm called no output buffer sort (NOBsort) that reduces the minimum memory required for sorting, has excellent performance for sorted or near-sorted data, and sorts on external memory such as SD cards or raw flash chips. When sorting large datasets, no output buffer sort reduces I/O and execution time by between 20% to 35% compared to standard external merge sort.
- 2020. Arduino Homepage. http://arduino.cc.Google Scholar
- 2020. Microchip AT89LP3240 Specifications. https://www.microchip.com/wwwproducts/en/AT89LP3240.Google Scholar
- 2020. Microchip ATmega2560 Specifications. https://www.microchip.com/wwwproducts/en/ATmega2560.Google Scholar
- 2020. Microchip ATSamd21 Specifications. https://www.microchip.com/wwwproducts/en/ATsamd21g18.Google Scholar
- 2020. Microchip ATtiny1634 Specifications. https://www.microchip.com/wwwproducts/en/ATtiny1634.Google Scholar
- 2020. Microchip ATtiny48 Specifications. https://www.microchip.com/wwwproducts/en/ATtiny48.Google Scholar
- 2020. Microchip PIC Specifications. https://www.microchip.com/wwwproducts/en/PIC18F57Q43.Google Scholar
- 2020. NXP LPC8N04 Specifications. https://www.nxp.com/docs/en/data-sheet/LPC8N04.pdf.Google Scholar
- 2020. Texas Instruments MSP430FR6007 Specifications. https://www.ti.com/product/MSP430FR6007.Google Scholar
- Panayiotis Andreou, Orestis Spanos, Demetrios Zeinalipour-Yazti, George Samaras, and Panos K. Chrysanthis. 2009. FSort: External sorting on flash-based sensor devices. In Data Management for Sensor Networks (DMSN’09). 1--6.Google Scholar
- Tyler Cossentine and Ramon Lawrence. 2010. Fast sorting on flash memory sensor nodes. In Proceedings of the 14th International Database Engineering and Applications Symposium (IDEAS’10). ACM, New York, NY, 105--113. DOI:https://doi.org/10.1145/1866480.1866496Google Scholar
Digital Library
- Tyler Cossentine and Ramon Lawrence. 2013. Efficient external sorting on flash memory embedded devices. International Journal of Database Management Systems 5 (2013), 1--20. DOI:https://doi.org/10.5121/ijdms.2013.5101Google Scholar
Cross Ref
- Martin A. Goetz. 1963. Internal and tape sorting using the replacement-selection technique. Commun. ACM 6, 5 (1963), 201--206. DOI:https://doi.org/10.1145/366552.366556Google Scholar
Digital Library
- Goetz Graefe. 2006. Implementing sorting in database systems. ACM Comput. Surv. 38, 3, Article 10 (Sept. 2006). DOI:https://doi.org/10.1145/1132960.1132964Google Scholar
Digital Library
- Hazar Harmouch and Felix Naumann. 2017. Cardinality estimation: An experimental survey. Proc. VLDB Endow. 11, 4 (Dec. 2017), 499--512. DOI:https://doi.org/10.1145/3186728.3164145Google Scholar
Digital Library
- Riley Jackson and Ramon Lawrence. 2019. Faster sorting for flash memory embedded devices. In 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE’19). IEEE, 1--5. DOI:https://doi.org/10.1109/CCECE.2019.8861811Google Scholar
Cross Ref
- A. Laga, J. Boukhobza, F. Singhoff, and M. Koskas. 2017. MONTRES: Merge on-the-run external sorting algorithm for large data volumes on SSD based storage systems. IEEE Trans. Comput. 66, 10 (Oct 2017), 1689--1702. DOI:https://doi.org/10.1109/TC.2017.2706678Google Scholar
Digital Library
- J. Lee, H. Roh, and S. Park. 2016. External mergesort for flash-based solid state drives. IEEE Trans. Comput. 65, 5 (May 2016), 1518--1527. DOI:https://doi.org/10.1109/TC.2015.2451631Google Scholar
Digital Library
- Yang Liu, Zhen He, Yi-Ping Phoebe Chen, and Thi Nguyen. 2011. External sorting on flash memory via natural page run generation. Comput. J. 54, 11 (2011), 1882--1990. DOI:https://doi.org/10.1093/comjnl/bxr051Google Scholar
Digital Library
- Samuel R. Madden, Michael J. Franklin, Joseph M. Hellerstein, and Wei Hong. 2005. TinyDB: An acquisitional query processing system for sensor networks. ACM Trans. Database Syst. 30, 1 (March 2005), 122--173. DOI:https://doi.org/10.1145/1061318.1061322Google Scholar
Digital Library
- Hyoungmin Park and Kyuseok Shim. 2009. FAST: Flash-aware external sorting for mobile database systems. Journal of Systems and Software 82, 8 (2009), 1298--1312. DOI:https://doi.org/DOI: 10.1016/j.jss.2009.02.028Google Scholar
Digital Library
- W. Penson, S. Fazackerley, and R. Lawrence. 2016. TEFS: A flash file system for use on memory constrained devices. In 2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE’16). 1--5. DOI:https://doi.org/10.1109/CCECE.2016.7726822Google Scholar
Cross Ref
- Chin-Hsien Wu and Kuo-Yi Huang. 2015. Data sorting in flash memory. Trans. Storage 11, 2, Article 7 (March 2015), 25 pages. DOI:https://doi.org/10.1145/2665067Google Scholar
Digital Library
Index Terms
Efficient External Sorting for Memory-Constrained Embedded Devices with Flash Memory
Recommendations
Adaptive flash sorting for memory-constrained embedded devices
SAC '21: Proceedings of the 36th Annual ACM Symposium on Applied ComputingDatabases running on embedded devices require efficient sorting algorithms for aggregation, joins, and output ordering. Previous research has produced algorithms optimized for small-memory, flash embedded devices that either use multiple read passes to ...
RAM-Efficient External Memory Sorting
In recent years a large number of problems have been considered in external memory models of computation, where the complexity measure is the number of blocks of data that are moved between slow external memory and fast internal memory (also called I/Os)...
Efficient Sorting and Join on NVM-Based Hybrid Memory
Algorithms and Architectures for Parallel ProcessingAbstractNon-volatile memory (NVM) as a new kind of future memories has a number of special properties such as non-volatility, read/write asymmetry, and byte addressability. This makes it difficult to directly replace DRAM with NVM in current memory ...






Comments