skip to main content
research-article

Efficient External Sorting for Memory-Constrained Embedded Devices with Flash Memory

Published:26 March 2021Publication History
Skip Abstract Section

Abstract

Embedded devices are ubiquitous in areas of industrial and environmental monitoring, health and safety, and consumer appliances. A common use case is data collection, processing, and performing actions based on data analysis. Although many Internet of Things (IoT) applications use the embedded device simply for data collection, there are benefits to having more data processing done closer to data collection to reduce network transmissions and power usage and provide faster response. This work implements and evaluates algorithms for sorting data on embedded devices with specific focus on the smallest memory devices. In devices with less than 4 KB of available RAM, the standard external merge sort algorithm has limited application as it requires a minimum of three memory buffers and is not flash-aware. The contribution is a memory-optimized external sorting algorithm called no output buffer sort (NOBsort) that reduces the minimum memory required for sorting, has excellent performance for sorted or near-sorted data, and sorts on external memory such as SD cards or raw flash chips. When sorting large datasets, no output buffer sort reduces I/O and execution time by between 20% to 35% compared to standard external merge sort.

References

  1. 2020. Arduino Homepage. http://arduino.cc.Google ScholarGoogle Scholar
  2. 2020. Microchip AT89LP3240 Specifications. https://www.microchip.com/wwwproducts/en/AT89LP3240.Google ScholarGoogle Scholar
  3. 2020. Microchip ATmega2560 Specifications. https://www.microchip.com/wwwproducts/en/ATmega2560.Google ScholarGoogle Scholar
  4. 2020. Microchip ATSamd21 Specifications. https://www.microchip.com/wwwproducts/en/ATsamd21g18.Google ScholarGoogle Scholar
  5. 2020. Microchip ATtiny1634 Specifications. https://www.microchip.com/wwwproducts/en/ATtiny1634.Google ScholarGoogle Scholar
  6. 2020. Microchip ATtiny48 Specifications. https://www.microchip.com/wwwproducts/en/ATtiny48.Google ScholarGoogle Scholar
  7. 2020. Microchip PIC Specifications. https://www.microchip.com/wwwproducts/en/PIC18F57Q43.Google ScholarGoogle Scholar
  8. 2020. NXP LPC8N04 Specifications. https://www.nxp.com/docs/en/data-sheet/LPC8N04.pdf.Google ScholarGoogle Scholar
  9. 2020. Texas Instruments MSP430FR6007 Specifications. https://www.ti.com/product/MSP430FR6007.Google ScholarGoogle Scholar
  10. Panayiotis Andreou, Orestis Spanos, Demetrios Zeinalipour-Yazti, George Samaras, and Panos K. Chrysanthis. 2009. FSort: External sorting on flash-based sensor devices. In Data Management for Sensor Networks (DMSN’09). 1--6.Google ScholarGoogle Scholar
  11. Tyler Cossentine and Ramon Lawrence. 2010. Fast sorting on flash memory sensor nodes. In Proceedings of the 14th International Database Engineering and Applications Symposium (IDEAS’10). ACM, New York, NY, 105--113. DOI:https://doi.org/10.1145/1866480.1866496Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Tyler Cossentine and Ramon Lawrence. 2013. Efficient external sorting on flash memory embedded devices. International Journal of Database Management Systems 5 (2013), 1--20. DOI:https://doi.org/10.5121/ijdms.2013.5101Google ScholarGoogle ScholarCross RefCross Ref
  13. Martin A. Goetz. 1963. Internal and tape sorting using the replacement-selection technique. Commun. ACM 6, 5 (1963), 201--206. DOI:https://doi.org/10.1145/366552.366556Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Goetz Graefe. 2006. Implementing sorting in database systems. ACM Comput. Surv. 38, 3, Article 10 (Sept. 2006). DOI:https://doi.org/10.1145/1132960.1132964Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Hazar Harmouch and Felix Naumann. 2017. Cardinality estimation: An experimental survey. Proc. VLDB Endow. 11, 4 (Dec. 2017), 499--512. DOI:https://doi.org/10.1145/3186728.3164145Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Riley Jackson and Ramon Lawrence. 2019. Faster sorting for flash memory embedded devices. In 2019 IEEE Canadian Conference of Electrical and Computer Engineering (CCECE’19). IEEE, 1--5. DOI:https://doi.org/10.1109/CCECE.2019.8861811Google ScholarGoogle ScholarCross RefCross Ref
  17. A. Laga, J. Boukhobza, F. Singhoff, and M. Koskas. 2017. MONTRES: Merge on-the-run external sorting algorithm for large data volumes on SSD based storage systems. IEEE Trans. Comput. 66, 10 (Oct 2017), 1689--1702. DOI:https://doi.org/10.1109/TC.2017.2706678Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. J. Lee, H. Roh, and S. Park. 2016. External mergesort for flash-based solid state drives. IEEE Trans. Comput. 65, 5 (May 2016), 1518--1527. DOI:https://doi.org/10.1109/TC.2015.2451631Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Yang Liu, Zhen He, Yi-Ping Phoebe Chen, and Thi Nguyen. 2011. External sorting on flash memory via natural page run generation. Comput. J. 54, 11 (2011), 1882--1990. DOI:https://doi.org/10.1093/comjnl/bxr051Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Samuel R. Madden, Michael J. Franklin, Joseph M. Hellerstein, and Wei Hong. 2005. TinyDB: An acquisitional query processing system for sensor networks. ACM Trans. Database Syst. 30, 1 (March 2005), 122--173. DOI:https://doi.org/10.1145/1061318.1061322Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Hyoungmin Park and Kyuseok Shim. 2009. FAST: Flash-aware external sorting for mobile database systems. Journal of Systems and Software 82, 8 (2009), 1298--1312. DOI:https://doi.org/DOI: 10.1016/j.jss.2009.02.028Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. W. Penson, S. Fazackerley, and R. Lawrence. 2016. TEFS: A flash file system for use on memory constrained devices. In 2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE’16). 1--5. DOI:https://doi.org/10.1109/CCECE.2016.7726822Google ScholarGoogle ScholarCross RefCross Ref
  23. Chin-Hsien Wu and Kuo-Yi Huang. 2015. Data sorting in flash memory. Trans. Storage 11, 2, Article 7 (March 2015), 25 pages. DOI:https://doi.org/10.1145/2665067Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Efficient External Sorting for Memory-Constrained Embedded Devices with Flash Memory

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            HTML Format

            View this article in HTML Format .

            View HTML Format
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!