skip to main content
research-article

A nine year study of file system and storage benchmarking

Published:28 May 2008Publication History
Skip Abstract Section

Abstract

Benchmarking is critical when evaluating performance, but is especially difficult for file and storage systems. Complex interactions between I/O devices, caches, kernel daemons, and other OS components result in behavior that is rather difficult to analyze. Moreover, systems have different features and optimizations, so no single benchmark is always suitable. The large variety of workloads that these systems experience in the real world also adds to this difficulty.

In this article we survey 415 file system and storage benchmarks from 106 recent papers. We found that most popular benchmarks are flawed and many research papers do not provide a clear indication of true performance. We provide guidelines that we hope will improve future performance evaluations. To show how some widely used benchmarks can conceal or overemphasize overheads, we conducted a set of experiments. As a specific example, slowing down read operations on ext2 by a factor of 32 resulted in only a 2--5% wall-clock slowdown in a popular compile benchmark. Finally, we discuss future work to improve file system and storage benchmarking.

References

  1. *Abd-El-Malek, M., II, W. V. C., Cranor, C., Ganger, G., Hendricks, J., Klosterman, A. J., Mesnier, M., Prasad, M., Salmon, B., Sambasivan, R. R., Sinnamohideen, S., Strunk, J. D., Thereska, E., Wachs, M., and Wylie, J. J. 2005.Ursa Minor: Versatile cluster-based storage. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 59--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. *Adya, A., Bolosky, W. J., Castro, M., Cermak, G., Chaiken, R., Douceur, J. R., Howell, J., Lorch, J. R., Theimer, M., and Wattenhofer, R. P. 2002. FARSITE: Federated, available, and reliable storage for an incompletely trusted environment. In Proceedings of the 5th Symposium on Operating System Design and Implementation, Boston, MA, 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Agrawal, N., Bolosky, W. J., Douceur, J. R., and Lorch, J. R. 2007. A five-year study of file-system metadata. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 31--45. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. *Aguilera, M. K., Ji, M., Lillibridge, M., MacCormick, J., Oertli, E., Andersen, D., Burrows, M., Mann, T., and Thekkath, C. A. 2003. Block-Level security for network-attached disks. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 159--174. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Akkerman, W. 2002. Strace software home page. www.liacs.nl/~wichert/strace/.Google ScholarGoogle Scholar
  6. *Anderson, D. C., Chase, J. S., and Vahdat, A. M. 2000. Interposed request routing for scalable network storage. In Proceedings of the 4th USENIX Symposium on Operating System Design and Implementation, San Diego, CA, 259--272. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. *Anderson, E., Hobbs, M., Keeton, K., Spence, S., Uysal, M., and Veitch, A. 2002. Hippodrome: Running circles around storage administration. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 175--188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. *Anderson, E., Kallahalla, M., Uysal, M., and Swaminathan, R. 2004. Buttress: A toolkit for flexible and high fidelity I/O benchmarking. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 45--58. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Andrerson, D. 2002. Fstress: A flexible network file service benchmark. Tech. Rep. TR-2001-2002, Duke University. May.Google ScholarGoogle Scholar
  10. *Aranya, A., Wright, C. P., and Zadok, E. 2004. Tracefs: A file system to trace them all. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 129--143. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. *Arpaci-Dusseau, A. C., Arpaci-Dusseau, R. H., Burnett, N. C., Denehy, T. E., Engle, T. J., Gunawi, H. S., Nugent, J. A., and Popovici, F. I. 2003. Transforming policies into mechanisms with Infokernel. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (ACM SIGOPS), Bolton Landing, NY, 90--105. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Blaze, M. 1992. NFS tracing by passive network monitoring. In Proceedings of the USENIX Winter Conference, San Francisco, CA.Google ScholarGoogle Scholar
  13. Bray, T. 1996. Bonnie home page. www.textuality.com/bonnie.Google ScholarGoogle Scholar
  14. Bryant, R., Forester, R., and Hawkes, J. 2002. Filesystem performance and scalability in Linux 2.4.17. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track, Monterey, CA, 259--274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Bryant, R., Raddatz, D., and Sunshine, R. 2001. PenguinoMeter: A new file-I/O benchmark for Linux. In Proceedings of the 5th Annual Linux Showcase and Conference, Oakland, CA, 5--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Chen, P. M. and Patterson, D. A. 1993. A new approach to I/O performance evaluation—Self-Scaling I/O benchmarks, predicted I/O performance. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems (ACM SIGOPS), Seattle, WA, 1--12. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. *Cipar, J., Corner, M. D., and Berger, E. D. 2007. TFS: A transparent file system for contributory storage. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 215--229. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Coker, R. 2001. Bonnie++ home page. www.coker.com.au/bonnie++.Google ScholarGoogle Scholar
  19. *Corbett, P., English, B., Goel, A., Grcanac, T., Kleiman, S., Leong, J., and Sankar, S. 2004. Row-Diagonal parity for double disk failure correction. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. *Dabek, F., Kaashoek, M. F., Karger, D., and Morris, R. 2001. Wide-Area cooperative storage with CFS. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, Banff, Canada. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. *DeBergalis, M., Corbett, P., Kleiman, S., Lent, A., Noveck, D., Talpey, T., and Wittle, M. 2003. The direct access file system. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 175--188. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. *Denehy, T. E., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2002. Bridging the information gap in storage protocol stacks. In Proceedings of the Annual USENIX Technical Conference, Monterey, CA, 177--190. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. *Denehy, T. E., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2005. Journal-guided resynchronization for software RAID. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 87--100. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. *Dimitrijevic, Z., Rangaswami, R., and Chang, E. 2003. Design and implementation of semi-preemptible IO. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 145--158. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. *Eisler, M., Corbett, P., Kazar, M., Nydick, D. S., and Wagner, J. C. 2007. Data ONTAP GX: A scalable storage cluster. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 139--152. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Ellard, D., Ledlie, J., Malkani, P., and Seltzer, M. 2003. Passive NFS tracing of email and research workloads. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Ellard, D. and Seltzer, M. 2003a. New NFS tracing tools and techniques for system analysis. In Proceedings of the Annual USENIX Conference on Large Installation Systems Administration, San Diego, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Ellard, D. and Seltzer, M. 2003b. NFS tricks and benchmarking traps. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track, San Antonio, TX, 101--114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. *Flinn, J., Sinnamohideen, S., Tolia, N., and Satyanaryanan, M. 2003. Data staging on untrusted surrogates. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 15--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. *Fraser, K. and Chang, F. 2003. Operating system I/O speculation: How two invocations are faster than one. In Proceedings of the Annual USENIX Technical Conference. San Antonio, TX, 325--338.Google ScholarGoogle Scholar
  31. *Fu, K., Kaashoek, M. F., and Mazières, D. 2000. Fast and secure distributed read-only file system. In Proceedings of the 4th USENIX Symposium on Operating System Design and Implementation, San Diego, CA, 181--196. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. *Ghemawat, S., Gobioff, H., and Leung, S. T. 2003. The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles. (ACM SIGOPS), Bolton Landing, NY, 29--43. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. *Gniady, C., Butt, A. R., and Hu, Y. C. 2004. Program-Counter-Based pattern classification in buffer caching. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (ACM SIGOPS), San Francisco, CA, 395--408. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. *Gopal, B. and Manber, U. 1999. Integrating content-based access mechanisms with hierarchical file systems. In Proceedings of the 3rd Symposium on Operating Systems Design and Implementation. (ACM SIGOPS), New Orleans, LA, 265--278. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. *Grönvall, B., Westerlund, A., and Pink, S. 1999. The design of a multicast-based distributed file system. In Proceedings of the 3rd Symposium on Operating Systems Design and Implementation (ACM SIGOPS), New Orleans, LA, 251--264. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. *Gulati, A., Naik, M., and Tewari, R. 2007. Nache: Design and implementation of a caching proxy for nfsv4. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 199--214. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. *Howard, J. H., Kazar, M. L., Menees, S. G., Nichols, D. A., Satyanarayanan, M., Sidebotham, R. N., and West, M. J. 1988. Scale and performance in a distributed file system. ACM Trans. Comput. Syst. 6, 1 (Feb.), 51--81. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. *Huang, H., Hung, W., and Shin, K. 2005. FS2: Dynamic data replication in free disk space for improving disk performance and energy consumption. In Proceedings of the 20th ACM Symposium on Operating Systems Principles, Brighton, UK, 263--276. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Huang, L. and Chiueh, T. 2001. Charm: An I/O-driven execution strategy for high-performance transaction processing. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 275--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. *Joglekar, A., Kounavis, M. E., and Berry, F. L. 2005. A scalable and high performance software iSCSI implementation. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 267--280. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Joukov, N., Traeger, A., Iyer, R., Wright, C. P., and Zadok, E. 2006. Operating system profiling via latency analysis. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation (ACM SIGOPS), Seattle, WA, 89--102. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. *Joukov, N., Wong, T., and Zadok, E. 2005. Accurate and efficient replaying of file system traces. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 337--350. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. *Kallahalla, M., Riedel, E., Swaminathan, R., Wang, Q., and Fu, K. 2003. Plutus: Scalable secure file sharing on untrusted storage. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 29--42. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Kaminsky, M., Savvides, G., Mazieres, D., and Kaashoek, M. F. 2003. Decentralized user authentication in a global file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (ACM SIGOPS), Bolton Landing, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. Katcher, J. 1997. PostMark: A new filesystem benchmark. Tech. Rep. TR3022, Network Appliance. www.netapp.com/tech_library/3022.html.Google ScholarGoogle Scholar
  46. *Kim, J. M., Choi, J., Kim, J., Noh, S. H., Min, S. L., Cho, Y., and Kim, C. S. 2000. A low-overhead, high-performance unified buffer management scheme that exploits sequential and looping references. In Proceedings of the 4th USENIX Symposium on Operating System Design and Implementation, San Diego, CA, 119--134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. *Kim, M., Cox, L., and Noble, B. 2002. Safety, visibility, and performance in a wide-area file system. In Proceedings of the 1st USENIX Conference on File and Storage Technologies. Monterey, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. *Kroeger, T. M. and Long, D. D. E. 2001. Design and implementation of a predictive file prefetching algorithm. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 105--118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. LASS. 2006. UMass trace repository. http://traces.cs.umass.edu.Google ScholarGoogle Scholar
  50. *Lee, Y., Leung, K., and Satyanarayanan, M. 1999. Operation-Based update propagation in a mobile file system. In Proceedings of the Annual USENIX Technical Conference, Monterey, CA, 43--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. *Li, J., Krohn, M., Mazières, D., and Shasha, D. 2004. Secure untrusted data repository (SUNDR). In Proceedings of the 6th Symposium on Operating Systems Design and Implementation, San Francisco, CA, 121--136. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. *Lu, C., Alvarez, G. A., and Wilkes, J. 2002. Aqueduct: Online data migration with performance guarantees. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. *Lumb, C. R., Merchant, A., and Alvarez, G. A. 2003. Façade: Virtual storage devices with performance guarantees. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 131--144. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. *Lumb, C. R., Schindler, J., and Ganger, G. R. 2002. Freeblock scheduling outside of disk firmware. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 275--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. *MacCormick, J., Murphy, N., Najork, M., Thekkath, C., and Zhou, L. 2004. Boxwood: Abstractions as the foundation for storage infrastructure. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation, San Francisco, CA, 105--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  56. *Magoutis, K., Addetia, S., Fedorova, A., and Seltzer, M. I. 2003. Making the most out of direct-access network attached storage. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 189--202. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. *Magoutis, K., Addetia, S., Fedorova, A., Seltzer, M. I., Chase, J. S., Gallatin, A. J., Kisley, R., Wickremesinghe, R. G., and Gabber, E. 2002. Structure and performance of the direct access file system. In Proceedings of the Annual USENIX Technical Conference, Monterey, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. *Maziéres, D. 2001. A toolkit for user-level file systems. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 261--274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. *Mazières, D., Kaminsky, M., Kaashoek, M. F., and Witchel, E. 1999. Separating key management from file system security. In Proceedings of the 17th ACM Symposium on Operating Systems Principles, Charleston, SC, 124--139. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. McDougall, R. and Mauro, J. 2005. FileBench. www.solarisinternals.com/si/tools/filebench/. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. *Memik, G., Kandemir, M., and Choudhary, A. 2002. Exploiting inter-file access patterns using multi-collective I/O. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Mesnier, M. P., Wachs, M., Sambasivan, R. R., Lopez, J., Hendricks, J., Ganger, G. R., and O'Hallaron, D. 2007. //TRACE: Parallel trace replay with approximate causal events. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 153--167. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. *Miller, E., Freeman, W., Long, D., and Reed, B. 2002. Strong security for network-attached storage. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 1--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. Mogul, J. 1999. Brittle metrics in operating systems research. In Proceedings of the IEEE Workshop on Hot Topics in Operating Systems (HOTOS), Rio Rica, AZ, 90--95. Google ScholarGoogle ScholarDigital LibraryDigital Library
  65. Mummert, L. and Satyanarayanan, M. 1994. Long term distributed file reference tracing: Implementation and experience. Tech. Rep. CMU-CS-94-213, Carnegie Mellon University, Pittsburgh, Pennsylvania. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. *Muniswamy-Reddy, K., Holland, D. A., Braun, U., and Seltzer, M. 2006. Provenance-Aware storage systems. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 43--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. *Muniswamy-Reddy, K., Wright, C. P., Himmer, A., and Zadok, E. 2004. A versatile and user-oriented versioning file system. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 115--128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  68. *Muthitacharoen, A., Chen, B., and Mazieres, D. 2001. A low-bandwidth network file system. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, Banff, Canada. Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. *Muthitacharoen, A., Morris, R., Gil, T. M., and Che, B. 2002. Ivy: A read/write peer-to-peer file system. In Proceedings of the 5th Symposium on Operating System Design and Implementation, Boston, MA, 31--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  70. *Ng, W. T., Sun, H., Hillyer, B., Shriver, E., Gabber, E., and Ozden, B. 2002. Obtaining high performance for storage outsourcing. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 145--158. Google ScholarGoogle ScholarDigital LibraryDigital Library
  71. *Nightingale, E. B., Chen, P., and Flinn, J. 2005. Speculative execution in a distributed file system. In Proceedings of the 20th ACM Symposium on Operating Systems Principles, Brighton, UK, 191--205. Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. *Nightingale, E. B. and Flinn, J. 2004. Energy-Efficiency and storage flexibility in the Blue file system. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation, San Francisco, CA, 363--378. Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. *Nightingale, E. B., Veeraraghavan, K., Chen, P. M., and Flinn, J. 2006. Rethink the sync. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, Seattle, WA, 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. *Nugent, J., Arpaci-Dusseau, A., and Arpaci-Dusseau, R. 2003. Controlling your PLACE in the file system with gray-box techniques. In Proceedings of the Annual USENIX Technical Conference, San Antonio, TX, 311--323.Google ScholarGoogle Scholar
  75. OSDL. 2004. Iometer project. www.iometer.org/.Google ScholarGoogle Scholar
  76. OSDL. 2007. Database test suite. www.osdl.org/lab_activities/kernel_testing/osdl_database_test_suite/.Google ScholarGoogle Scholar
  77. Ousterhout, J. 1990. Why aren't operating systems getting faster as fast as hardware? In Proceedings of the Summer USENIX Technical Conference, Anaheim, CA, 247--256.Google ScholarGoogle Scholar
  78. Ousterhout, J., Costa, H., Harrison, D., Kunze, J., Kupfer, M., and Thompson, J. 1985. A trace-driven analysis of the UNIX 4.2 BSD file system. In Proceedings of the 10th ACM Symposium on Operating System Principles, Orcas Island, WA, 15--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  79. *Padioleau, Y. and Ridoux, O. 2003. A logic file system. In Proceedings of the Annual USENIX Technical Conference, San Antonio, TX, 99--112.Google ScholarGoogle Scholar
  80. *Papathanasiou, A. E. and Scott, M. L. 2004. Energy efficient prefetching and caching. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 255--268. Google ScholarGoogle ScholarDigital LibraryDigital Library
  81. *Patterson, H., Manley, S., Federwisch, M., Hitz, D., Kleinman, S., and Owara, S. 2002. SnapMirror: File system based asynchronous mirroring for disaster recovery. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 117--129. Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. *Peek, D. and Flinn, J. 2006. EnsemBlue: Integrating distributed storage and consumer electronics. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, Seattle, WA, 219--232. Google ScholarGoogle ScholarDigital LibraryDigital Library
  83. PEL. 2001. BYU trace distribution center. http://tds.cs.byu.edu/tds.Google ScholarGoogle Scholar
  84. *Peterson, Z. N. J., Burns, R., Ateniese, G., and Bono, S. 2007. Design and implementation of verifiable audit trails for a versioning file system. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 93--106. Google ScholarGoogle ScholarDigital LibraryDigital Library
  85. *Peterson, Z. N. J., Burns, R., J. Herring, A. S., and Rubin, A. D. 2005. Secure deletion for a versioning file system. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 143--154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  86. *Prabhakaran, V., Agrawal, N., Bairavasundaram, L. N., Gunawi, H. S., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2005a. IRON file systems. In Proceedings of the 20th ACM Symposium on Operating Systems Principles, Brighton, UK, 206--220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  87. *Prabhakaran, V., Arpaci-Dusseau, A. C., and Arpaci-Dussea, R. H. 2005b. Analysis and evolution of journaling file systems. In Proceedings of the Annual USENIX Technical Conference, Anaheim, CA, 105--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  88. *Quinlan, S. and Dorward, S. 2002. Venti: A new approach to archival storage. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 89--101. Google ScholarGoogle ScholarDigital LibraryDigital Library
  89. *Radkov, P., Yin, L., Goyal, P., Sarkar, P., and Shenoy, P. 2004. A performance comparison of NFS and iSCSI for IP-networked storage. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 101--114. Google ScholarGoogle ScholarDigital LibraryDigital Library
  90. *Rhea, S., Eaton, P., Geels, D., Weatherspoon, H., Zhao, B., and Kubiatowicz, J. 2003. Pond: The OceanStore prototype. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 1--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  91. Robinson, D. 1999. The advancement of NFS benchmarking: SFS 2.0. In Proceedings of the 13th USENIX Systems Administration Conference, Seattle, WA, 175--185. Google ScholarGoogle ScholarDigital LibraryDigital Library
  92. Roselli, D., Lorch, J. R., and Anderson, T. E. 2000. A comparison of file system workloads. In Proceedings of the Annual USENIX Technical Conference, San Diego, CA, 41--54. Google ScholarGoogle ScholarDigital LibraryDigital Library
  93. Rosenblum, M. 1992. The design and implementation of a log-structured file system. Ph.D. thesis, Electrical Engineering and Computer Sciences, Computer Science Division, University of California. Google ScholarGoogle ScholarDigital LibraryDigital Library
  94. *Rowstron, A. and Druschel, P. 2001. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. In Proceedings of the 18th ACM Symposium on Operating Systems Principles, Banff, Canada. Google ScholarGoogle ScholarDigital LibraryDigital Library
  95. Ruemmler, C. and Wilkes, J. 1993. UNIX disk access patterns. In Proceedings of the Winter USENIX Technical Conference, San Diego, CA, 405--420.Google ScholarGoogle Scholar
  96. Ruwart, T. M. 2001. File system performance benchmarks, then, now, and tomorrow. In Proceedings of the 14th IEEE Symposium on Mass Storage Systems, San Diego, CA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  97. *Saito, Y., Karamanolis, C., Karlsson, M., and Mahalingam, M. 2002. Taming aggressive replication in the Pangaea wide-area file system. In Proceedings of the 5th Symposium on Operating System Design and Implementation, Boston, MA, 15--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  98. Sandberg, R., Goldberg, D., Kleiman, S., Walsh, D., and Lyon, B. 1985. Design and implementation of the Sun network filesystem. In Proceedings of the Summer USENIX Technical Conference, Portland, Oregon, 119--130.Google ScholarGoogle Scholar
  99. *Santry, D. S., Feeley, M. J., Hutchinson, N. C., Veitch, A. C., Carton, R. W., and Ofir, J. 1999. Deciding when to forget in the Elephant file system. In Proceedings of the 17th ACM Symposium on Operating Systems Principles, Charleston, SC, 110--123. Google ScholarGoogle ScholarDigital LibraryDigital Library
  100. *Sarkar, P., Uttamchandani, S., and Voruganti, K. 2003. Storage over IP: When does hardware support help? In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 231--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  101. *Schindler, J., Griffin, J. L., Lumb, C. R., and Ganger, G. R. 2002. Track-Aligned extents: Matching access patterns to disk drive characteristics. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 259--274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  102. *Schindler, J., Schlosser, S. W., Shao, M., and Ailamaki, A. 2004. Atropos: A disk array volume manager for orchestrated use of disks. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 159--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  103. *Schlosser, S. W., Schindler, J., Papadomanolakis, S., Shao, M., Ailamaki, A., Faloutsos, C., and Ganger, G. R. 2005. On multidimensional data and modern disks. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 225--238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  104. Schmidt, A., Waas, F., Kersten, M., Florescu, D., Carey, M. J., Manolescu, I., and Busse, R. 2001. Why and how to benchmark XML databases. ACM SIGMOD Rec. 30, 3 (Sept.), 27--32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  105. *Schmuck, F. and Haskin, R. 2002. GPFS: A shared-disk file system for large computing clusters. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 231--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  106. *Seltzer, M. I., Ganger, G. R., McKusick, M. K., Smith, K. A., Soules, C. A. N., and Stein, C. A. 2000. Journaling versus soft updates: Asynchronous meta-data protection in file systems. In Proceedings of the Annual USENIX Technical Conference, San Diego, CA, 71--84. Google ScholarGoogle ScholarDigital LibraryDigital Library
  107. Seltzer, M. I., Krinsky, D., Smith, K. A., and Zhang, X. 1999. The case for application-specific benchmarking. In Proceedings of the IEEE Workshop on Hot Topics in Operating Systems (HOTOS), Rio Rica, AZ, 102--107. Google ScholarGoogle ScholarDigital LibraryDigital Library
  108. Shein, B., Callahan, M., and Woodbury, P. 1989. NFSSTONE: A network file server performance benchmark. In Proceedings of the Summer USENIX Technical Conference, Baltimore, MD, 269--275.Google ScholarGoogle Scholar
  109. Shepler, S. 2005. NFS version 4. In Proceedings of the Annual USENIX Technical Conference, Anaheim, CA. http://mediacast.sun.com/share/shepler/20050414_usenix_ext.pdf.Google ScholarGoogle Scholar
  110. *Shrira, L. and Xu, H. 2006. Thresher: An efficient storage manager for copy-on-write snapshots. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 57--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  111. *Sivathanu, G., Sundararaman, S., and Zadok, E. 2006. Type-Safe disks. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, Seattle, WA, 15--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  112. *Sivathanu, M., Bairavasundaram, L. N., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2004a. Life or death at block-level. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation, San Francisco, CA, 379--394. Google ScholarGoogle ScholarDigital LibraryDigital Library
  113. *Sivathanu, M., Bairavasundaram, L. N., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2005. Database-Aware semantically-smart storage. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 239--252. Google ScholarGoogle ScholarDigital LibraryDigital Library
  114. *Sivathanu, M., Prabhakaran, V., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2004b. Improving storage system availability with D-GRAID. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 15--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  115. *Sivathanu, M., Prabhakaran, V., Popovici, F. I., Denehy, T. E., Arpaci-Dusseau, A. C., and Arpaci-Dusseau, R. H. 2003. Semantically-Smart disk systems. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 73--88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  116. Small, C., Ghosh, N., Saleeb, H., Seltzer, M., and Smith, K. 1997. Does systems research measure up? Tech. Rep. TR-16-97, Harvard University. November.Google ScholarGoogle Scholar
  117. Smith, K. A. and Seltzer, M. I. 1997. File system aging—Increasing the relevance of file system benchmarks. In Proceedings of the ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, Seattle, WA, 203--213. Google ScholarGoogle ScholarDigital LibraryDigital Library
  118. SNIA. 2007. SNIA—Storage network industry association: IOTTA repository. http://iotta.snia.org.Google ScholarGoogle Scholar
  119. *Sobti, S., Garg, N., Zhang, C., Yu, X., Krishnamurthy, A., and Wang, R. 2002. PersonalRAID: Mobile storage for distributed and disconnected computers. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 159--174. Google ScholarGoogle ScholarDigital LibraryDigital Library
  120. *Soules, C. A. N., Goodson, G. R., Strunk, J. D., and Ganger, G. R. 2003. Metadata efficiency in versioning file systems. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 43--58. Google ScholarGoogle ScholarDigital LibraryDigital Library
  121. Spadavecchia, J. and Zadok, E. 2002. Enhancing NFS cross-administrative domain access. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track, Monterey, CA, 181--194. Google ScholarGoogle ScholarDigital LibraryDigital Library
  122. SPC. 2007. Storage performance council. www.storageperformance.org.Google ScholarGoogle Scholar
  123. SPEC. 2001. SPEC SFS97_R1 V3.0. www.spec.org/sfs97r1.Google ScholarGoogle Scholar
  124. SPEC. 2003. SPEC SMT97. www.spec.org/osg/smt97/.Google ScholarGoogle Scholar
  125. SPEC. 2004. SPEC SDM Suite. www.spec.org/osg/sdm91/.Google ScholarGoogle Scholar
  126. SPEC. 2005a. The SPEC organization. www.spec.org/.Google ScholarGoogle Scholar
  127. SPEC. 2005b. SPECweb99. www.spec.org/web99.Google ScholarGoogle Scholar
  128. SPEC. 2007. SPECviewperf 9. www.spec.org/gpc/opc.static/vp9info.html.Google ScholarGoogle Scholar
  129. *Stein, C. A., Howard, J. H., and Seltzer, M. I. 2001. Unifying file system protection. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 79--90. Google ScholarGoogle ScholarDigital LibraryDigital Library
  130. *Strunk, J. D., Goodson, G. R., Scheinholtz, M. L., Soules, C. A. N., and Ganger, G. R. 2000. Self-Securing storage: Protecting data in compromised systems. In Proceedings of the 4th USENIX Symposium on Operating System Design and Implementation, San Diego, CA, 165--180. Google ScholarGoogle ScholarDigital LibraryDigital Library
  131. Swartz, K. L. 1996. The brave little toaster meets Usenet. In Proceedings of the 10th USENIX System Administration Conference (LISA), Chicago, IL, 161--170. Google ScholarGoogle ScholarDigital LibraryDigital Library
  132. *Tan, Y., Wong, T., Strunk, J. D., and Ganger, G. R. 2005. Comparison-Based file server verification. In Proceedings of the Annual USENIX Technical Conference, Anaheim, CA, 121--133. Google ScholarGoogle ScholarDigital LibraryDigital Library
  133. Tang, D. 1995. Benchmarking filesystems. Tech. Rep. TR-19-95, Harvard University.Google ScholarGoogle Scholar
  134. Tang, D. and Seltzer, M. 1994. Lies, damned lies, and file system benchmarks. Tech. Rep. TR-34-94, Harvard University. December. In VINO: The 1994 Fall Harvest.Google ScholarGoogle Scholar
  135. *Thereska, E., Schindler, J., Bucy, J., Salmon, B., Lumb, C. R., and Ganger, G. R. 2004. A framework for building unobtrusive disk maintenance applications. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 213--226. Google ScholarGoogle ScholarDigital LibraryDigital Library
  136. *Tian, L., Feng, D., Jiang, H., Zhou, K., Zeng, L., Chen, J., Wang, Z., and Song, Z. 2007. PRO: A popularity-based multi-threaded reconstruction optimization for RAID-structured storage systems. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 277--290. Google ScholarGoogle ScholarDigital LibraryDigital Library
  137. *Tolia, N., Harkes, J., Kozuch, M., and Satyanarayanan, M. 2004. Integrating portable and distributed storage. In Proceedings of the 3rd USENIX Conference on File and Storage Technologies, San Francisco, CA, 227--238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  138. *Tolia, N., Kozuch, M., Satyanarayanan, M., Karp, B., Bressoud, T., and Perrig, A. 2003. Opportunistic use of content addressable storage for distributed file systems. In Proceedings of the Annual USENIX Technical Conference, San Antonio, TX, 127--140.Google ScholarGoogle Scholar
  139. TPC. 2005. Transaction processing performance council. www.tpc.org.Google ScholarGoogle Scholar
  140. Tridgell, A. 1999. Dbench-3.03 README. http://samba.org/ftp/tridge/dbench/README.Google ScholarGoogle Scholar
  141. Van Meter, R. 1997. Observing the effects of multi-zone disks. In Proceedings of the Annual USENIX Technical Conference, Anaheim, CA, 19--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  142. *Van Meter, R. and Gao, M. 2000. Latency management in storage systems. In Proceedings of the 4th USENIX Symposium on Operating System Design and Implementation, San Diego, CA, 103--118. Google ScholarGoogle ScholarDigital LibraryDigital Library
  143. *Veeraraghavan, K., Myrick, A., and Flinn, J. 2007. Cobalt: Separating content distribution from authorization in distributed file systems. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 231--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  144. VERITAS Software. 1999. VERITAS file server edition performance brief: A PostMark 1.11 benchmark comparison. Tech. Rep., Veritas Software Corporation. June. http://eval.veritas.com/webfiles/docs/fsedition-postmark.pdf.Google ScholarGoogle Scholar
  145. VeriTest. 2002. NetBench. www.veritest.com/benchmarks/netbench/.Google ScholarGoogle Scholar
  146. *Vilayannur, M., Nath, P., and Sivasubramaniam, A. 2005. Providing tunable consistency for a parallel file store. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 17--30. Google ScholarGoogle ScholarDigital LibraryDigital Library
  147. Vogels, W. 1999. File system usage in Windows NT 4.0. In Proceedings of the 17th ACM Symposium on Operating Systems Principles, Charleston, SC, 93--109. Google ScholarGoogle ScholarDigital LibraryDigital Library
  148. *Wachs, M., Abd-El-Malek, M., Thereska, E., and Ganger, G. R. 2007. Argon: Performance insulation for shared storage servers. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 61--76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  149. *Wang, A. A., Reiher, P., Popek, G. J., and Kuenning, G. H. 2002. Conquest: Better performance through a disk/persistent-RAM hybrid file system. In Proceedings of the Annual USENIX Technical Conference, Monterey, CA, 15--28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  150. *Wang, R. Y., Anderson, T. E., and Patterson, D. A. 1999. Virtual log based file systems for a programmable disk. In Proceedings of the 3rd Symposium on Operating Systems Design and Implementation, New Orleans, LA, 29--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  151. *Wang, Y. and Merchant, A. 2007. Proportional-Share scheduling for distributed storage systems. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 47--60. Google ScholarGoogle ScholarDigital LibraryDigital Library
  152. Watson, A. and Nelson, B. 1992. LADDIS: A multi-vendor and vendor-neutral SPEC NFS benchmark. In Proceedings of the 6th USENIX Systems Administration Conference (LISA VI), Long Beach, CA, 17--32.Google ScholarGoogle Scholar
  153. *Weddle, C., Oldham, M., Qian, J., Wang, A. A., Reiher, P., and Kuenning, G. 2007. PARAID: A gear-shifting power-aware RAID. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 245--260. Google ScholarGoogle ScholarDigital LibraryDigital Library
  154. *Weil, S., Brandt, S., Miller, E., Long, D., and Maltzahn, C. 2006. Ceph: A scalable, high-performance distributed file system. In Proceedings of the 7th Symposium on Operating Systems Design and Implementation, Seattle, WA, 307--320. Google ScholarGoogle ScholarDigital LibraryDigital Library
  155. Wittle, M. and Keith, B. E. 1993. LADDIS: The next generation in NFS file server benchmarking. In Proceedings of the Summer USENIX Technical Conference, Cincinnati, OH, 111--128. Google ScholarGoogle ScholarDigital LibraryDigital Library
  156. Wright, C. P., Dave, J., and Zadok, E. 2003a. Cryptographic file systems performance: What you don't know can hurt you. In Proceedings of the 2nd IEEE International Security In Storage Workshop. IEEE Computer Society, Washington, DC, 47--61. Google ScholarGoogle ScholarDigital LibraryDigital Library
  157. Wright, C. P., Joukov, N., Kulkarni, D., Miretskiy, Y., and Zadok, E. 2005. Auto-Pilot: A platform for system software benchmarking. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track, Anaheim, CA, 175--187. Google ScholarGoogle ScholarDigital LibraryDigital Library
  158. *Wright, C. P., Martino, M., and Zadok, E. 2003b. NCryptfs: A secure and convenient cryptographic file system. In Proceedings of the Annual USENIX Technical Conference, San Antonio, TX, 197--210.Google ScholarGoogle Scholar
  159. *Yu, X., Gum, B., Chen, Y., Wang, R. Y., Li, K., Krishnamurthy, A., and Anderson, T. E. 2000. Trading capacity for performance. In Proceedings of the 4th USENIX Symposium on Operating System Design and Implementation, San Diego, CA, 243--258. Google ScholarGoogle ScholarDigital LibraryDigital Library
  160. *Yumerefendi, A. R. and Chase, J. S. 2007. Strong accountability for network storage. In Proceedings of the 5th USENIX Conference on File and Storage Technologies, San Jose, CA, 77--92. Google ScholarGoogle ScholarDigital LibraryDigital Library
  161. *Zadok, E. 2002. Overhauling Amd for the '00s: A case study of GNU autotools. In Proceedings of the Annual USENIX Technical Conference, FREENIX Track, Monterey, CA, 287--297. Google ScholarGoogle ScholarDigital LibraryDigital Library
  162. *Zadok, E., Anderson, J. M., Bădulescu, I., and Nieh, J. 2001. Fast indexing: Support for size-changing algorithms in stackable file systems. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 289--304. Google ScholarGoogle ScholarDigital LibraryDigital Library
  163. *Zadok, E., Bădulescu, I., and Shender, A. 1999. Extending file systems using stackable templates. In Proceedings of the Annual USENIX Technical Conference, Monterey, CA, 57--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  164. *Zadok, E. and Nieh, J. 2000. FiST: A language for stackable file systems. In Proceedings of the Annual USENIX Technical Conference, San Diego, CA, 55--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  165. *Zhang, C., Yu, X., Krishnamurthy, A., and Wang, R. Y. 2002. Configuring and scheduling an eager-writing disk array for a transaction processing workload. In Proceedings of the 1st USENIX Conference on File and Storage Technologies, Monterey, CA, 289--304. Google ScholarGoogle ScholarDigital LibraryDigital Library
  166. *Zhang, Z. and Ghose, K. 2003. yFS: A journaling file system design for handling large data sets with reduced seeking. In Proceedings of the 2nd USENIX Conference on File and Storage Technologies, San Francisco, CA, 59--72. Google ScholarGoogle ScholarDigital LibraryDigital Library
  167. *Zhou, Y., Philbin, J., and Li, K. 2001. The multi-queue replacement algorithm for second level buffer caches. In Proceedings of the Annual USENIX Technical Conference, Boston, MA, 91--104. Google ScholarGoogle ScholarDigital LibraryDigital Library
  168. *Zhu, N., Chen, J., and Chiueh, T. 2005a. TBBT: Scalable and accurate trace replay for file server evaluation. In Proceedings of the 4th USENIX Conference on File and Storage Technologies, San Francisco, CA, 323--336. Google ScholarGoogle ScholarDigital LibraryDigital Library
  169. *Zhu, Q., Chen, Z., Tan, L., Zhou, Y., Keeton, K., and Wilkes, J. 2005b. Hibernator: Helping disk arrays sleep through the winter. In Proceedings of the 20th ACM Symposium on Operating Systems Principles, Brighton, UK, 177--190. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. A nine year study of file system and storage benchmarking

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Storage
          ACM Transactions on Storage  Volume 4, Issue 2
          May 2008
          80 pages
          ISSN:1553-3077
          EISSN:1553-3093
          DOI:10.1145/1367829
          Issue’s Table of Contents

          Copyright © 2008 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 28 May 2008
          • Revised: 1 November 2007
          • Accepted: 1 November 2007
          • Received: 1 May 2007
          Published in tos Volume 4, Issue 2

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!