article

Free access

Scale and performance in a distributed file system

Editor: Anita K. Jones Authors:

Robert N. Sidebotham,

Michael J. WestAuthors Info & Claims

ACM Transactions on Computer Systems (TOCS), Volume 6, Issue 1

Pages 51 - 81

https://doi.org/10.1145/35037.35059

Published: 01 February 1988 Publication History

PDF eReader

Abstract

The Andrew File System is a location-transparent distributed tile system that will eventually span more than 5000 workstations at Carnegie Mellon University. Large scale affects performance and complicates system operation. In this paper we present observations of a prototype implementation, motivate changes in the areas of cache validation, server process structure, name translation, and low-level storage representation, and quantitatively demonstrate Andrews ability to scale gracefully. We establish the importance of whole-file transfer and caching in Andrew by comparing its performance with that of Sun Microsystems NFS tile system. We also show how the aggregation of files into volumes improves the operability of the system.

References

[1]

BROWNBRIDGE, D. R., MARSHALL, L. F., AND RANDELL, B. The Newcastle connection. Softw. Pract. Exper. 12 (1982), 1147-1162.

Google Scholar

[2]

KAZAR, M.L. Synchronization and caching issues in the Andrew file system. Tech. Rep. CMU- ITC-058, Information Technology Center, Carnegie Mellon Univ., Pittsburgh, Pa., June, 1987.

Google Scholar

[3]

MORRIS, J. H., SATYANARAYANAN, M., CONNER, M. H., HOWARD, J. H., ROSENTHAL, D. S., AND SMITH, F.D. Andrew: A distributed personal computing environment. Commun. ACM 29, 3 (Mar. 1986), 184-201.

Crossref

Google Scholar

[4]

OUSTERHOUT, J., DA COSTA, H., HARRISON, D., KUNZE, J., KUPFER, M., AND THOMPSON, J. A trace-driven analysis of the UNIX 4.2BSD file system. In Proceedings of the lOth ACM Symposium on Operating System Principles (Orcas Island, Wash., Dec. 1-4, 1985). ACM, New York, 1985, pp. 15-24.

Crossref

Google Scholar

[5]

RIFKIN, A. P., FORBES, M. P., HAMILTON, R. L., SABRIO, M., SHAH, S., AND YUEH, K. RFS architectural overview. In Usenix Conference Proceedings (Atlanta, Ga., Summer 1986). USENIX Association, Berkeley, Calif., pp 248-259.

Google Scholar

[6]

SATYANARAYANAN, M., HOWARD, J. H., NICHOLS, D. A., SIDEBOTHAM, a. N., SPECTOR, A. Z., AND WEST, M.J. The ITC distributed file system: Principles and design. In Proceedings of the lOth ACM Symposium on Operating System Principles (Orcas Island, Wash., Dec. 1-4, 1985). ACM, New York, 1985, pp. 35-50.

Crossref

Google Scholar

[7]

SCHROEDER, i. D., GIFFORD, D. K., AND NEEDHAM, R. U. A caching file system for a programmer's workstation. In Proceedings of the l Oth A CM Symposium on Operating System Principles (Orcas Island, Wash., Dec. 1-4, 1985). ACM, New York, 1985, pp. 25-34.

Crossref

Google Scholar

[8]

SIDEBOTHAM, a.N. Volumes: The Andrew file system data structuring primitive. European Unix User Group Conference Proceedings (August 1986). Also available as Tech. Rep. CMU- ITC-053, Information Technology Center, Carnegie Mellon Univ., Pittsburgh, Pa.

Google Scholar

[9]

SUN MICROSYSTEMS, INC. Networking on the SUN Workstation. Sun Microsystems, Mountain View, Calif., 1986.

Google Scholar

[10]

SVOBODOVA, L. File servers for network-based distributed systems. Comput. Surv. 16, 4 (Dec. 1984), 353-398.

Crossref

Google Scholar

[11]

TICH~, W. F., AND RUAN, Z. Towards a distributed file system. Tech. Rep. CSD-TR-480, Computer Science Dept., Purdue Univ., West Lafayette, In. 1984.

Google Scholar

[12]

WALKER, B., POPEK, G., ENGLISH, R., KLINE, C., AND THIEL, G. The LOCUS distributed operating system. In Proceedings of the 9th A CM Symposium on Operating System Principles (Bretton Woods, N.H., Oct. 11-13, 1983). ACM, New York, 1983, pp. 49-70.

Crossref

Google Scholar

Cited By

View all

Alimena JBeyer CBrüers BGaede FGillwald NHernandez Villanueva MJones EKemp YMadlener TSchwarz KWissing C(2024)Sustainable computing workshops in high-energy physics at DESYFrontiers in Computer Science10.3389/fcomp.2024.15027846Online publication date: 12-Dec-2024
https://doi.org/10.3389/fcomp.2024.1502784
Sly-Delgado BTovar BZhou JThain D(2024)Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVineSC24: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41406.2024.00068(1-13)Online publication date: 17-Nov-2024
https://doi.org/10.1109/SC41406.2024.00068
Gessert FWingerath WRitter NGessert FWingerath WRitter N(2024)Caching in Forschung und IndustrieSchnelles und skalierbares Cloud-Datenmanagement10.1007/978-3-031-54388-3_5(91-140)Online publication date: 3-May-2024
https://doi.org/10.1007/978-3-031-54388-3_5
Show More Cited By

Index Terms

Scale and performance in a distributed file system
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
2. Software and its engineering
  1. Software organization and properties
    1. Contextual software domains
      1. Operating systems
        Memory management
        Distributed memory
    2. Software system structures
      1. Distributed systems organizing principles

Recommendations

Scale and performance in a distributed file system

Andrew is a distributed computing environment being developed in a joint project by Carnegie Mellon University and IBM. One of the major components of Andrew is a distributed file system which constitutes underlying mechanism for sharing information. ...
File system performance and transaction support
File System Performance and Transaction Support

Reviews

Reviewer: Charles William Bash

The Andrew file system described in this paper is a location-transparent distributed file system that is planned to span more than 5,000 workstations. Such a scale affects performance and complicates system operation. The authors discuss several versions of the prototype and the observed performance of each. They conclude that the existing version could easily grow to handle 500–700 workstations but additional work will be necessary to meet the design goal. The system described is implemented in BSD UNIX 4.2. The basic concept is to move entire files to a local cache as needed rather than involve the file servers in actual I/O. Two versions with drastically different reactions to scale-up are described. The authors conclude with a statement of further revisions they have done that simplify the establishment of user disk quotas, ease volume maintenance, and assist in backing the system up. I was quite impressed that, although I am not familiar with BSD UNIX, the paper, which goes into considerable depth, was understandable without constant reference to a manual. This quality of technical writing deserves to be praised (and repeated). The paper touches on a phenomenon that is very popular in the modern computing world (distributed file systems) and one that needs considerable study before being implemented. This study is not just applicable to the academic (university or research) environment but also to many industrial environments where file sharing is needed.

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer Systems

ACM Transactions on Computer Systems Volume 6, Issue 1

Feb. 1988

152 pages

ISSN:0734-2071

EISSN:1557-7333

DOI:10.1145/35037

Editor:
Anita K. Jones

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 1988

Published in TOCS Volume 6, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1,054
Total Citations
View Citations
9,538
Total Downloads

Downloads (Last 12 months)637
Downloads (Last 6 weeks)121

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Alimena JBeyer CBrüers BGaede FGillwald NHernandez Villanueva MJones EKemp YMadlener TSchwarz KWissing C(2024)Sustainable computing workshops in high-energy physics at DESYFrontiers in Computer Science10.3389/fcomp.2024.15027846Online publication date: 12-Dec-2024
https://doi.org/10.3389/fcomp.2024.1502784
Sly-Delgado BTovar BZhou JThain D(2024)Reshaping High Energy Physics Applications for Near-Interactive Execution Using TaskVineSC24: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41406.2024.00068(1-13)Online publication date: 17-Nov-2024
https://doi.org/10.1109/SC41406.2024.00068
Gessert FWingerath WRitter NGessert FWingerath WRitter N(2024)Caching in Forschung und IndustrieSchnelles und skalierbares Cloud-Datenmanagement10.1007/978-3-031-54388-3_5(91-140)Online publication date: 3-May-2024
https://doi.org/10.1007/978-3-031-54388-3_5
Gessert FWingerath WRitter NGessert FWingerath WRitter N(2024)HTTP für global verteilte AnwendungenSchnelles und skalierbares Cloud-Datenmanagement10.1007/978-3-031-54388-3_3(35-60)Online publication date: 3-May-2024
https://doi.org/10.1007/978-3-031-54388-3_3
Kolosov OAktaş MSoljanin EYadgar G(2023)Theory vs. Practice in Modeling Edge Storage Systems2023 IEEE International Performance, Computing, and Communications Conference (IPCCC)10.1109/IPCCC59175.2023.10253868(313-322)Online publication date: 17-Nov-2023
https://doi.org/10.1109/IPCCC59175.2023.10253868
Shao XLi CXu Y(2023)MUSE: A Programmable Metadata Load Estimation Interface for Ceph File System2023 IEEE 29th International Conference on Parallel and Distributed Systems (ICPADS)10.1109/ICPADS60453.2023.00267(1951-1958)Online publication date: 17-Dec-2023
https://doi.org/10.1109/ICPADS60453.2023.00267
Kong WWang TLiu LLuo PCui JWang LHua XDuan WSu F(2023)A novel design and application of spatial data management platform for natural resourcesJournal of Cleaner Production10.1016/j.jclepro.2023.137183411(137183)Online publication date: Jul-2023
https://doi.org/10.1016/j.jclepro.2023.137183
(2023)LiteratureCloud Computing10.1016/B978-0-32-385277-7.00023-3(597-620)Online publication date: 2023
https://doi.org/10.1016/B978-0-32-385277-7.00023-3
Marinescu D(2023)Cloud data storageCloud Computing10.1016/B978-0-32-385277-7.00014-2(215-256)Online publication date: 2023
https://doi.org/10.1016/B978-0-32-385277-7.00014-2
Trinh TLe HVuongThi NHoangDuc HVuThi K(2023)A novel ensemble-based paradigm to process large-scale dataMultimedia Tools and Applications10.1007/s11042-023-16624-y83:9(26663-26685)Online publication date: 2-Sep-2023
https://doi.org/10.1007/s11042-023-16624-y
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations