GridFS


GridFS is a concept to enable easier access to grid computing resources by leveraging existing distributed filesystems like AFS, Coda, and DCE/DFS.

Since the first implementation of this system we will be using is based on AFS, GridFS and GridAFS will likely be used interchangeably. AFS is widely deployed among universities and research labs, and has a long history in the research community. However, it is often overlooked as "old technology". Now that there are 3 working open source AFS clients for the Linux operating system OpenAFS, Arla, and an Linux in-kernel implementation called kAFS. This has brought about a renewed interest in the viability of this distributed filesystem, and has attracted new developers. Even though AFS is still very similar to the original Andrew Filesystem project at CMU, there has never been another distributed filesystem that has approached the level of deployment AFS has reached. The codebases are mature, they are well understood. The part that is lacking is better support for cross-realm operations, and dealing with some cross-realm scaling issues, and the associated administrative overhead.

It is our belief that the renewed interest and development, and most importantly access to the codebases under open-source licenses will allow new innovation not so much in the core protocols, but in the way users interact with the system. Our goal at the SCl is to provide a filesystem infrastructure to our researchers from whatever computing resource they happen to be using. We also want to provide our computing resources to off-site collaborators, and encourage them to solve their filesystem access problems by setting up their own AFS servers and providing integration tools to easily allow our cluster computing resources to store files on remote AFS sites.

A significant portion of this effort will be placed into on-site collaborations with the Iowa State University central IT department, which operations the iastate.edu afs cell, the College of Engineering, which is also exploring providing AFS services, and the CMP group at Ames Lab.

The first step with collaboration with Iowa State has been accomplished by setting up cross-realm Kerberos keys, and configuring a gateway machine to allow users authenticated by Iowa State's Kerberos servers and authorized by our SCL LDAP infrastructure to log in. Further work and coding changes are needed on OpenSSH, and internal PAM modules to fully support this in a seamless manner.