GFS Performance and Practices

Master Operations

Name Space management and locking

• Multiple operations are to be active and use locks over regions of the namespace
• GFS does not have a per-directory data structure
• GFS logically represents its namespace as a lookup table
• Each master operation acquires a set of locks before it runs

Replica placement

• GFS cluster distributed across many machine racks
• Need communication across several network switches
• Challenge to distribute data
• Chunk replica
  • Maximize data reliability
  • Maximize network bandwidth utilization
• Spread replicas across racks (survives even if entire rack offline)
• R can exploit aggregate bandwidth of multiple racks
• Re-replicate
  • When number of replicas fall below goal:
  • Chunkserver unavailable, corrupted etc.
  • Replicate based on priority
• Rebalance
  • Periodically moves replicas for better disk space and load balancing
  • Gradually fills up new chunk server
  • Removes replicas from chunk servers with below average space

Garbage collection

• When to delete file, file renamed to hidden name including delete timestamp
• During regular scan of file namespace, Hidden files remove if existed > 3 days
• Until then can be undeleted
• When removes, in memory metadata erased
• Orphaned chunks identified and erased
• With HeartBeat message, chunk server/master exchange info about files, master tells
  chunk server about files it can delete, chunk server free to delete

Advantages

• Simple, reliable in large scale distributed system
• Chunk creation may success on some servers but not others
• Replica deletion messages may be lost and resent
• Uniform and dependable way to clean up replicas
• Merges storage reclamation with background activities of master
• Done in batches
• Done only when master free
• Delay in reclaiming storage provides against accidental deletion

Disadvantages

• Delay hinders user effort to fine tune usage when storage tight
• Applications that create/delete may not be able to reuse space right away
• Expedite storage reclamation if file explicitly deleted again
• Allow users to apply different replication and reclamation policies

Shadow master

Master replication

• Replicated for reliability
• One master remains in charge of all mutations and background activities
• If fails, start instantly
• If machine or disk mails, monitor outside GFS starts new master with replicated log
• Clients only use canonical name of master

Shadow master

• Read only access to file systems even when primary master down
• Not mirrors so may lag primary slightly
• Enhance read availability for files not actively mutated
• Shadow master read replica of operation log, applies the same sequence of changes to data structures as primary does
• Polls chunk server at startup, monitors their status etc
• Depends only on primary for replica location updates

Fault Tolerance

High Availability

Two simple effective strategies:

1. fast recovery and
2. replication

Fast Recovery

• Master and chunk servers designed to restore their state in seconds
• No matter how they terminated, no distinction between normal and abnormal termination (servers remotely shutdown just by killing process)
• Clients and other servers experience minor hiccup (timeout) on outstanding requests, reconnect to the restarted server, and retry

Chunk Replication

• Chunk replicated on multiple chunk servers on different racks (different parts of the file namespace can have different replication level)
• Master clones existing replicas as chunk servers go offline or detect corrupted replicas (checksum verification)

Master Replication

• Master operation log and checkpoints replicated on multiple machines for reliability
• If the machine or disk failed, monitoring process outside of GFS starts a new master process elsewhere with the replicated log
• Clients use only the canonical name of the master (DNS alias that can be changed)
• Shadow masters provide read-only access to file system even when master is down

Data Integrity

• Chunkserver uses checksumming
  • Chunk broken into 64 KB blocks with associated 32 bits checksum Checksums are metadata kept in memory and stored persistently with logging, separate from user data
• Diagnostic tool
  • GFS servers generate diagnostic logs (sequentially + asynchronously) about significant events (chunk servers going up/down) and all RPC requests/replies
  • RPC logs include exact requests and responses sent on the wire without data
  • Interaction history can be reconstructed by matching requests with replies and collating RPC records on different machines
  • Logs used for load testing and performance analysis – Most recent events kept in memory and available for continuous online monitoring

Challenges

• Storage size
• Bottleneck for the clients
• Time

References:

1. "Google FIlesystem : Google Research" , S Ghemawat , ‎2003
2. "Distributed file system", Books,LLC
3. "Googles colossus makes search real time by dumping mop reduce",Shay Banon, 2010 at highscalablity.com