Chubby

Chubby offer distributed lock system, to be used by other services.

Most interesting part of this work is, Chubby is out of practical reason - programmers are familiar with locking, and often retroactively add scale-out feature, so distributed lock system is more suitable than something like Paxos library.

This implies an important lesson. Large-scale distributed system evolves over long time period, and lots of programmers (with different skill level) write code for it. This means that 'easy-to-use' is often more essential than super-efficiency or elegant implementation.

Also, this kind of 'infrastructure service' for providing various services will be more prevalent, as this is something like 'middle ware for cloud'!