Wednesday, March 18, 2009

Scribe

Scribe is an open-source distributed logging system developed at Facebook.

It is designed to handle daunting amount of log, say 10 billion messages a day.

This logging system has three components:

1. Client Code interface - Thrift (generates glue-code for various languages), Category & Message
2. Distribution System - Scribe process runs on every node, it forward log message according to configuration file. (classified by category)
3. Do Something Usefullizer - log file? datawarehouse? HDFS/Hadoop?
claim: if your backend is scalable, why bother distributed logging system? (-> Chukwa)

Their major design decision is as following:
1. Don't assume a particular network topology: easily configurable
2. Reliability : reliable enough that we can expect to get all of the data almost all of the time, but not reliable enough to require heavyweight protocols and disk usage.
3. Simple Data model : a category and the actual message.

Simple, Unified, Scalable component is useful

1 comment:

  1. Thanks for sharing information on cloud computing. I got a complete information about the Cloud computing and its services from the Cloud Computing and Virtual Conference 2009 which is the World's largest and virtual conference on Cloud computing. I got a good opportunity to meet and talk with the World's leading experts on Cloud computing. I found the complete information about the conference by http://cloudslam09.com

    ReplyDelete