Scribe has recently undergone a major architectural revamp and simplification, and its new architecture is currently in production. To put this workload into perspective, the output of the Large Hadron Collider at CERN during its latest run was estimated to reach only 25 gigabytes per second. Scribe processes logs with an input rate that can exceed 2.5 terabytes per second and an output rate that can exceed 7 terabytes per second. Our solution is Scribe, a distributed queueing system that encapsulates all the complexity behind moving service logs from point A to point B. The task of collecting, aggregating, and delivering this volume of logs (with low latency and high throughput) requires a systematic approach. ![]() The outputs are generally processed somewhere other than where they were generated: They can be relevant to a variety of downstream processing pipelines and may each need to be accessed at different times. ![]() The total size of these logs is several petabytes every hour. Our hardware infrastructure comprises millions of machines, all of which generate logs that we need to process, store, and serve.
0 Comments
Leave a Reply. |