Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki. Loki supports multi-tenancy so that data between tenants is completely separated. Multi-tenancy is achieved through a tenant ID (which is represented as an alphanumeric string). When multi-tenancy mode is disabled, all requests are internally given a tenant ID of “fake”.
Features
- Loki is optimized for both running locally (or at small scale) and for scaling horizontally
- Loki comes with a single process mode that runs all of the required microservices in one process
- The single process mode is great for testing Loki or for running it at a small scale
- For horizontal scalability, the microservices of Loki can be broken out into separate processes, allowing them to scale independently of each other
- To ensure consistent query results, Loki uses Dynamo-style quorum consistency on reads and writes
- In general, all lines pushed to Loki for a given stream (unique combination of labels) must have a newer timestamp than the line received before it
Project Samples
