Disque is a distributed, fault-tolerant, in-memory message broker designed for job-/message-queue scenarios, created as an experiment to capture the “Redis as a jobs queue” use case but elevate it into a self-contained and scalable system. The design focuses on simplicity, providing a minimal and clear API for producers to enqueue jobs (messages) and consumers to process them, while handling replication, failure and re-queueing internally. Disque supports multi-master clusters where any node can serve producers and consumers, and messages are asynchronously replicated to a configurable number of nodes to ensure availability even in the face of failures. It offers both at-least-once and at-most-once delivery semantics depending on per-message configuration, and retries or re-queueing of un-acknowledged jobs are built in. The system trades off strong FIFO ordering semantics in favor of simplicity and scalability, giving best-effort ordering rather than strict global message ordering.
Features
- Distributed and multi-master architecture with any node accepting producers or consumers
- Configurable synchronous replication (W nodes) for job durability and failure resilience
- Support for both at-least-once and at-most-once delivery for each job
- Built-in retry and re-queueing of un-acknowledged tasks after a configurable time
- Lightweight API inspired by Redis but for queue semantics rather than key-value store
- Explicit introspection, monitoring and debugability to understand message state
Project Samples
