smallpond is a lightweight distributed data processing framework built by DeepSeek, designed to scale DuckDB workloads over clusters using their 3FS (Fire-Flyer File System) backend. The idea is to preserve DuckDB’s fast analytics engine but lift it from single-node to multi-node settings, giving you the ability to operate on large datasets (e.g. petabyte scale) without moving to a heavyweight system like Spark. Users write Python-like code (via DataFrame APIs or SQL strings) to express their transformations; behind the scenes, tasks are scheduled (often via Ray) and pushed into DuckDB instances operating on partitioned data. Because the storage layer (3FS) is optimized for random access and high throughput, smallpond can shuffle data, repartition, and manage intermediate results across nodes.
Features
- Distributed extension of DuckDB: support for running SQL / DataFrame operations across nodes
- Uses 3FS as the shared data backend to manage data storage and shuffle operations
- APIs for transformations via SQL strings or Python functions (map, partial_sql)
- Support for repartitioning by number of partitions, row count, or hash on a column
- Two execution modes: high-level dynamic (Ray-based) and low-level static graph execution
- Optimized for large-scale workloads (benchmarked at ~100 TiB sorting)
Project Samples
