Datapipe Files

Real-time, incremental ETL library for ML with record-level depend

Status: Alpha

Brought to you by: otatarinova

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2023-07-19	975 Bytes	0
v0.13.0-alpha.3 source code.tar.gz	2023-07-19	604.1 kB	0
v0.13.0-alpha.3 source code.zip	2023-07-19	648.7 kB	0
Totals: 3 Items		1.3 MB	0

WIP 0.13.0

Major changes

Add datapipe.metastore.TransformMetaTable. Now each transform gets it's own meta table that tracks status of each transformation
Generalize BatchTransform and DatatableBatchTransform through BaseBatchTransformStep
Add transform_keys to *BatchTransform
Move changed idx computation out of DataStore to BaseBatchTransformStep
Add column priority to transform meta table, sort work by priority

New features

Add step reset-metadata CLI command
Add step fill-metadata CLI command that populates transform meta-table with all indices to process
Add helm chart for running regular loops in k8s as CronJob
Switch from vanilla tqdm to tqdm_loggable for better display in logs
Add step run-idx CLI command
Executors: datapipe.executor.SingleThreadExecutor, datapipe.executor.ray.RayExecutor

Bugfixes

Fix QdrantStore.read_rows when no idx is specified

Source: README.md, updated 2023-07-19

You Might Also Like

Gen AI apps are built with MongoDB Atlas Icon

Gen AI apps are built with MongoDB Atlas

Build gen AI apps with an all-in-one modern database: MongoDB Atlas

MongoDB Atlas provides built-in vector search and a flexible document model so developers can build, scale, and run gen AI apps without stitching together multiple databases. From LLM integration to semantic search, Atlas simplifies your AI architecture—and it’s free to get started.

Start Free

Simple, Secure Domain Registration Icon

Simple, Secure Domain Registration

Get your domain at wholesale price. Cloudflare offers simple, secure registration with no markups, plus free DNS, CDN, and SSL integration.

Register or renew your domain and pay only what we pay. No markups, hidden fees, or surprise add-ons. Choose from over 400 TLDs (.com, .ai, .dev). Every domain is integrated with Cloudflare's industry-leading DNS, CDN, and free SSL to make your site faster and more secure. Simple, secure, at-cost domain registration.

Sign up for free

Recommended Projects

CloverDX
Please, visit www.cloverdx.com for latest product versions. Data integration platform; can be used to transform/map/manipulate data in batch and near-realtime modes. Suppors various input/output formats (CSV,FIXLEN,Excel,XML,JSON,Parquet, Avro,EDI/X12,HL7,COBOL,LOTUS, etc.). Connects to...
Apache Kafka
Apache Kafka is a distributed streaming platform built around durable, partitioned logs called topics, enabling high-throughput, fault-tolerant event pipelines. Producers append records to partitions, brokers replicate them for durability, and consumer groups read them at their own pace while...
StarRocks
StarRocks is the next generation of real-time SQL engines for enterprise analytics. Real-time analytics is notoriously difficult. Complex data pipelines and de-normalized tables have always been a necessary evil. Processing any updates or deletes once data arrives has not been possible- until...
apache spark data pipeline osDQ
This is an offshoot project of open source data quality (osDQ) project https://sourceforge.net/projects/dataquality/ This sub project will create apache spark based data pipeline where JSON based metadata (file) will be used to run data processing , data pipeline , data quality and data...
Mage.ai
Open-source data pipeline tool for transforming and integrating data. The modern replacement for Airflow. Effortlessly integrate and synchronize data from 3rd party sources. Build real-time and batch pipelines to transform data using Python, SQL, and R. Run, monitor, and orchestrate thousands of...