Search Results for "data cleaning"
Sort By:
Showing 77 open source projects for "data cleaning"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Intelligent Retail ManagementQuant offers a wide range of solutions for retail. Within one integrated software system, it allows you to efficiently combine the management of retail space, shelf labels and marketing materials with task management, reporting and automatic replenishment.
-
1
NYC Taxi Data
Import public NYC taxi and for-hire vehicle (Uber, Lyft)
The nyc-taxi-data repository is a rich dataset and exploratory project around New York City taxi trip records. It collects and preprocesses large-scale trip datasets (fares, pickup/dropoff, timestamps, locations, passenger counts) to enable data analysis, modeling, and visualization efforts. The project includes scripts and notebooks for cleaning and filtering the raw data, memory-efficient processing for large CSV/Parquet files, and aggregation workflows (e.g. trips per hour, heatmaps of pickups/dropoffs). ... -
2
AI Data Science Team
An AI-powered data science team of agents
AI Data Science Team is a Python library and agent ecosystem designed to accelerate and automate common data science workflows by modeling them as specialized AI “agents” that can be orchestrated to perform tasks like data cleaning, transformation, analysis, visualization, and machine learning. It provides a modular agent framework where each agent focuses on a step in the typical data science pipeline — for example, loading data from CSV/Excel files, cleaning and wrangling messy datasets, engineering predictive features, building models with AutoML, connecting to SQL databases, and producing visual outputs — all driven by natural language or programmatic instructions. ... -
3
The Data Engineering Handbook
Links to everything you'd ever want to learn about data engineering
...It includes beginner and intermediate boot camps, interview guides, data cleaning and transformation resources, and curated lists of newsletters and industry communities, making it useful both for self-study and technical interview preparation. The repository is actively maintained and widely starred, reflecting its role as a go-to reference for newcomers and experienced practitioners alike. -
4
janitor
Simple tools for data cleaning in R
janitor provides simple, convenient tools for data cleaning, formatting, and exploration in R. It is especially useful for cleaning messy data frames, removing duplicates, formatting column names, and producing frequency tables in a tidy workflow. -
ManageEngine Endpoint Central for IT ProfessionalsManageEngine's Endpoint Central is a Unified Endpoint Management Solution, that takes care of enterprise mobility management (including all features of mobile application management and mobile device management), as well as client management for a diversified range of endpoints - mobile devices, laptops, computers, tablets, server machines etc. With ManageEngine Endpoint Central, users can automate their regular desktop management routines like distributing software, installing patches, managing IT assets, imaging and deploying OS, and more.
-
5
Agentic Data Scientist
An end-to-end Data Scientist
Agentic Data Scientist is an experimental AI-driven research framework that orchestrates data science workflows through autonomous agents that can reason, plan, and execute complex analytics tasks. Unlike traditional scripted pipelines, this project lets AI agents break down high-level research goals into sub-tasks such as data acquisition, cleaning, modeling, evaluation, and reporting, with minimal human direction. -
6
litlyx
Analytics for developers, setup Analytics in 30 seconds
The easiest, developer-centric analytics tool. Litlyxis an open-source, self-hostable analytics solution for the modern framework. Litlyx offers a unique eyewear cleaning system that includes a special cleaning solution and reusable microfiber swabs. This system is designed to provide a more thorough and eco-friendly way to clean glasses, lenses, and screens. The brand emphasizes sustainability by reducing single-use plastics and promoting long-term use of their products. Their cleaning kit... -
7
Perfect Roadmap To Learn Data Science
Basic To Intermediate Python data science guide
...What makes it particularly valuable is its holistic nature: rather than focusing only on modeling or theory, it also addresses the broader lifecycle of data-science work, data ingestion, cleaning, EDA, feature engineering, model building, validation, deployment, etc. -
8
ExtractThinker
ExtractThinker is a Document Intelligence library for LLMs
ExtractThinker is a tool designed to facilitate the extraction and analysis of information from various data sources, aiding in data processing and knowledge discovery. -
9
CSV Lint
CSV Lint plug-in for Notepad++ for syntax highlighting
CSV Lint plug-in for Notepad++ for syntax highlighting, csv validation, automatic column and datatype detecting fixed width datasets, change datetime format, decimal separator, sort data, count unique values, convert to xml, json, sql etc. A plugin for data cleaning and working with messy data files. Use CSV Lint for metadata discovery, technical data validation, and reformatting on tabular data files. It is not meant to be a replacement for spreadsheet programs like Excel or SPSS, but rather it's a quality control tool to examine, verify or polish up a dataset before further processing. -
ShareCRM is an AI-powered enterprise CRM platform designed to connect data and teams across the entire customer lifecycle.ShareCRM is an AI-powered, customizable and affordable enterprise CRM solution to seamlessly integrate and empower every aspect of your business.
-
10
DOLMA
Data and tools for generating and inspecting OLMo pre-training data
DOLMA (Data Optimization and Learning for Model Alignment) is a framework designed to manage large-scale datasets for training and fine-tuning language models efficiently. -
11
MeshLab
The open source mesh processing system
...The open source system for processing and editing 3D triangular meshes. It provides a set of tools for editing, cleaning, healing, inspecting, rendering, texturing and converting meshes. It offers features for processing raw data produced by 3D digitization tools/devices and for preparing models for 3D printing. -
12
nb-clean
Clean Jupyter notebooks of outputs, metadata, and empty cells
nb-clean cleans Jupyter notebooks of cell execution counts, metadata, outputs, and (optionally) empty cells, preparing them for committing to version control. It provides both a Git filter and pre-commit hook to automatically clean notebooks before they're staged, and can also be used with other version control systems, as a command line tool, and as a Python library. It can determine if a notebook is clean or not, which can be used as a check in your continuous integration pipelines.... -
13
FDUPES
FDUPES is a program for identifying or deleting duplicate files
...Because it operates directly on file content rather than just filenames, fdupes can accurately detect true copies and guide cleaning operations in data cleanup or migration tasks. It’s a simple, efficient, and widely used utility on Unix-like systems, appreciated by administrators, developers, and power users. -
14
PandasAI
PandasAI is a Python library that integrates generative AI
PandasAI is a Python library that adds Generative AI capabilities to pandas, the popular data analysis and manipulation tool. It is designed to be used in conjunction with pandas, and is not a replacement for it. PandasAI makes pandas (and all the most used data analyst libraries) conversational, allowing you to ask questions to your data in natural language. For example, you can ask PandasAI to find all the rows in a DataFrame where the value of a column is greater than 5, and it will... -
15
Java Tablesaw
Java dataframe and visualization library
Tablesaw is a dataframe and visualization library that supports loading, cleaning, transforming, filtering, and summarizing data. If you work with data in Java, it may save you time and effort. Tablesaw also supports descriptive statistics and can be used to prepare data for working with machine learning libraries like Smile, Tribuo, H20.ai, DL4J. Import data from RDBMS, Excel, CSV, TSV, JSON, HTML, or Fixed Width text files, whether they are local or remote (http, S3, etc.) ... -
16
Streamline Analyst
AI agent that streamlines the entire process of data analysis
Streamline Analyst is a cutting-edge, open-source application powered by Large Language Models (LLMs) designed to revolutionize data analysis. This Data Analysis Agent effortlessly automates all the tasks such as data cleaning, preprocessing, and even complex operations like identifying target objects, partitioning test sets, and selecting the best-fit models based on your data. With Streamline Analyst, results visualization and evaluation become seamless. -
17
Miller
Miller is like awk, sed, cut, join, and sort for name-indexed data
Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. With Miller, you get to use named fields without needing to count positional indices, using familiar formats such as CSV, TSV, JSON, JSON Lines, and positionally-indexed. Then, on the fly, you can add new fields which are functions of existing fields, drop fields, sort, aggregate statistically, pretty-print, and more. Miller operates on key-value-pair data while the... -
18
CryoUtilities
A utility to improve performance and help manage storage on Steam Deck
The CryoUtilities repository is a collection of scripts and graphical utilities developed to help Steam Deck owners optimize the performance of their handheld gaming systems and manage storage more easily under SteamOS (and other Linux distributions in a best-effort manner). It provides tools to tweak memory management parameters such as swap size and swappiness, adjust HugePages and kernel memory behaviors, and free up space by cleaning leftover game data, shader caches, and compatibility data for uninstalled titles. The utilities aim to balance performance and system responsiveness, especially in gaming scenarios where constrained RAM and storage throughput can cause stutters or slow load times. The project includes an installer, an easy GUI launcher, and numerous toggles for Linux kernel parameters, and it abstracts many common tweaks that would otherwise require manual command-line intervention. -
19
Crowbook LaTeX
Converts books written in Markdown to HTML, LaTeX/PDF and EPUB
Crowbook's aim is to allow you to write a book in Markdown without worrying about formatting or typography and let the program generate HTML, PDF and EPUB output for you. Its focus is novels and fiction, and the default settings should (hopefully) generate readable books with correct typography without requiring you to worry about it. -
20
Mac Cleaner CLI
Scan and remove junk files, caches, logs, and more
Mac Cleaner CLI is a free and open-source terminal-based utility that helps users scan, identify, and remove unnecessary files from their macOS systems to reclaim storage space and keep systems tidy. Through a simple command-line interface, the tool performs deep scans to find caches, temporary files, logs, browser data, and other clutter, presenting results in an organized interactive menu where users can choose exactly what to clean. It emphasizes safety by allowing users to exclude... -
21
Automated Tool for Optimized Modelling
Automated Tool for Optimized Modelling
During the exploration phase of a machine learning project, a data scientist tries to find the optimal pipeline for his specific use case. This usually involves applying standard data cleaning steps, creating or selecting useful features, trying out different models, etc. Testing multiple pipelines requires many lines of code, and writing it all in the same notebook often makes it long and cluttered. -
22
NeMo Curator
Scalable data pre processing and curation toolkit for LLMs
NeMo Curator is a Python library specifically designed for fast and scalable dataset preparation and curation for large language model (LLM) use-cases such as foundation model pretraining, domain-adaptive pretraining (DAPT), supervised fine-tuning (SFT) and paramter-efficient fine-tuning (PEFT). It greatly accelerates data curation by leveraging GPUs with Dask and RAPIDS, resulting in significant time savings. The library provides a customizable and modular interface, simplifying pipeline... -
23
stylish-haskell
Haskell code prettifier
A simple Haskell code prettifier. The goal is not to format all of the code in a file since I find that kind of tools often "get in the way". However, manually cleaning up import statements, etc. gets tedious very quickly. This tool tries to help where necessary without getting in the way. Aligns and sorts import statements. Groups and wraps {-# LANGUAGE #-} pragmas, can remove (some) redundant pragmas. Removes trailing whitespace. Aligns branches in case and fields in records. Converts line endings (customizable) Replaces tabs by four spaces (turned off by default) Replaces some ASCII sequences by their Unicode equivalents (turned off by default) Format data constructors and fields in records. -
24
Open Interpreter
A natural language interface for computers
Open Interpreter is an open-source tool that provides a natural-language interface for interacting with your computer. It lets large language models (LLMs) run code locally (Python, JavaScript, shell, etc.), enabling you to ask your computer to do tasks like data analysis, file manipulation, browsing, etc. in human terms (“chat with your computer”), with safeguards. Runs locally or via configured remote LLM servers/inference backends, giving flexibility to use models you trust or have... -
25
HtmlSanitizer
Cleans HTML to avoid XSS attacks
HtmlSanitizer is a .NET library for cleaning HTML fragments and documents from constructs that can lead to XSS attacks. It uses AngleSharp to parse, manipulate, and render HTML and CSS. Because HtmlSanitizer is based on a robust HTML parser it can also shield you from deliberate or accidental "tag poisoning" where invalid HTML in one fragment can corrupt the whole document leading to broken layout or style. In order to facilitate different use cases, HtmlSanitizer can be customized at...