Search Results for "xxhash"

xxHash is an extremely fast non-cryptographic hash algorithm, working at RAM speed limit. It is proposed in four flavors (XXH32, XXH64, XXH3_64bits and XXH3_128bits). The latest variant, XXH3, offers improved performance across the board, especially on small data. It successfully completes the SMHasher test suite which evaluates collision, dispersion and randomness qualities of hash functions.

A Golang lock-free thread-safe HashMap optimized for fastest read access. It is not a general-use HashMap and currently has slow write performance for write-heavy use. The minimal supported Golang version is 1.19 as it makes use of Generics and the new atomic package helpers. Reading from the hash map for numeric key types in a thread-safe way is faster than reading from a standard Golang map in an unsafe way and four times faster than Golang's sync.Map. Technical design decisions have been...

xxhsum 0.8.0 by Yann Collet Print or check xxHash (32, 64 or 128 bits) checksums. When FILE is -, read standard input. xxhsum supports a command line syntax similar but not identical to md5sum(1). Differences are: xxhsum doesn´t have text/binary mode switch (-b, -t); xxhsum always treats files as binary files; xxhsum has a hash bit width switch (-H); As xxHash is a fast non-cryptographic checksum algorithm, xxhsum should not be used for security related purposes.

This project has moved to www.quickhash-gui.org as of 2016-12-04. I kept v2.6.9.2 and below hosted here since Dec 16 but too many people were ignoring the fact that no updates were being posted here. For the latest QuickHash v2.8.4 release (Aug 28th 2017), go to www.quickhash-gui.org, and note that as of 29/12/16 a Debian package is also available

...Unlike scripts that crash on large files, EntropyGuard uses Polars LazyFrames to process datasets significantly larger than available RAM (e.g., 100GB+ on a standard laptop) without OOM errors. Hybrid Architecture: Exact Dedup: Uses xxHash to instantly strip ~60% of identical noise (CPU-based). Semantic Dedup: Uses local AI embeddings (SentenceTransformers + FAISS) to detect fuzzy duplicates (e.g., "Hello world" vs "Hi world"). Key Features: Zero API Costs: No external calls or privacy risks. Fault Tolerant: Built-in checkpoint/resume system. Pipe-friendly: Integrates seamlessly with Unix pipelines. ...