[go: up one dir, main page]

Lakshminarasimhan et al., 2013 - Google Patents

ISABELA for effective in situ compression of scientific data

Lakshminarasimhan et al., 2013

View PDF
Document ID
3679066874805203317
Author
Lakshminarasimhan S
Shah N
Ethier S
Ku S
Chang C
Klasky S
Latham R
Ross R
Samatova N
Publication year
Publication venue
Concurrency and Computation: Practice and Experience

External Links

Snippet

Exploding dataset sizes from extreme‐scale scientific simulations necessitates efficient data management and reduction schemes to mitigate I/O costs. With the discrepancy between I/O bandwidth and computational power, scientists are forced to capture data infrequently …
Continue reading at nshah.net (PDF) (other versions)

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30067—File systems; File servers
    • G06F17/30129—Details of further file system functionalities
    • G06F17/3015—Redundancy elimination performed by the file system
    • G06F17/30156—De-duplication implemented within the file system, e.g. based on file segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30067—File systems; File servers
    • G06F17/30129—Details of further file system functionalities
    • G06F17/3015—Redundancy elimination performed by the file system
    • G06F17/30153—Redundancy elimination performed by the file system using compression, e.g. sparse files
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30312—Storage and indexing structures; Management thereof
    • G06F17/30321—Indexing structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30289—Database design, administration or maintenance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30286—Information retrieval; Database structures therefor; File system structures therefor in structured data stores
    • G06F17/30587—Details of specialised database models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30—Information retrieval; Database structures therefor; File system structures therefor
    • G06F17/30943—Information retrieval; Database structures therefor; File system structures therefor details of database functions independent of the retrieved data type
    • G06F17/30946—Information retrieval; Database structures therefor; File system structures therefor details of database functions independent of the retrieved data type indexing structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/50—Computer-aided design
    • G06F17/5009—Computer-aided design using simulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for programme control, e.g. control unit
    • G06F9/06—Arrangements for programme control, e.g. control unit using stored programme, i.e. using internal store of processing equipment to receive and retain programme
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Error detection; Error correction; Monitoring responding to the occurence of a fault, e.g. fault tolerance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06F—ELECTRICAL DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING; COUNTING
    • G06N—COMPUTER SYSTEMS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N99/00—Subject matter not provided for in other groups of this subclass

Similar Documents

Publication Publication Date Title
Lakshminarasimhan et al. ISABELA for effective in situ compression of scientific data
Lakshminarasimhan et al. Compressing the incompressible with ISABELA: In-situ reduction of spatio-temporal data
Liu et al. High-ratio lossy compression: Exploring the autoencoder to compress scientific data
Di et al. A survey on error-bounded lossy compression for scientific datasets
Schendel et al. ISOBAR preconditioner for effective and high-throughput lossless data compression
Isenko et al. Where is my training bottleneck? hidden trade-offs in deep learning preprocessing pipelines
Su et al. In-situ bitmaps generation and efficient data analysis based on bitmaps
US11107187B2 (en) Graph upscaling method for preserving graph properties
Knorr et al. ndzip-gpu: efficient lossless compression of scientific floating-point data on GPUs
Mostak An overview of MapD (massively parallel database)
Gomez et al. Improving floating point compression through binary masks
Di et al. Optimization of error-bounded lossy compression for hard-to-compress HPC data
US10726005B2 (en) Virtual split dictionary for search optimization
Chen et al. Fcbench: Cross-domain benchmarking of lossless compression for floating-point data
CN108958973B (en) Distributed file system erasure code data reconstruction storage node selection method and device
Zou et al. Performance optimization for relative-error-bounded lossy compression on scientific data
Lu et al. Canopus: A paradigm shift towards elastic extreme-scale data analytics on hpc storage
Kress et al. Preparing for in situ processing on upcoming leading-edge supercomputers
Wang et al. A speculative parallel decompression algorithm on apache spark
Lakshminarasimhan et al. DIRAQ: scalable in situ data-and resource-aware indexing for optimized query performance
US20240136024A1 (en) Apparatus and method with molecular dynamics simulation
Hishida et al. Beyond Compression: A Comprehensive Evaluation of Lossless Floating-Point Compression
Hentschel et al. Entropy-learned hashing: Constant time hashing with controllable uniformity
Anuradha et al. A detailed review on the prominent compression methods used for reducing the data volume of big data
Liu et al. A high performance compression method for climate data