Crate zstd_sys[−][src]

Streaming

< Copy dictionary content internally

< Reference dictionary content -- the dictionary buffer must outlive its users.

ZDICT_trainFromBuffer(): Train a dictionary from an array of samples. Redirect towards ZDICT_optimizeTrainFromBuffer_cover() single-threaded, with d=8 and steps=4. Samples must be stored concatenated in a single flat buffer samplesBuffer, supplied with an array of sizes samplesSizes, providing the size of each sample, in order. The resulting dictionary will be saved into dictBuffer. @return: size of dictionary stored into dictBuffer (<= dictBufferCapacity) or an error code, which can be tested with ZDICT_isError(). Note: ZDICT_trainFromBuffer() requires about 9 bytes of memory for each input byte. Tips: In general, a reasonable dictionary has a size of ~ 100 KB. It's possible to select smaller or larger size, just by specifying dictBufferCapacity. In general, it's recommended to provide a few thousands samples, though this can vary a lot. It's recommended that total size of all samples be about ~x100 times the target size of dictionary.

ZSTDMT_compressStream_generic() : Combines ZSTDMT_compressStream() with optional ZSTDMT_flushStream() or ZSTDMT_endStream() depending on flush directive. @return : minimum amount of data still to be flushed 0 if fully flushed or an error code note : needs to be init using any ZSTD_initCStream*() variant

ZSTDMT_initCStream_internal() : Private use only. Init streaming operation. expects params to be valid. must receive dict, or cdict, or none, but not both. @return : 0, or an error code

ZSTDMT_updateCParams_whileCompressing() : Updates only a selected set of compression parameters, to remain compatible with current frame. New parameters will be applied to next compression job.

ZSTD_CCtxParam_getParameter() : Similar to ZSTD_CCtx_getParameter. Get the requested value of one compression parameter, selected by enum ZSTD_cParameter. @result : 0, or an error code (which can be tested with ZSTD_isError()).

ZSTD_CCtxParam_setParameter() : Similar to ZSTD_CCtx_setParameter. Set one compression parameter, selected by enum ZSTD_cParameter. Parameters must be applied to a ZSTD_CCtx using ZSTD_CCtx_setParametersUsingCCtxParams(). Note : when value is an enum, cast it to unsigned for proper type checking. @result : 0, or an error code (which can be tested with ZSTD_isError()).

ZSTD_CCtxParams_init() : Initializes the compression parameters of cctxParams according to compression level. All other parameters are reset to their default values.

ZSTD_CCtxParams_init_advanced() : Initializes the compression and frame parameters of cctxParams according to params. All other parameters are reset to their default values.

ZSTD_CCtxParams_reset() : Reset params to default values.

ZSTD_CCtx_getParameter() : Get the requested value of one compression parameter, selected by enum ZSTD_cParameter. @result : 0, or an error code (which can be tested with ZSTD_isError()).

ZSTD_CCtx_loadDictionary() : Create an internal CDict from dict buffer. Decompression will have to use same dictionary. @result : 0, or an error code (which can be tested with ZSTD_isError()). Special: Adding a NULL (or 0-size) dictionary invalidates previous dictionary, meaning "return to no-dictionary mode". Note 1 : Dictionary will be used for all future compression jobs. To return to "no-dictionary" situation, load a NULL dictionary Note 2 : Loading a dictionary involves building tables, which are dependent on compression parameters. For this reason, compression parameters cannot be changed anymore after loading a dictionary. It's also a CPU consuming operation, with non-negligible impact on latency. Note 3 :dict content will be copied internally. Use ZSTD_CCtx_loadDictionary_byReference() to reference dictionary content instead. In such a case, dictionary buffer must outlive its users. Note 4 : Use ZSTD_CCtx_loadDictionary_advanced() to precisely select how dictionary content must be interpreted.

ZSTD_CCtx_refCDict() : Reference a prepared dictionary, to be used for all next compression jobs. Note that compression parameters are enforced from within CDict, and supercede any compression parameter previously set within CCtx. The dictionary will remain valid for future compression jobs using same CCtx. @result : 0, or an error code (which can be tested with ZSTD_isError()). Special : adding a NULL CDict means "return to no-dictionary mode". Note 1 : Currently, only one dictionary can be managed. Adding a new dictionary effectively "discards" any previous one. Note 2 : CDict is just referenced, its lifetime must outlive CCtx.

ZSTD_CCtx_refPrefix() : Reference a prefix (single-usage dictionary) for next compression job. Decompression need same prefix to properly regenerate data. Prefix is only used once. Tables are discarded at end of compression job (ZSTD_e_end). @result : 0, or an error code (which can be tested with ZSTD_isError()). Special: Adding any prefix (including NULL) invalidates any previous prefix or dictionary Note 1 : Prefix buffer is referenced. It must outlive compression job. Its contain must remain unmodified up to end of compression (ZSTD_e_end). Note 2 : Referencing a prefix involves building tables, which are dependent on compression parameters. It's a CPU consuming operation, with non-negligible impact on latency. If there is a need to use same prefix multiple times, consider loadDictionary instead. Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent). Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode.

ZSTD_CCtx_reset() : Return a CCtx to clean state. Useful after an error, or to interrupt an ongoing compression job and start a new one. Any internal data not yet flushed is cancelled. The parameters and dictionary are kept unchanged, to reset them use ZSTD_CCtx_resetParameters().

ZSTD_CCtx_resetParameters() : All parameters are back to default values (compression level is ZSTD_CLEVEL_DEFAULT). Dictionary (if any) is dropped. Resetting parameters is only possible during frame initialization (before starting compression). To reset the context use ZSTD_CCtx_reset(). @return 0 or an error code (which can be checked with ZSTD_isError()).

ZSTD_CCtx_setParameter() : Set one compression parameter, selected by enum ZSTD_cParameter. Setting a parameter is generally only possible during frame initialization (before starting compression). Exception : when using multi-threading mode (nbThreads >= 1), following parameters can be updated during compression (within same frame): => compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy. new parameters will be active on next job, or after a flush(). Note : when value type is not unsigned (int, or enum), cast it to unsigned for proper type checking. @result : informational value (typically, value being set, correctly clamped), or an error code (which can be tested with ZSTD_isError()).

ZSTD_CCtx_setParametersUsingCCtxParams() : Apply a set of ZSTD_CCtx_params to the compression context. This can be done even after compression is started, if nbWorkers==0, this will have no impact until a new compression is started. if nbWorkers>=1, new parameters will be picked up at next job, with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated).

ZSTD_CCtx_setPledgedSrcSize() : Total input data size to be compressed as a single frame. This value will be controlled at the end, and result in error if not respected. @result : 0, or an error code (which can be tested with ZSTD_isError()). Note 1 : 0 means zero, empty. In order to mean "unknown content size", pass constant ZSTD_CONTENTSIZE_UNKNOWN. ZSTD_CONTENTSIZE_UNKNOWN is default value for any new compression job. Note 2 : If all data is provided and consumed in a single round, this value is overriden by srcSize instead.

ZSTD_DCtx_loadDictionary() : Create an internal DDict from dict buffer, to be used to decompress next frames. @result : 0, or an error code (which can be tested with ZSTD_isError()). Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary, meaning "return to no-dictionary mode". Note 1 : dict content will be copied internally. Use ZSTD_DCtx_loadDictionary_byReference() to reference dictionary content instead. In which case, the dictionary buffer must outlive its users. Note 2 : Loading a dictionary involves building tables, which has a non-negligible impact on CPU usage and latency. Note 3 : Use ZSTD_DCtx_loadDictionary_advanced() to select how dictionary content will be interpreted and loaded.

ZSTD_DCtx_refDDict() : Reference a prepared dictionary, to be used to decompress next frames. The dictionary remains active for decompression of future frames using same DCtx. @result : 0, or an error code (which can be tested with ZSTD_isError()). Note 1 : Currently, only one dictionary can be managed. Referencing a new dictionary effectively "discards" any previous one. Special : adding a NULL DDict means "return to no-dictionary mode". Note 2 : DDict is just referenced, its lifetime must outlive its usage from DCtx.

ZSTD_DCtx_refPrefix() : Reference a prefix (single-usage dictionary) for next compression job. Prefix is only used once. Reference is discarded at end of frame. End of frame is reached when ZSTD_DCtx_decompress_generic() returns 0. @result : 0, or an error code (which can be tested with ZSTD_isError()). Note 1 : Adding any prefix (including NULL) invalidates any previously set prefix or dictionary Note 2 : Prefix buffer is referenced. It must outlive decompression job. Prefix buffer must remain unmodified up to the end of frame, reached when ZSTD_DCtx_decompress_generic() returns 0. Note 3 : By default, the prefix is treated as raw content (ZSTD_dm_rawContent). Use ZSTD_CCtx_refPrefix_advanced() to alter dictMode. Note 4 : Referencing a raw content prefix has almost no cpu nor memory cost. A fulldict prefix is more costly though.

ZSTD_DCtx_reset() : Return a DCtx to clean state. If a decompression was ongoing, any internal data not yet flushed is cancelled. All parameters are back to default values, including sticky ones. Dictionary (if any) is dropped. Parameters can be modified again after a reset.

ZSTD_DCtx_setFormat() : Instruct the decoder context about what kind of data to decode next. This instruction is mandatory to decode data without a fully-formed header, such ZSTD_f_zstd1_magicless for example. @return : 0, or an error code (which can be tested using ZSTD_isError()).

ZSTD_DCtx_setMaxWindowSize() : Refuses allocating internal buffers for frames requiring a window size larger than provided limit. This is useful to prevent a decoder context from reserving too much memory for itself (potential attack scenario). This parameter is only useful in streaming mode, since no internal buffer is allocated in direct mode. By default, a decompression context accepts all window sizes <= (1 << ZSTD_WINDOWLOG_MAX) @return : 0, or an error code (which can be tested using ZSTD_isError()).

ZSTD_adjustCParams() : optimize params for a given srcSize and dictSize. both values are optional, select 0 if unknown.

ZSTD_checkCParams() : Ensure param values remain within authorized range

Simple API / /*! ZSTD_compress() : Compresses src content as a single zstd compressed frame into already allocated dst. Hint : compression runs faster if dstCapacity >= ZSTD_compressBound(srcSize). @return : compressed size written into dst (<= `dstCapacity), or an error code if it fails (which can be tested using ZSTD_isError()).

Buffer-less streaming compression (synchronous mode)

ZSTD_compressCCtx() : Same as ZSTD_compress(), requires an allocated ZSTD_CCtx (see ZSTD_createCCtx()).

ZSTD_compress_advanced() : Same as ZSTD_compress_usingDict(), with fine-tune control over each compression parameter

ZSTD_compress_generic() : Behave about the same as ZSTD_compressStream. To note :

ZSTD_compress_generic_simpleArgs() : Same as ZSTD_compress_generic(), but using only integral types as arguments. Argument list is larger than ZSTD_{in,out}Buffer, but can be helpful for binders from dynamic languages which have troubles handling structures containing memory pointers.

ZSTD_compress_usingCDict() : Compression using a digested Dictionary. Faster startup than ZSTD_compress_usingDict(), recommended when same dictionary is used multiple times. Note that compression level is decided during dictionary creation. Frame parameters are hardcoded (dictID=yes, contentSize=yes, checksum=no)

ZSTD_compress_usingCDict_advanced() : Same as ZSTD_compress_usingCDict(), with fine-tune control over frame parameters

Simple dictionary API / /*! ZSTD_compress_usingDict() : Compression using a predefined Dictionary (see dictBuilder/zdict.h). Note : This function loads the dictionary, resulting in significant startup delay. Note : When dict == NULL || dictSize < 8 no dictionary is used.

ZSTD_CCtx_params : Quick howto :

ZSTD_createCDict() : When compressing multiple messages / blocks with the same dictionary, it's recommended to load it just once. ZSTD_createCDict() will create a digested dictionary, ready to start future compression operations without startup delay. ZSTD_CDict can be created once and shared by multiple threads concurrently, since its usage is read-only. dictBuffer can be released after ZSTD_CDict creation, since its content is copied within CDict

ZSTD_createCDict_byReference() : Create a digested dictionary for compression Dictionary content is simply referenced, and therefore stays in dictBuffer. It is important that dictBuffer outlives CDict, it must remain read accessible throughout the lifetime of CDict

ZSTD_createDDict() : Create a digested dictionary, ready to start decompression operation without startup delay. dictBuffer can be released after DDict creation, as its content is copied inside DDict

ZSTD_createDDict_byReference() : Create a digested dictionary, ready to start decompression operation without startup delay. Dictionary content is referenced, and therefore stays in dictBuffer. It is important that dictBuffer outlives DDict, it must remain read accessible throughout the lifetime of DDict

ZSTD_decompress() : compressedSize : must be the exact size of some number of compressed and/or skippable frames. dstCapacity is an upper bound of originalSize to regenerate. If user cannot imply a maximum upper bound, it's better to use streaming mode to decompress data. @return : the number of bytes decompressed into dst (<= dstCapacity), or an errorCode if it fails (which can be tested using ZSTD_isError()).

ZSTD_decompressDCtx() : Same as ZSTD_decompress(), requires an allocated ZSTD_DCtx (see ZSTD_createDCtx())

ZSTD_decompress_generic() : Behave the same as ZSTD_decompressStream. Decompression parameters cannot be changed once decompression is started. @return : an error code, which can be tested using ZSTD_isError() if >0, a hint, nb of expected input bytes for next invocation. 0 means : a frame has just been fully decoded and flushed.

ZSTD_decompress_generic_simpleArgs() : Same as ZSTD_decompress_generic(), but using only integral types as arguments. Argument list is larger than ZSTD_{in,out}Buffer, but can be helpful for binders from dynamic languages which have troubles handling structures containing memory pointers.

ZSTD_decompress_usingDDict() : Decompression using a digested Dictionary. Faster startup than ZSTD_decompress_usingDict(), recommended when same dictionary is used multiple times.

ZSTD_decompress_usingDict() : Decompression using a predefined Dictionary (see dictBuilder/zdict.h). Dictionary must be identical to the one used during compression. Note : This function loads the dictionary, resulting in significant startup delay. Note : When dict == NULL || dictSize < 8 no dictionary is used.

ZSTD_estimate*() : These functions make it possible to estimate memory usage of a future {D,C}Ctx, before its creation. ZSTD_estimateCCtxSize() will provide a budget large enough for any compression level up to selected one. It will also consider src size to be arbitrarily "large", which is worst case. If srcSize is known to always be small, ZSTD_estimateCCtxSize_usingCParams() can provide a tighter estimation. ZSTD_estimateCCtxSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. ZSTD_estimateCCtxSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbWorkers is >= 1. Note : CCtx size estimation is only correct for single-threaded compression.

ZSTD_estimate?DictSize() : ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, like ZSTD_createCDict(). ZSTD_estimateCDictSize_advanced() makes it possible to control compression parameters precisely, like ZSTD_createCDict_advanced(). Note : dictionaries created by reference (ZSTD_dlm_byRef) are logically smaller.

ZSTD_estimateCStreamSize() : ZSTD_estimateCStreamSize() will provide a budget large enough for any compression level up to selected one. It will also consider src size to be arbitrarily "large", which is worst case. If srcSize is known to always be small, ZSTD_estimateCStreamSize_usingCParams() can provide a tighter estimation. ZSTD_estimateCStreamSize_usingCParams() can be used in tandem with ZSTD_getCParams() to create cParams from compressionLevel. ZSTD_estimateCStreamSize_usingCCtxParams() can be used in tandem with ZSTD_CCtxParam_setParameter(). Only single-threaded compression is supported. This function will return an error code if ZSTD_p_nbWorkers is >= 1. Note : CStream size estimation is only correct for single-threaded compression. ZSTD_DStream memory budget depends on window Size. This information can be passed manually, using ZSTD_estimateDStreamSize, or deducted from a valid frame Header, using ZSTD_estimateDStreamSize_fromFrame(); Note : if streaming is init with function ZSTD_init?Stream_usingDict(), an internal ?Dict will be created, which additional size is not estimated here. In this case, get total size by adding ZSTD_estimate?DictSize

ZSTD_findDecompressedSize() : src should point the start of a series of ZSTD encoded and/or skippable frames srcSize must be the exact size of this series (i.e. there should be a frame boundary exactly at srcSize bytes after src) @return : - decompressed size of all data in all successive frames

ZSTD_findFrameCompressedSize() : src should point to the start of a ZSTD encoded frame or skippable frame srcSize must be >= first frame size @return : the compressed size of the first frame starting at src, suitable to pass to ZSTD_decompress or similar, or an error code if input is invalid

ZSTD_frameHeaderSize() : srcSize must be >= ZSTD_frameHeaderSize_prefix. @return : size of the Frame Header, or an error code (if srcSize is too small)

ZSTD_freeCDict() : Function frees memory allocated by ZSTD_createCDict().

ZSTD_freeDDict() : Function frees memory allocated with ZSTD_createDDict()

ZSTD_getCParams() : @return ZSTD_compressionParameters structure for a selected compression level and estimated srcSize. estimatedSrcSize value is optional, select 0 if not known

ZSTD_getDecompressedSize() : NOTE: This function is now obsolete, in favor of ZSTD_getFrameContentSize(). Both functions work the same way, but ZSTD_getDecompressedSize() blends "empty", "unknown" and "error" results to the same return value (0), while ZSTD_getFrameContentSize() gives them separate return values. @return : decompressed size of src frame content if known and not empty, 0 otherwise.

ZSTD_getDictID_fromDDict() : Provides the dictID of the dictionary loaded into ddict. If @return == 0, the dictionary is not conformant to Zstandard specification, or empty. Non-conformant dictionaries can still be loaded, but as content-only dictionaries.

ZSTD_getDictID_fromDict() : Provides the dictID stored within dictionary. if @return == 0, the dictionary is not conformant with Zstandard specification. It can still be loaded, but as a content-only dictionary.

ZSTD_getDictID_fromFrame() : Provides the dictID required to decompressed the frame stored within src. If @return == 0, the dictID could not be decoded. This could for one of the following reasons :

ZSTD_getFrameHeader() : decode Frame Header, or requires larger srcSize. @return : 0, zfhPtr is correctly filled,

ZSTD_getFrameHeader_advanced() : same as ZSTD_getFrameHeader(), with added capability to select a format (like ZSTD_f_zstd1_magicless)

ZSTD_getParams() : same as ZSTD_getCParams(), but @return a full ZSTD_parameters object instead of sub-component ZSTD_compressionParameters. All fields of ZSTD_frameParameters are set to default : contentSize=1, checksum=0, noDictID=0

Advanced streaming functions

ZSTD_initStatic*() : Initialize an object using a pre-allocated fixed-size buffer. workspace: The memory area to emplace the object into. Provided pointer must be 8-bytes aligned. Buffer must outlive object. workspaceSize: Use ZSTD_estimate*Size() to determine how large workspace must be to support target scenario. @return : pointer to object (same address as workspace, just different type), or NULL if error (size too small, incorrect alignment, etc.) Note : zstd will never resize nor malloc() when using a static buffer. If the object requires more memory than available, zstd will just error out (typically ZSTD_error_memory_allocation). Note 2 : there is no corresponding "free" function. Since workspace is allocated externally, it must be freed externally too. Note 3 : cParams : use ZSTD_getCParams() to convert a compression level into its associated cParams. Limitation 1 : currently not compatible with internal dictionary creation, triggered by ZSTD_CCtx_loadDictionary(), ZSTD_initCStream_usingDict() or ZSTD_initDStream_usingDict(). Limitation 2 : static cctx currently not compatible with multi-threading. Limitation 3 : static dctx is incompatible with legacy support.

ZSTD_isFrame() : Tells if the content of buffer starts with a valid Frame Identifier. Note : Frame Identifier is 4 bytes. If size < 4, @return will always be 0. Note 2 : Legacy Frame Identifiers are considered valid only if Legacy Support is enabled. Note 3 : Skippable Frame Identifiers are considered valid.

ZSTD_resetCStream() : start a new compression job, using same parameters from previous job. This is typically useful to skip dictionary loading stage, since it will re-use it in-place.. Note that zcs must be init at least once before using ZSTD_resetCStream(). If pledgedSrcSize is not known at reset time, use macro ZSTD_CONTENTSIZE_UNKNOWN. If pledgedSrcSize > 0, its value must be correct, as it will be written in header, and controlled at the end. For the time being, pledgedSrcSize==0 is interpreted as "srcSize unknown" for compatibility with older programs, but it will change to mean "empty" in future version, so use macro ZSTD_CONTENTSIZE_UNKNOWN instead. @return : 0, or an error code (which can be tested using ZSTD_isError())

ZSTD_sizeof_*() : These functions give the current memory usage of selected object. Object memory usage can evolve when re-used.

Explicit context

Bulk processing dictionary API

Custom memory allocation : These prototypes make it possible to pass your own allocation/free functions. ZSTD_customMem is provided at creation time, using ZSTD_create*_advanced() variants listed below. All allocation/free operations will be completed using these custom variants instead of regular <stdlib.h> ones.

New advanced API (experimental)