What is it?
sval is a serialization-only framework for Rust. It has a simple, but expressive design that can express any Rust data structure, plus some it can't yet. It was originally designed as a niche framework for structured logging, targeting serialization to JSON, protobuf, and Rust's Debug-style formatting. The project has evolved beyond this point into a fully general and capable framework for introspecting runtime data.
The core of sval is the Stream trait. It defines the data model and features of the framework. sval makes a few different API design decisions compared to serde, the de-facto choice, to better accommodate the needs of Rust diagnostic frameworks:
- Simple API. The
Streamtrait has only a few required members that all values are forwarded to. This makes it easy to write bespoke handling for specific data types without needing to implement unrelated methods. dyn-friendly. TheStreamtrait is internally mutable, so is trivial to makedyn-compatible without intermediate boxing, making it possible to use in no-std environments.- Buffer-friendly. The
Streamtrait is non-recursive, so values can be buffered as a flat stream of tokens and replayed later. - Borrowing as an optimization. The
Streamtrait may accept borrowed text or binary fragments for a specific lifetime'sval, but is also required to accept temporary ones too. This makes it possible to optimize away allocations where possible, but still force them if it's required. - Broadly compatible.
svalimposes very few constraints of its own, so it can trivially translate implementations ofserde::Serializeinto implementations ofsval::Value.
sval's data model takes inspiration from CBOR, specifically:
- Small core. The base data model of
svalis small. The required members of theStreamtrait only includes nulls, booleans, text, 64-bit signed integers, and sequences. All other types, like arbitrary-precision floating point numbers, records, and tuples, are representable in the base model. - Extensible tags. Users can define tags that extend
sval's data model with new semantics. Examples of tags include Rust'sSomeandNonevariants, constant-sized arrays, text that doesn't require JSON escaping, and anything else you might need.
Getting started
This section is a high-level guided tour of sval's design and API. To get started, add sval to your Cargo.toml:
[]
= "2.16.0"
= ["derive"]
Serializing values
As a quick example, here's how you can use sval to serialize a runtime value as JSON.
First, add sval_json to your Cargo.toml:
[]
= "2.16.0"
= ["std"]
Next, derive the Value trait on the type you want to serialize, including on any other types it uses in its fields:
Finally, use stream_to_string to serialize an instance of your type as JSON:
let my_record = MyRecord ;
// Produces:
//
// {"field_0":1,"field_1":true,"field_2":"some text"}
let json: String = stream_to_string?;
The Value trait
The previous example didn't reveal a lot of detail about how sval works, only that there's a Value trait involved, and it somehow allows us to convert an instance of the MyRecord struct into a JSON object. Using cargo expand, we can peek behind the covers and see what the Value trait does. The previous example expands to something like this:
The Value trait has a single required method, stream, which is responsible for driving an instance of a Stream with its fields. The Stream trait defines sval's data model and the mechanics of how data is described in it. In this example, the MyRecord struct is represented as a record tuple, a type that can be either a record with fields named by a Label, or a tuple with fields indexed by an Index. Labels and indexes can be annotated with a Tag which add user-defined semantics to them. In this case, the labels carry the VALUE_IDENT tag meaning they're valid Rust identifiers, and the indexes carry the VALUE_OFFSET tag meanings they're zero-indexed field offsets. The specific type of Stream can decide whether to treat the MyRecord type as either a record (in the case of JSON) or a tuple (in the case of protobuf), and whether it understands that tags it sees or not.
The Stream trait
Something to notice about the Stream API in the expanded MyRecord example is that it is flat. The call to record_tuple_begin doesn't return a new type like serde's serialize_struct does. The implementor of Value is responsible for issuing the correct sequence of Stream calls as it works through its structure. The Stream can then rely on markers like record_tuple_value_begin and record_tuple_value_end to know what position within a value it is without needing to track that state itself. The flat API makes dyn-compatibility and buffering simpler, but makes implementing non-trivial streams more difficult, because you can't rely on recursive to manage state.
Recall the way MyRecord was converted into JSON earlier:
let json: String = stream_to_string?;
Internally, stream_to_string uses an instance of Stream that writes JSON tokens for each piece of the value it encounters. For example, record_tuple_begin and record_tuple_end will emit the corresponding { } characters for a JSON object.
sval's data model is layered. The required methods on Stream represent the base data model that more expressive constructs map down to. Here's what a minimal Stream that just formats values in sval's base data model looks like:
;
let my_record = MyRecord ;
// Prints:
//
// [["field_0",1,],["field_1",true,],["field_2","some text",],],
stream;
Recall that the MyRecord struct mapped to a record_tuple in sval's data model. record_tuples in turn are represented in the base data model as a sequence of 2-dimensional sequences where the first element is the field label and the second is its value.
Streams aren't limited to just serializing data into interchange formats. They can manipulate or interrogate a value any way it likes. Here's an example of a Stream that attempts to extract a specific field of a value as an i32:
let my_record = MyRecord ;
assert_eq!;
Streaming text
Strings in sval don't need to be streamed in a single call. As an example, say we have a template type like this:
;
If we wanted to serialize Template to a string, we could implement Value, handling each literal and property as a separate fragment:
When streamed as JSON, Template would produce something like this:
let template = Template;
// Produces:
//
// "some literal text and {x} and more literal text"
let json = stream_to_string?;
Borrowed data
The Stream trait carries a 'sval lifetime it can use to accept borrowed text and binary values. Borrowing in sval is an optimization. Even if a Stream uses a concrete 'sval lifetime, it still needs to handle computed values. Here's an example of a Stream that attempts to extract a borrowed string from a value by making use of the 'sval lifetime:
Implementations of Value should provide a Stream with borrowed data where possible, and only compute it if it needs to.
Error handling
sval's Error type doesn't carry any state of its own. It only signals early termination of the Stream which may be because its job is done, or because it failed. It's up to the Stream to carry whatever state it needs to provide meaningful errors.
Data model
This section descibes sval's data model in detail using examples in Rust syntax. Some types in sval's model aren't representable in Rust yet, so they use pseudo syntax.
Base model
Nulls
null
stream.null?;
Booleans
bool
stream.bool?;
64bit signed integers
i64
stream.i64?;
Text
stream.text_begin?;
stream.text_fragment?;
stream.text_fragment?;
stream.text_end?;
Note that sval text is an array of strings.
Sequences
stream.seq_begin?;
stream.seq_value_begin?;
stream.i64?;
stream.seq_value_end?;
stream.seq_value_begin?;
stream.bool?;
stream.seq_value_end?;
stream.seq_end?;
Note that Rust arrays are homogeneous, but sval sequences are heterogeneous.
Extended model
8bit unsigned integers
u8
stream.u8?;
8bit unsigned integers reduce to 64bit signed integers in the base model.
16bit unsigned integers
u16
stream.u16?;
16bit unsigned integers reduce to 64bit signed integers in the base model.
32bit unsigned integers
u32
stream.u32?;
32bit unsigned integers reduce to 64bit signed integers in the base model.
64bit unsigned integers
u64
stream.u64?;
64bit unsigned integers reduce to 64bit signed integers in the base data model if they fit, or base10 ASCII text if they don't.
128bit unsigned integers
u128
stream.u128?;
128bit unsigned integers reduce to 64bit signed integers in the base data model if they fit, or base10 ASCII text if they don't.
8bit signed integers
i8
stream.i8?;
8bit signed integers reduce to 64bit signed integers in the base model.
16bit signed integers
i16
stream.i16?;
16bit signed integers reduce to 64bit signed integers in the base model.
32bit signed integers
i32
stream.i32?;
32bit signed integers reduce to 64bit signed integers in the base model.
128bit signed integers
i128
stream.i128?;
128bit signed integers reduce to 64bit signed integers in the base data model if they fit, or base10 ASCII text if they don't.
32bit binary floating point numbers
f32
stream.f32?;
32bit binary floating point numbers reduce to base10 ASCII text in the base model.
64bit binary floating point numbers
f64
stream.f64?;
64bit binary floating point numbers reduce to base10 ASCII text in the base model.
Binary
stream.binary_begin?;
stream.binary_fragment?;
stream.binary_fragment?;
stream.binary_end?;
Binary values reduce to sequences of numbers in the base model.
Maps
stream.map_begin?;
stream.map_key_begin?;
stream.i64?;
stream.map_key_end?;
stream.map_value_begin?;
stream.bool?;
stream.map_value_end?;
stream.map_key_begin?;
stream.i64?;
stream.map_key_end?;
stream.map_value_begin?;
stream.bool?;
stream.map_value_end?;
stream.map_end?;
Note that most Rust maps are homogeneous, but sval maps are heterogeneous.
Maps reduce to a sequence of 2D sequences in the base model.
Tags
stream.tag?;
Tags reduce to null in the base model.
Tagged values
;
stream.tagged_begin?;
stream.i64?;
stream.tagged_end?;
Tagged values reduce to their wrapped value in the base model.
Tuples
stream.tuple_begin?;
stream.tuple_value_begin?;
stream.i64?;
stream.tuple_value_end?;
stream.tuple_value_begin?;
stream.bool?;
stream.tuple_value_end?;
stream.tuple_end?;
sval tuples may also be unnamed:
stream.tuple_begin?;
stream.tuple_value_begin?;
stream.i64?;
stream.tuple_value_end?;
stream.tuple_value_begin?;
stream.bool?;
stream.tuple_value_end?;
stream.tuple_end?;
Tuples reduce to sequences in the base model.
Records
stream.record_begin?;
stream.record_value_begin?;
stream.i64?;
stream.record_value_end?;
stream.record_value_begin?;
stream.bool?;
stream.record_value_end?;
stream.record_end?;
sval records may also be unnamed:
stream.record_begin?;
stream.record_value_begin?;
stream.i64?;
stream.record_value_end?;
stream.record_value_begin?;
stream.bool?;
stream.record_value_end?;
stream.record_end?;
Records reduce to a sequence of 2D sequences in the base model.
Enums
sval enums wrap a variant, which may be any of the following types:
- Tags
- Tagged values
- Records
- Tuples
- Enums
Tag
stream.enum_begin?;
stream.tag?;
stream.enum_end?;
Tagged
stream.enum_begin?;
stream.tagged_begin?;
stream.i64?;
stream.tagged_end?;
stream.enum_end?;
Tuple
stream.enum_begin?;
stream.tuple_begin?;
stream.tuple_value_begin?;
stream.i64?;
stream.tuple_value_end?;
stream.tuple_value_begin?;
stream.bool?;
stream.tuple_value_end?;
stream.tuple_end?;
stream.enum_end?;
Record
stream.enum_begin?;
stream.record_begin?;
stream.record_value_begin?;
stream.i64?;
stream.record_value_end?;
stream.record_value_begin?;
stream.bool?;
stream.record_value_end?;
stream.record_end?;
stream.enum_end?;
sval enum variants may also be unnamed:
stream.enum_begin?;
stream.tagged_begin?;
stream.i64?;
stream.tagged_end?;
stream.enum_end?;
stream.enum_begin?;
stream.tuple_begin?;
stream.tuple_value_begin?;
stream.i64?;
stream.tuple_value_end?;
stream.tuple_value_begin?;
stream.bool?;
stream.tuple_value_end?;
stream.tuple_end?;
stream.enum_end?;
a: i64, b: bool }>
stream.enum_begin?;
stream.record_begin?;
stream.record_value_begin?;
stream.i64?;
stream.record_value_end?;
stream.record_value_begin?;
stream.bool?;
stream.record_value_end?;
stream.record_end?;
stream.enum_end?;
sval enum variants may be other enums:
Tagged
stream.enum_begin?;
stream.enum_begin?;
stream.tagged_begin?;
stream.i64?;
stream.tagged_end?;
stream.enum_end?;
stream.enum_end?;
User-defined tags
sval tags, tagged values, records, tuples, enums, and their values can carry a user-defined Tag that alters their semantics. A Stream may understand a Tag and treat its annotated value differently, or it may ignore them. An example of a Tag is NUMBER, which is for text that encodes an arbitrary-precision decimal floating point number with a standardized format. A Stream may parse these numbers and encode them differently to regular text.
Here's an example of a user-defined Tag for treating integers as Unix timestamps, and a Stream that understands them:
// Define a tag as a constant.
//
// Tags are expected to have unique names.
//
// The rules of our tag are that 64bit unsigned integers that carry it are seconds since
// the Unix epoch.
pub const UNIX_TIMESTAMP: Tag = new;
// Derive `Value` on a type, annotating it with our tag.
//
// We could also implement `Value` manually using `stream.tagged_begin(Some(&UNIX_TIMESTAMP), ..)`.
;
// Here's an example of a `Stream` that understands our tag.
The Label and Index types can also carry a Tag. An example of a Tag you might use on a Label is VALUE_IDENT, for labels that hold a valid Rust identifier.
Type system
sval has an implicit structural type system based on the sequence of calls a Stream receives, and the values of any Label, Index, or Tag on them, with the following exceptions:
- Text type does not depend on the composition of fragments, or on their length.
- Binary type does not depend on the composition of fragments, or on their length.
- Sequences are untyped. Their type doesn't depend on the types of their elements, or on their length.
- Maps are untyped. Their type doesn't depend on the types of their keys or values, or on their length.
- Enums holding differently typed variants have the same type.
These rules may be better formalized in the future.
Ecosystem
sval is a general framework with specific serialization formats and utilities provided as external libraries:
sval_fmt: Colorized Rust-style debug formatting.sval_json: Serialize values as JSON in aserde-compatible format.sval_protobuf: Serialize values as protobuf messages.sval_serde: Convert betweenserdeandsval.sval_buffer: Losslessly buffers anyValueinto an owned, thread-safe variant.sval_flatten: Flatten the fields of a value onto its parent, like#[serde(flatten)].sval_nested: Buffersval's flatStreamAPI into a recursive one likeserde's. For types that#[derive(Value)], the translation is non-allocating.sval_ref: A variant ofValuefor types that are internally borrowed (likeMyType<'a>) instead of externally (like&'a MyType).