Yore

Rust library for decoding/encoding character sets according to OEM code pages.

Features

• Fast *
• Minimal memory usage due to Cow and shrink_to_fit
• Simple API
• Many supported code pages
• Supports code pages with redefined ascii(<0x80), for example '٪' in CP864

Usage

Add yore to Cargo.toml.

[dependencies]
yore = "0.3.0"

Examples

Use specific code page

use yore::code_pages::{CP857, CP850};
use yore::{DecodeError, EncodeError};

// Vec contains ascii "text"
let bytes = vec![116, 101, 120, 116];
// Vec contains ascii "text " and codepoint 231
let bytes_undefined = vec![116, 101, 120, 116, 32, 231]; 

// notice that decoding CP850 can't fail because it is completely defined
assert_eq!(CP850.decode(&bytes), "text");

// But CP857 can fail
assert_eq!(CP857.decode(&bytes).unwrap(), "text");

//"text " + codepoint 231 
assert!(matches!(CP857.decode(&bytes_undefined), DecodeError));

// lossy decoding will not fail because of fallback
assert_eq!(CP857.decode_lossy(&bytes_undefined), "text �");

// encoding
assert_eq!(CP850.encode("text").unwrap(), bytes);
assert!(matches!(CP850.encode("text 🦀"), EncodeError));
assert_eq!(CP850.encode_lossy("text 🦀", 231), bytes_undefined);

Use trait object

use yore::CodePage;
fn do_something(code_page: &dyn CodePage, bytes: &[u8]) {
    println!("{}", code_page.decode(bytes).unwrap());
}

Supported code pages

* Benchmarks

encoding_rs supports only a few of the encodings that oem_cp and yore supports. Furthermore, the use case of encoding_rs is focused on streaming.

See bench crate for details