Downloaded 50 times
Uploaded byOwen O'Malley
ORC Deep Dive 2020
This document provides an overview of the ORC file format. It describes the key requirements and design decisions, including file structure, stripe structure, encoding columns, run length encoding, compression, indexing, and versioning. It also discusses optimizations, debugging, and using ORC from SQL, Java, C++, and the command line. The document is intended to help users and developers better understand how ORC works.
In this document
Powered by AI
Introduction to ORC presentation by Owen O'Malley, with contact details and overview.
Key requirements for ORC files: self-describing files, schema, file version, tight compression, column projection, easy division, and compatibility.
ORC file structure includes metadata in the footer, stripe information, and organization into stripes that contain columnar data.
The read path for ORC files involves reading the stripe footer and required streams, serialization of streams, and various compression techniques.
Data types and encoding methods including compound types and run-length encoding for efficient storage.
ORC's compression methods, indexing techniques for row pruning, and how to efficiently access row groups.
Bloom filters for value detection, version control, schema evolution, stripe concatenation, column encryption, and additional developer tools.
Instructions on using ORC in Hive, Spark, Java, C++, and command line tools for debugging and optimization.
Key optimization strategies including stripe size, HDFS block padding, and calculations for file splitting for performance improvements.Conclusion of the presentation and additional resources for more information on the ORC format and its specifications.
More Related Content
ORC Deep Dive 2020
- 1. ORC DEEP DIVE OwenO’Malley omalley@apache.org January 2020 @owen_omalley
- 2.
- 3. © 2019 Cloudera,Inc. All rights reserved. 3 REQUIREMENTS • Files had to be completely self describing • Schema • File version • Tight compression ⇒ Run Length Encoding (RLE) & compression • Column projection ⇒ segregate column data • Predicate pushdown ⇒ understand & index user’s types • Files had to be easy & fast to divide • Compatible with a write once file systems
- 4. © 2019 Cloudera,Inc. All rights reserved. 4 FILE STRUCTURE • The file footer contains: • Metadata – schema, file statistics • Stripe information – metadata and location of stripes • Postscript with the compression, buffer size, & file version • ORC file data is divided into stripes. • Stripes are self contained sets of rows organized by columns. • Stripes are the smallest unit of work for tasks. • Default is ~64MB, but often configured larger.
- 5. © 2019 Cloudera,Inc. All rights reserved. 5 STRIPE STRUCTURE • Within a stripe, the metadata data is in the stripe footer. • List of streams • Column encoding information (eg. direct or dictionary) • Columns are written as a set of streams. There are 3 kinds: • Index streams • Data streams • Dictionary streams
- 6. © 2019 Cloudera,Inc. All rights reserved. 6 FILE STRUCTURE
- 7. © 2019 Cloudera,Inc. All rights reserved. 7 READ PATH • The Reader reads last 16k of file, extra as needed • The RowReader reads • Stripe footer • Required streams
- 8. © 2019 Cloudera,Inc. All rights reserved. 8 STREAMS • Streams are an independent sequence of bytes • Serialization into streams depends on column type & encoding • Optional pipeline stages: • Run Length Encoding (RLE) – first pass integer compression • Generic compression – Zlib, Snappy, LZO, Zstd • Encryption – AES/CTR
- 9.
- 10. © 2019 Cloudera,Inc. All rights reserved. 10 COMPOUND TYPES • Compound types are serialized as trees of columns. • struct, list, map, uniontype all have child columns • Types are numbered in a preorder traversal • The column reading classes are called TreeReadera: int, b: map<string, struct<c: string, d: double>>, e: timestamp
- 11. © 2019 Cloudera,Inc. All rights reserved. 11 ENCODING COLUMNS • To interpret a stream, you need three pieces of information: • Column type • Column encoding (direct, dictionary) • Stream kind (present, data, length, etc.) • All columns, if they have nulls, will have a present stream • Serialized using a boolean RLE • Integer columns are serialized with • A data stream using integer RLE
- 12. © 2019 Cloudera,Inc. All rights reserved. 12 ENCODING COLUMNS • Binary columns are serialized with: • Length stream of integer RLE • Data stream of raw sequence of bytes • String columns may be direct or dictionary encoded • Direct looks like binary column, but dictionary is different • Dictionary_data is raw sequence of dictionary bytes • Length is an integer RLE stream of the dictionary lengths • Data is an integer RLE stream of indexes into dictionary
- 13. © 2019 Cloudera,Inc. All rights reserved. 13 ENCODING COLUMNS • Lists and maps record the number of child elements • Length is an integer RLE stream • Structs only have the present stream • Timestamps need nanosecond resolution (ouch!) • Data is an integer RLE of seconds from Jan 2015 • Secondary is an integer RLE of nanoseconds with 0 suppress
- 14. © 2019 Cloudera,Inc. All rights reserved. 14 RUN LENGTH ENCODING • Goal is to get some cheap quick compression • Handles repeating/incrementing values • Handles integer byte packing • Two versions • Version 1 – relative simple repeat/literal encoding • Version 2 – complex encoding with 4 variants • Column encoding of *_V2 means use RLE version 2
- 15.
- 16. © 2019 Cloudera,Inc. All rights reserved. 16 ROW PRUNING • Three levels of indexing/row pruning • File – uses file statistics in file footer • Stripe – uses stripe statistics before file footer • Row group (default of 10k rows) – uses index stream • The index stream for each column includes for each row group • Column statistics (min, max, count, sum) • The start positions of each stream
- 17. © 2019 Cloudera,Inc. All rights reserved. 17 SEARCH ARGUMENTS • Engines can pass Search Arguments (SArgs) to the RowReader. • Limited set of operations (=, <=>, <, <=, in, between, is null) • Compare one column to literal(s) • Can only eliminate entire row groups, stripes, or files. • Engine must still filter the individual rows afterwards • For Hive, ensure hive.optimize.index.filter is true.
- 18. © 2019 Cloudera,Inc. All rights reserved. 18 COMPRESSION • All of the generic compression is done in chunks • Codec is reinitialized at start of chunk • Each chunk is compressed separately • Each uncompressed chunk is at most the buffer size • Each chunk has a 3 byte header giving: • Compressed size of chunk • Whether it is the original or compressed
- 19. © 2019 Cloudera,Inc. All rights reserved. 19 INDEXES • Wanted ability to seek to each row group • Allows fine grain seeking & row pruning • Could have flushed stream compression pipeline • Would have dramatically lowered compression • Instead treat compression & RLE has gray boxes • Use our knowledge of compression & RLE • Always start fresh at beginning of chunk or run
- 20. © 2019 Cloudera,Inc. All rights reserved. 20 INDEX POSITIONS • Records information to seek to a given row in all of a column’s streams • Includes: • C Compressed bytes • U Uncompressed bytes • V RLE values • C, U, & V jump to RG 4
- 21. © 2019 Cloudera,Inc. All rights reserved. 21 BLOOM FILTERS • For use cases where you need to find particular values • Sorting by that column allows min/max filtering • But you can only sort on one column effectively • Bloom filters are probabilistic data structures • Only useful for equality, not less than or greater than • Need ~10 bits/distinct value ⇒ opt in • ORC uses a bloom_filter_utf8 stream to record a bloom filter per a row group
- 22. © 2019 Cloudera,Inc. All rights reserved. 22 ROW PRUNING EXAMPLE • TPC-DS from tpch1000.lineitem where l_orderkey = 1212000001; Index Rows Read Time Nothing 5,999,989,709 74 sec Min/Max 540,000 4.5 sec Bloom 10,000 1.3 sec
- 23.
- 24. © 2019 Cloudera,Inc. All rights reserved. 24 COMPATIBILITY • Within a file version, old readers must be able to read all files. • A few exceptions (eg. new codecs, types) • Version 0 (from Hive 0.11) • Only RLE V1 & string dictionary encoding • Version 1 (from Hive 0.12 forward) • Version 2 (under development) • The library includes ability to write any file version. • Enables smooth upgrades across clusters
- 25. © 2019 Cloudera,Inc. All rights reserved. 25 WRITER VERSION • When fixes or feature additions are made to the writer, we bump the writer version. • Allows reader to work around bugs, especially in index • Does not affect reader compatibility • We should require each minor version adds a new one. • We also record which writer wrote the file: • Java, C++, Presto, Go
- 26. © 2019 Cloudera,Inc. All rights reserved. 26 EXAMPLE WORKAROUND FOR HIVE-8746 • Timestamps suck! • ORC uses an epoch of 01-01-2015 00:00:00. • Timestamp columns record seconds offset from epoch • Unfortunately, the original code use local time zone. • If reader and writer were in time zones with the same rules, it worked. • Fix involved writing the writer time zone into file. • Forwards and backwards compatible
- 27.
- 28. © 2019 Cloudera,Inc. All rights reserved. 28 SCHEMA EVOLUTION • User passes desired schema to RecordReader factory. • SchemaEvolution class maps between file & reader schemas. • The mapping can be positional or name based. • Conversions based on legacy Hive behavior… • The RecordReader uses the mapping to translate • Choosing streams uses the file schema column ids • Type translation is done by ConvertTreeReaderFactory. • Adds an additional TreeReader that does conversion.
- 29. © 2019 Cloudera,Inc. All rights reserved. 29 STRIPE CONCATENATION & FLUSH • ORC has a special operator to concatenate files • Requires consistent options & schema • Concatenates stripes without reserialization • ORC can flush the current contents including a file footer while still writing to the file. • Writes a side file with the current offset of the file tail • When the file closes the intermediate file footers are ignored
- 30. © 2019 Cloudera,Inc. All rights reserved. 30 COLUMN ENCRYPTION • Released in ORC 1.6 • Allows consistent column level access control across engines • Writes two variants of data • Encrypted original • Unencrypted statically masked • Each variant has its own streams & encodings • Each column has a unique local key, which is encrypted by KMS
- 31. © 2019 Cloudera,Inc. All rights reserved. 31 OTHER DEVELOPER TOOLS • Benchmarks • Hive & Spark • Avro, Json, ORC, and Parquet • Three data sets (taxi, sales, github) • Docker • Allows automated builds on all supported Linux variants • Site source code is with C++ & Java
- 32.
- 33. © 2019 Cloudera,Inc. All rights reserved. 33 WHICH VERSION IS IT? Engine Version ORC Version Hive 0.11 to 2.2 Hive ORC 0.11 to 2.2 2.3 ORC 1.3 3.0 ORC 1.4 3.1 ORC 1.5 Spark hive * Hive ORC 1.2 Spark native 2.3 ORC 1.4 2.4 to 3.0 ORC 1.5
- 34. © 2019 Cloudera,Inc. All rights reserved. 34 FROM SQL • Hive: • Add “stored as orc” to table definition • Table properties override configuration for ORC • Spark’s “spark.sql.orc.impl” controls implementation • native – Use ORC 1.5 • hive – Use ORC from Hive 1.2
- 35. © 2019 Cloudera,Inc. All rights reserved. 35 FROM JAVA • Use the ORC project rather than Hive’s ORC. • Maven group id: org.apache.orc version: 1.6.2 • nohive classifier avoids interfering with Hive’s packages • Two levels of access • orc-core – Faster access, but uses Hive’s vectorized API • orc-mapreduce – Row by row access, simpler OrcStruct API • MapReduce API implements WritableComparable • Can be shuffled • Need to specify type information in configuration for shuffle or output
- 36. © 2019 Cloudera,Inc. All rights reserved. 36 FROM C++ • Pure C++ client library • No JNI or JDK so client can estimate and control memory • Uses pure C++ HDFS client from HDFS-8707 • Reader and writer are stable and in production use. • Runs on Linux, Mac OS, and Windows. • Docker scripts for CentOS 6-8, Debian 8-10, Ubuntu 14-18 • CI builds on Mac OS, Ubuntu, and Windows
- 37. © 2019 Cloudera,Inc. All rights reserved. 37 FROM COMMAND LINE • Using hive –orcfiledump from Hive • -j -p – pretty prints the metadata as JSON • -d – prints data as JSON • Using java -jar orc-tools-*-uber.jar from ORC • meta -j -p – print the metadata as JSON • data – print data as JSON • convert – convert CSV, JSON, or ORC to ORC • json-schema – scan a set of JSON documents to find schema
- 38. © 2019 Cloudera,Inc. All rights reserved. 38 DEBUGGING • Things to look for: • Stripe size • Rows/Stripe • File version • Writer version • Width of schema • Sanity of statistics • Column encoding • Size of dictionaries
- 39.
- 40. © 2019 Cloudera,Inc. All rights reserved. 40 STRIPE SIZE • Makes a huge difference in performance • orc.stripe.size or hive.exec.orc.default.stripe.size • Controls the amount of buffer in writer. Default is 64MB • Trade off • Large = Large more efficient reads • Small = Less memory and more granular processing splits • Multiple files written at the same time will shrink stripes
- 41. © 2019 Cloudera,Inc. All rights reserved. 41 HDFS BLOCK PADDING • The stripes don’t align exactly with HDFS blocks • Unless orc.write.variable.length.blocks • HDFS scatters blocks around cluster • Often want to pad to block boundaries • Costs space, but improves performance • orc.default.block.padding • orc.block.padding.tolerance
- 42. © 2019 Cloudera,Inc. All rights reserved. 42 SPLIT CALCULATION • BI Small fast queries Splits based on HDFS blocks • ETL Large queries Read file footer and apply SearchArg to stripes Can include footer in splits (hive.orc.splits.include.file.footer) • Hybrid If small files or lots of files, use BI
- 43.
- 44. © 2019 Cloudera,Inc. All rights reserved. 44 FOR MORE INFORMATION • The orc_proto.proto defines the ORC metadata • Read code and especially OrcConf, which has all of the knobs • Website on https://orc.apache.org/ • /bugs ⇒ jira repository • /src ⇒ github repository • /specification ⇒ format specification • Apache email list dev@orc.apache.org
- 45.