Downloaded 261 times
Uploaded byOwen O'Malley
ORC File Introduction
The document describes the ORC file format. It discusses the file structure, stripe structure, file layout, integer and string column serialization, compression techniques, projection and predicate filtering capabilities, example file sizes, and compares ORC files to RCFiles and Trevni. The document was authored by Owen O'Malley of Hortonworks for December 2012.
More Related Content
Similar to ORC File Introduction
ORC File Introduction
- 1. ORC Files Owen O’Malley owen@hortonworks.com December2012 © Hortonworks Inc. 2012 Page 1
- 2. Top Level Page 2 © Hortonworks Inc. 2012
- 3. File Structure Page 3 © Hortonworks Inc. 2012
- 4. Stripe Structure Page 4 © Hortonworks Inc. 2012
- 5. File Layout Page 5 © Hortonworks Inc. 2012
- 6. Integer Column Serialization Page 6 © Hortonworks Inc. 2012
- 7. String Column Serialization Page 7 © Hortonworks Inc. 2012
- 8. Compression Page 8 © Hortonworks Inc. 2012
- 9. Projection and PredicateFiltering Page 9 © Hortonworks Inc. 2012
- 10. Example File Sizes Page 10 © Hortonworks Inc. 2012
- 11. Final notes Page 11 © Hortonworks Inc. 2012
- 12. Comparison RC File Trevni ORC File Hive Type Model N N Y Separate complex columns N Y Y Splits found quickly N Y Y Default column group size 4MB 64MB* 250MB Files per a bucket 1 >1 1 Store min, max, sum, count N N Y Versioned metadata N Y Y Run length data encoding N N Y Store strings in dictionary N N Y Store row count N Y Y Skip compressed blocks N N Y Store internal indexes N N Y Page 12 © Hortonworks Inc. 2012