org.apache.orc.impl

Class ReaderImpl

java.lang.Object

org.apache.orc.impl.ReaderImpl

All Implemented Interfaces:

Reader

public class ReaderImpl
extends Object
implements Reader

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReaderImpl.StripeInformationImpl

Nested classes/interfaces inherited from interface org.apache.orc.Reader

Reader.Options

Field Summary

Fields

Modifier and Type	Field and Description
protected int	bufferSize
protected CompressionKind	compressionKind
protected Configuration	conf
protected FileSystem	fileSystem
protected OrcProto.Metadata	metadata
protected Path	path
protected int	rowIndexStride
protected OrcTail	tail
protected List<OrcProto.Type>	types
protected boolean	useUTCTimestamp

Constructor Summary

Constructors

Constructor and Description
ReaderImpl(Path path, OrcFile.ReaderOptions options) Constructor that let's the user specify additional options.

Method Summary

Modifier and Type	Method and Description
protected static void	checkOrcVersion(Path path, OrcProto.PostScript postscript) Check to see if this ORC file is from a future version and if so, warn the user that we may not be able to read all of the column encodings.
static ColumnStatistics[]	deserializeStats(TypeDescription schema, List<OrcProto.ColumnStatistics> fileStats)
protected static void	ensureOrcFooter(ByteBuffer buffer, int psLen) Ensure this is an ORC file to prevent users from trying to read text files or RC files as ORC files.
protected static void	ensureOrcFooter(FSDataInputStream in, Path path, int psLen, ByteBuffer buffer) Ensure this is an ORC file to prevent users from trying to read text files or RC files as ORC files.
static OrcTail	extractFileTail(ByteBuffer buffer)
static OrcTail	extractFileTail(ByteBuffer buffer, long fileLength, long modificationTime)
protected OrcTail	extractFileTail(FileSystem fs, Path path, long maxFileLength)
static OrcProto.Metadata	extractMetadata(ByteBuffer bb, int metadataAbsPos, int metadataSize, CompressionCodec codec, int bufferSize)
CompressionKind	getCompressionKind() Get the compression kind.
int	getCompressionSize() Get the buffer size for the compression.
long	getContentLength() Get the length of the file.
OrcProto.FileTail	getFileTail() Get the file tail (footer + postscript)
OrcFile.Version	getFileVersion() Get the file format version.
static OrcFile.Version	getFileVersion(List<Integer> versionList)
List<String>	getMetadataKeys() Get the user metadata keys.
int	getMetadataSize()
ByteBuffer	getMetadataValue(String key) Get a user metadata value.
long	getNumberOfRows() Get the number of rows in the file.
List<OrcProto.ColumnStatistics>	getOrcProtoFileStatistics()
List<OrcProto.StripeStatistics>	getOrcProtoStripeStatistics()
List<OrcProto.UserMetadataItem>	getOrcProtoUserMetadata()
long	getRawDataSize() Get the deserialized data size of the file
long	getRawDataSizeFromColIndices(List<Integer> colIndices) Get the deserialized data size of the specified columns ids
static long	getRawDataSizeFromColIndices(List<Integer> colIndices, List<OrcProto.Type> types, List<OrcProto.ColumnStatistics> stats)
long	getRawDataSizeOfColumns(List<String> colNames) Get the deserialized data size of the specified columns
int	getRowIndexStride() Get the number of rows per a entry in the row index.
TypeDescription	getSchema() Get the type of rows in this ORC file.
ByteBuffer	getSerializedFileFooter()
ColumnStatistics[]	getStatistics() Get the statistics about the columns in the file.
List<StripeInformation>	getStripes() Get the list of stripes.
List<StripeStatistics>	getStripeStatistics()
List<OrcProto.Type>	getTypes() Get the list of types contained in the file.
List<Integer>	getVersionList()
OrcFile.WriterVersion	getWriterVersion() Get the version of the writer of this file.
static OrcFile.WriterVersion	getWriterVersion(int writerVersion) Get the WriterVersion based on the ORC file postscript.
boolean	hasMetadataValue(String key) Did the user set the given metadata value.
Reader.Options	options() Create a default options object that can be customized for creating a RecordReader.
RecordReader	rows() Create a RecordReader that reads everything with the default options.
RecordReader	rows(Reader.Options options) Create a RecordReader that uses the options given.
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

fileSystem

protected final FileSystem fileSystem

path

protected final Path path

compressionKind

protected final CompressionKind compressionKind

bufferSize

protected int bufferSize

metadata

protected OrcProto.Metadata metadata

types

protected final List<OrcProto.Type> types

rowIndexStride

protected final int rowIndexStride

conf

protected final Configuration conf

useUTCTimestamp

protected final boolean useUTCTimestamp

tail

protected OrcTail tail

Constructor Detail

ReaderImpl

public ReaderImpl(Path path,
                  OrcFile.ReaderOptions options)
           throws IOException

Constructor that let's the user specify additional options.

Parameters:

path - pathname for file

options - options for reading

Throws:

IOException

Method Detail

getNumberOfRows

public long getNumberOfRows()

Description copied from interface: Reader

Get the number of rows in the file.

Specified by:

getNumberOfRows in interface Reader

Returns:

the number of rows

getMetadataKeys

public List<String> getMetadataKeys()

Description copied from interface: Reader

Get the user metadata keys.

Specified by:

getMetadataKeys in interface Reader

Returns:

the set of metadata keys

getMetadataValue

public ByteBuffer getMetadataValue(String key)

Description copied from interface: Reader

Get a user metadata value.

Specified by:

getMetadataValue in interface Reader

Parameters:

key - a key given by the user

Returns:

the bytes associated with the given key

hasMetadataValue

public boolean hasMetadataValue(String key)

Description copied from interface: Reader

Did the user set the given metadata value.

Specified by:

hasMetadataValue in interface Reader

Parameters:

key - the key to check

Returns:

true if the metadata value was set

getCompressionKind

public CompressionKind getCompressionKind()

Description copied from interface: Reader

Get the compression kind.

Specified by:

getCompressionKind in interface Reader

Returns:

the kind of compression in the file

getCompressionSize

public int getCompressionSize()

Description copied from interface: Reader

Get the buffer size for the compression.

Specified by:

getCompressionSize in interface Reader

Returns:

number of bytes to buffer for the compression codec.

getStripes

public List<StripeInformation> getStripes()

Description copied from interface: Reader

Get the list of stripes.

Specified by:

getStripes in interface Reader

Returns:

the information about the stripes in order

getContentLength

public long getContentLength()

Description copied from interface: Reader

Get the length of the file.

Specified by:

getContentLength in interface Reader

Returns:

the number of bytes in the file

getTypes

public List<OrcProto.Type> getTypes()

Description copied from interface: Reader

Get the list of types contained in the file. The root type is the first type in the list.

Specified by:

getTypes in interface Reader

Returns:

the list of flattened types

getFileVersion

public static OrcFile.Version getFileVersion(List<Integer> versionList)

getFileVersion

public OrcFile.Version getFileVersion()

Description copied from interface: Reader

Get the file format version.

Specified by:

getFileVersion in interface Reader

getWriterVersion

public OrcFile.WriterVersion getWriterVersion()

Description copied from interface: Reader

Get the version of the writer of this file.

Specified by:

getWriterVersion in interface Reader

getFileTail

public OrcProto.FileTail getFileTail()

Description copied from interface: Reader

Get the file tail (footer + postscript)

Specified by:

getFileTail in interface Reader

Returns:

- file tail

getRowIndexStride

public int getRowIndexStride()

Description copied from interface: Reader

Get the number of rows per a entry in the row index.

Specified by:

getRowIndexStride in interface Reader

Returns:

the number of rows per an entry in the row index or 0 if there is no row index.

getStatistics

public ColumnStatistics[] getStatistics()

Description copied from interface: Reader

Get the statistics about the columns in the file.

Specified by:

getStatistics in interface Reader

Returns:

the information about the column

deserializeStats

public static ColumnStatistics[] deserializeStats(TypeDescription schema,
                                                  List<OrcProto.ColumnStatistics> fileStats)

getSchema

public TypeDescription getSchema()

Description copied from interface: Reader

Get the type of rows in this ORC file.

Specified by:

getSchema in interface Reader

ensureOrcFooter

protected static void ensureOrcFooter(FSDataInputStream in,
                                      Path path,
                                      int psLen,
                                      ByteBuffer buffer)
                               throws IOException

Ensure this is an ORC file to prevent users from trying to read text files or RC files as ORC files.

Parameters:

in - the file being read

path - the filename for error messages

psLen - the postscript length

buffer - the tail of the file

Throws:

IOException

ensureOrcFooter

protected static void ensureOrcFooter(ByteBuffer buffer,
                                      int psLen)
                               throws IOException

Ensure this is an ORC file to prevent users from trying to read text files or RC files as ORC files.

Parameters:

psLen - the postscript length

buffer - the tail of the file

Throws:

IOException

checkOrcVersion

protected static void checkOrcVersion(Path path,
                                      OrcProto.PostScript postscript)
                               throws IOException

Check to see if this ORC file is from a future version and if so, warn the user that we may not be able to read all of the column encodings.

Parameters:

path - the data source path for error messages

postscript - the parsed postscript

Throws:

IOException

getWriterVersion

public static OrcFile.WriterVersion getWriterVersion(int writerVersion)

Get the WriterVersion based on the ORC file postscript.

Parameters:

writerVersion - the integer writer version

Returns:

the version of the software that produced the file

extractMetadata

public static OrcProto.Metadata extractMetadata(ByteBuffer bb,
                                                int metadataAbsPos,
                                                int metadataSize,
                                                CompressionCodec codec,
                                                int bufferSize)
                                         throws IOException

Throws:

IOException

extractFileTail

public static OrcTail extractFileTail(ByteBuffer buffer)
                               throws IOException

Throws:

IOException

extractFileTail

public static OrcTail extractFileTail(ByteBuffer buffer,
                                      long fileLength,
                                      long modificationTime)
                               throws IOException

Throws:

IOException

extractFileTail

protected OrcTail extractFileTail(FileSystem fs,
                                  Path path,
                                  long maxFileLength)
                           throws IOException

Throws:

IOException

getSerializedFileFooter

public ByteBuffer getSerializedFileFooter()

Specified by:

getSerializedFileFooter in interface Reader

Returns:

Serialized file metadata read from disk for the purposes of caching, etc.

options

public Reader.Options options()

Description copied from interface: Reader

Create a default options object that can be customized for creating a RecordReader.

Specified by:

options in interface Reader

Returns:

a new default Options object

rows

public RecordReader rows()
                  throws IOException

Description copied from interface: Reader

Create a RecordReader that reads everything with the default options.

Specified by:

rows in interface Reader

Returns:

a new RecordReader

Throws:

IOException

rows

public RecordReader rows(Reader.Options options)
                  throws IOException

Description copied from interface: Reader

Create a RecordReader that uses the options given. This method can't be named rows, because many callers used rows(null) before the rows() method was introduced.

Specified by:

rows in interface Reader

Parameters:

options - the options to read with

Returns:

a new RecordReader

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Description copied from interface: Reader

Get the deserialized data size of the file

Specified by:

getRawDataSize in interface Reader

Returns:

raw data size

getRawDataSizeFromColIndices

public long getRawDataSizeFromColIndices(List<Integer> colIndices)

Description copied from interface: Reader

Get the deserialized data size of the specified columns ids

Specified by:

getRawDataSizeFromColIndices in interface Reader

Parameters:

colIndices - - internal column id (check orcfiledump for column ids)

Returns:

raw data size of columns

getRawDataSizeFromColIndices

public static long getRawDataSizeFromColIndices(List<Integer> colIndices,
                                                List<OrcProto.Type> types,
                                                List<OrcProto.ColumnStatistics> stats)

getRawDataSizeOfColumns

public long getRawDataSizeOfColumns(List<String> colNames)

Description copied from interface: Reader

Get the deserialized data size of the specified columns

Specified by:

getRawDataSizeOfColumns in interface Reader

Returns:

raw data size of columns

getOrcProtoStripeStatistics

public List<OrcProto.StripeStatistics> getOrcProtoStripeStatistics()

Specified by:

getOrcProtoStripeStatistics in interface Reader

Returns:

Stripe statistics, in original protobuf form.

getOrcProtoFileStatistics

public List<OrcProto.ColumnStatistics> getOrcProtoFileStatistics()

Specified by:

getOrcProtoFileStatistics in interface Reader

Returns:

File statistics, in original protobuf form.

getStripeStatistics

public List<StripeStatistics> getStripeStatistics()
                                           throws IOException

Specified by:

getStripeStatistics in interface Reader

Returns:

Stripe statistics.

Throws:

IOException

getOrcProtoUserMetadata

public List<OrcProto.UserMetadataItem> getOrcProtoUserMetadata()

getVersionList

public List<Integer> getVersionList()

Specified by:

getVersionList in interface Reader

Returns:

List of integers representing version of the file, in order from major to minor.

getMetadataSize

public int getMetadataSize()

Specified by:

getMetadataSize in interface Reader

Returns:

Gets the size of metadata, in bytes.

toString

public String toString()

Overrides:

toString in class Object