org.apache.orc.impl.writer

Class WriterImplV2

java.lang.Object

org.apache.orc.impl.writer.WriterImplV2

All Implemented Interfaces:

Closeable, AutoCloseable, WriterInternal, MemoryManager.Callback, Writer

public class WriterImplV2
extends Object
implements WriterInternal, MemoryManager.Callback

An ORCv2 file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams. This class is unsynchronized like most Stream objects, so from the creation of an OrcFile and all access to a single instance has to be from a single thread. There are no known cases where these happen between different threads today. Caveat: the MemoryManager is created during WriterOptions create, that has to be confined to a single thread as well.

Constructor Summary

Constructors

Constructor and Description
WriterImplV2(FileSystem fs, Path path, OrcFile.WriterOptions opts)

Method Summary

Modifier and Type	Method and Description
void	addRowBatch(VectorizedRowBatch batch) Add a row batch to the ORC file.
void	addUserMetadata(String name, ByteBuffer value) Add arbitrary meta-data to the ORC file.
void	appendStripe(byte[] stripe, int offset, int length, StripeInformation stripeInfo, OrcProto.StripeStatistics stripeStatistics) Fast stripe append to ORC file.
void	appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata) When fast stripe append is used for merging ORC stripes, after appending the last stripe from a file, this interface must be used to merge any user metadata.
boolean	checkMemory(double newScale) The writer needs to check its memory usage
void	close() Flush all of the buffers and close the file.
CompressionCodec	getCompressionCodec()
long	getNumberOfRows() Row count gets updated when flushing the stripes.
long	getRawDataSize() Raw data size will be compute when writing the file footer.
TypeDescription	getSchema() Get the schema for this writer
ColumnStatistics[]	getStatistics() Get the statistics about the columns in the file.
void	increaseCompressionSize(int newSize) Increase the buffer size for this writer.
long	writeIntermediateFooter() Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WriterImplV2

public WriterImplV2(FileSystem fs,
                    Path path,
                    OrcFile.WriterOptions opts)
             throws IOException

Throws:

IOException

Method Detail

checkMemory

public boolean checkMemory(double newScale)
                    throws IOException

Description copied from interface: MemoryManager.Callback

The writer needs to check its memory usage

Specified by:

checkMemory in interface MemoryManager.Callback

Parameters:

newScale - the current scale factor for memory allocations

Returns:

true if the writer was over the limit

Throws:

IOException

increaseCompressionSize

public void increaseCompressionSize(int newSize)

Description copied from interface: WriterInternal

Increase the buffer size for this writer. This function is internal only and should only be called by the ORC file merger.

Specified by:

increaseCompressionSize in interface WriterInternal

Parameters:

newSize - the new buffer size.

getSchema

public TypeDescription getSchema()

Description copied from interface: Writer

Get the schema for this writer

Specified by:

getSchema in interface Writer

Returns:

the file schema

addUserMetadata

public void addUserMetadata(String name,
                            ByteBuffer value)

Description copied from interface: Writer

Add arbitrary meta-data to the ORC file. This may be called at any point until the Writer is closed. If the same key is passed a second time, the second value will replace the first.

Specified by:

addUserMetadata in interface Writer

Parameters:

name - a key to label the data with.

value - the contents of the metadata.

addRowBatch

public void addRowBatch(VectorizedRowBatch batch)
                 throws IOException

Description copied from interface: Writer

Add a row batch to the ORC file.

Specified by:

addRowBatch in interface Writer

Parameters:

batch - the rows to add

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: Writer

Flush all of the buffers and close the file. No methods on this writer should be called afterwards.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Writer

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.

Specified by:

getRawDataSize in interface Writer

Returns:

raw data size

getNumberOfRows

public long getNumberOfRows()

Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.

Specified by:

getNumberOfRows in interface Writer

Returns:

row count

writeIntermediateFooter

public long writeIntermediateFooter()
                             throws IOException

Description copied from interface: Writer

Write an intermediate footer on the file such that if the file is truncated to the returned offset, it would be a valid ORC file.

Specified by:

writeIntermediateFooter in interface Writer

Returns:

the offset that would be a valid end location for an ORC file

Throws:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         StripeInformation stripeInfo,
                         OrcProto.StripeStatistics stripeStatistics)
                  throws IOException

Description copied from interface: Writer

Fast stripe append to ORC file. This interface is used for fast ORC file merge with other ORC files. When merging, the file to be merged should pass stripe in binary form along with stripe information and stripe statistics. After appending last stripe of a file, use appendUserMetadata() to append any user metadata.

Specified by:

appendStripe in interface Writer

Parameters:

stripe - - stripe as byte array

offset - - offset within byte array

length - - length of stripe within byte array

stripeInfo - - stripe information

stripeStatistics - - stripe statistics (Protobuf objects can be merged directly)

Throws:

IOException

appendUserMetadata

public void appendUserMetadata(List<OrcProto.UserMetadataItem> userMetadata)

Description copied from interface: Writer

When fast stripe append is used for merging ORC stripes, after appending the last stripe from a file, this interface must be used to merge any user metadata.

Specified by:

appendUserMetadata in interface Writer

Parameters:

userMetadata - - user metadata

getStatistics

public ColumnStatistics[] getStatistics()
                                 throws IOException

Description copied from interface: Writer

Get the statistics about the columns in the file. The output of this is based on the time at which it is called. It shall use all of the currently written data to provide the statistics. Please note there are costs involved with invoking this method and should be used judiciously.

Specified by:

getStatistics in interface Writer

Returns:

the information about the column

Throws:

IOException

getCompressionCodec

public CompressionCodec getCompressionCodec()