org.apache.orc.impl

Class MemoryManagerImpl

java.lang.Object

org.apache.orc.impl.MemoryManagerImpl

All Implemented Interfaces:

MemoryManager

Direct Known Subclasses:

MemoryManager

public class MemoryManagerImpl
extends Object
implements MemoryManager

Implements a memory manager that keeps a global context of how many ORC writers there are and manages the memory between them. For use cases with dynamic partitions, it is easy to end up with many writers in the same task. By managing the size of each allocation, we try to cut down the size of each allocation and keep the task from running out of memory. This class is not thread safe, but is re-entrant - ensure creation and all invocations are triggered from the same thread.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.orc.MemoryManager

MemoryManager.Callback

Constructor Summary

Constructors

Constructor and Description
MemoryManagerImpl(Configuration conf) Create the memory manager.

Method Summary

Modifier and Type	Method and Description
void	addedRow(int rows) Give the memory manager an opportunity for doing a memory check.
void	addWriter(Path path, long requestedAllocation, MemoryManager.Callback callback) Add a new writer's memory allocation to the pool.
double	getAllocationScale() The scaling factor for each allocation to ensure that the pool isn't oversubscribed.
long	getTotalMemoryPool() Get the total pool size that is available for ORC writers.
void	notifyWriters() Notify all of the writers that they should check their memory usage.
void	removeWriter(Path path) Remove the given writer from the pool.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MemoryManagerImpl

public MemoryManagerImpl(Configuration conf)

Create the memory manager.

Parameters:

conf - use the configuration to find the maximum size of the memory pool.

Method Detail

addWriter

public void addWriter(Path path,
                      long requestedAllocation,
                      MemoryManager.Callback callback)
               throws IOException

Add a new writer's memory allocation to the pool. We use the path as a unique key to ensure that we don't get duplicates.

Specified by:

addWriter in interface MemoryManager

Parameters:

path - the file that is being written

requestedAllocation - the requested buffer size

Throws:

IOException

removeWriter

public void removeWriter(Path path)
                  throws IOException

Remove the given writer from the pool.

Specified by:

removeWriter in interface MemoryManager

Parameters:

path - the file that has been closed

Throws:

IOException

getTotalMemoryPool

public long getTotalMemoryPool()

Get the total pool size that is available for ORC writers.

Returns:

the number of bytes in the pool

getAllocationScale

public double getAllocationScale()

The scaling factor for each allocation to ensure that the pool isn't oversubscribed.

Returns:

a fraction between 0.0 and 1.0 of the requested size that is available for each writer.

addedRow

public void addedRow(int rows)
              throws IOException

Give the memory manager an opportunity for doing a memory check.

Specified by:

addedRow in interface MemoryManager

Parameters:

rows - number of rows added

Throws:

IOException

notifyWriters

public void notifyWriters()
                   throws IOException

Notify all of the writers that they should check their memory usage.

Throws:

IOException