org.apache.orc

Interface MemoryManager

All Known Implementing Classes:

MemoryManager, MemoryManagerImpl

public interface MemoryManager

A memory manager that keeps a global context of how many ORC writers there are and manages the memory between them. For use cases with dynamic partitions, it is easy to end up with many writers in the same task. By managing the size of each allocation, we try to cut down the size of each allocation and keep the task from running out of memory. This class is not thread safe, but is re-entrant - ensure creation and all invocations are triggered from the same thread.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static interface	MemoryManager.Callback

Method Summary

Modifier and Type	Method and Description
void	addedRow(int rows) Give the memory manager an opportunity for doing a memory check.
void	addWriter(Path path, long requestedAllocation, MemoryManager.Callback callback) Add a new writer's memory allocation to the pool.
void	removeWriter(Path path) Remove the given writer from the pool.

Method Detail

addWriter

void addWriter(Path path,
               long requestedAllocation,
               MemoryManager.Callback callback)
        throws IOException

Add a new writer's memory allocation to the pool. We use the path as a unique key to ensure that we don't get duplicates.

Parameters:

path - the file that is being written

requestedAllocation - the requested buffer size

Throws:

IOException

removeWriter

void removeWriter(Path path)
           throws IOException

Remove the given writer from the pool.

Parameters:

path - the file that has been closed

Throws:

IOException

addedRow

void addedRow(int rows)
       throws IOException

Give the memory manager an opportunity for doing a memory check.

Parameters:

rows - number of rows added

Throws:

IOException