org.apache.drill.exec.physical.impl.join

Class HashJoinMemoryCalculatorImpl.BuildSidePartitioningImpl

java.lang.Object

org.apache.drill.exec.physical.impl.join.HashJoinMemoryCalculatorImpl.BuildSidePartitioningImpl

All Implemented Interfaces:

HashJoinMemoryCalculator.BuildSidePartitioning, HashJoinStateCalculator<HashJoinMemoryCalculator.PostBuildCalculations>

Enclosing class:

HashJoinMemoryCalculatorImpl

public static class HashJoinMemoryCalculatorImpl.BuildSidePartitioningImpl
extends Object
implements HashJoinMemoryCalculator.BuildSidePartitioning

At this point we need to reserve memory for the following:

1. An incoming batch
2. An incomplete batch for each partition

If there is available memory we keep the batches for each partition in memory. If we run out of room and need to start spilling, we need to specify which partitions need to be spilled.

Life Cycle

• Step 0: Call initialize(boolean, boolean, RecordBatch, RecordBatch, Set, boolean, long, int, int, int, int, int, int, double). This will initialize the StateCalculate with the additional information it needs.
• Step 1: Call getNumPartitions() to see the number of partitions that fit in memory.
• Step 2: Call shouldSpill() To determine if spilling needs to occurr.
• Step 3: Call next() and get the next memory calculator associated with your next state.

Constructor Summary

Constructors

Constructor and Description
BuildSidePartitioningImpl(BatchSizePredictor.Factory batchSizePredictorFactory, HashTableSizeCalculator hashTableSizeCalculator, HashJoinHelperSizeCalculator hashJoinHelperSizeCalculator, double fragmentationFactor, double safetyFactor, boolean semiJoin)

Method Summary

Modifier and Type	Method and Description
long	getBuildReservedMemory()
long	getMaxReservedMemory()
int	getNumPartitions()
HashJoinState	getState() The current HashJoinState corresponding to this calculator.
protected void	initialize(boolean firstCycle, boolean reserveHash, CaseInsensitiveMap<Long> keySizes, long memoryAvailable, int initialPartitions, boolean probeEmpty, BatchSizePredictor buildSizePredictor, BatchSizePredictor probeSizePredictor, int recordsPerPartitionBatchBuild, int recordsPerPartitionBatchProbe, int maxBatchNumRecordsBuild, int maxBatchNumRecordsProbe, int outputBatchSize, double loadFactor)
void	initialize(boolean firstCycle, boolean reserveHash, RecordBatch buildBatch, RecordBatch probeBatch, Set<String> joinColumns, boolean probeEmpty, long memoryAvailable, int initialPartitions, int recordsPerPartitionBatchBuild, int recordsPerPartitionBatchProbe, int maxBatchNumRecordsBuild, int maxBatchNumRecordsProbe, int outputBatchSize, double loadFactor)
String	makeDebugString()
HashJoinMemoryCalculator.PostBuildCalculations	next() Signifies that the current state is complete and returns the next HashJoinStateCalculator.
void	setPartitionStatSet(HashJoinMemoryCalculator.PartitionStatSet partitionStatSet)
boolean	shouldSpill()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BuildSidePartitioningImpl

public BuildSidePartitioningImpl(BatchSizePredictor.Factory batchSizePredictorFactory,
                                 HashTableSizeCalculator hashTableSizeCalculator,
                                 HashJoinHelperSizeCalculator hashJoinHelperSizeCalculator,
                                 double fragmentationFactor,
                                 double safetyFactor,
                                 boolean semiJoin)

Method Detail

initialize

public void initialize(boolean firstCycle,
                       boolean reserveHash,
                       RecordBatch buildBatch,
                       RecordBatch probeBatch,
                       Set<String> joinColumns,
                       boolean probeEmpty,
                       long memoryAvailable,
                       int initialPartitions,
                       int recordsPerPartitionBatchBuild,
                       int recordsPerPartitionBatchProbe,
                       int maxBatchNumRecordsBuild,
                       int maxBatchNumRecordsProbe,
                       int outputBatchSize,
                       double loadFactor)

Specified by:

initialize in interface HashJoinMemoryCalculator.BuildSidePartitioning

initialize

protected void initialize(boolean firstCycle,
                          boolean reserveHash,
                          CaseInsensitiveMap<Long> keySizes,
                          long memoryAvailable,
                          int initialPartitions,
                          boolean probeEmpty,
                          BatchSizePredictor buildSizePredictor,
                          BatchSizePredictor probeSizePredictor,
                          int recordsPerPartitionBatchBuild,
                          int recordsPerPartitionBatchProbe,
                          int maxBatchNumRecordsBuild,
                          int maxBatchNumRecordsProbe,
                          int outputBatchSize,
                          double loadFactor)

setPartitionStatSet

public void setPartitionStatSet(HashJoinMemoryCalculator.PartitionStatSet partitionStatSet)

Specified by:

setPartitionStatSet in interface HashJoinMemoryCalculator.BuildSidePartitioning

getNumPartitions

public int getNumPartitions()

Specified by:

getNumPartitions in interface HashJoinMemoryCalculator.BuildSidePartitioning

getBuildReservedMemory

public long getBuildReservedMemory()

Specified by:

getBuildReservedMemory in interface HashJoinMemoryCalculator.BuildSidePartitioning

getMaxReservedMemory

public long getMaxReservedMemory()

Specified by:

getMaxReservedMemory in interface HashJoinMemoryCalculator.BuildSidePartitioning

shouldSpill

public boolean shouldSpill()

Specified by:

shouldSpill in interface HashJoinMemoryCalculator.BuildSidePartitioning

next

public HashJoinMemoryCalculator.PostBuildCalculations next()

Description copied from interface: HashJoinStateCalculator

Signifies that the current state is complete and returns the next HashJoinStateCalculator. Returns null in the case where there is no next state.

Specified by:

next in interface HashJoinStateCalculator<HashJoinMemoryCalculator.PostBuildCalculations>

Returns:

The next HashJoinStateCalculator or null if this was the last state.

getState

public HashJoinState getState()

Description copied from interface: HashJoinStateCalculator

The current HashJoinState corresponding to this calculator.

Specified by:

getState in interface HashJoinStateCalculator<HashJoinMemoryCalculator.PostBuildCalculations>

Returns:

makeDebugString

public String makeDebugString()

Specified by:

makeDebugString in interface HashJoinMemoryCalculator.BuildSidePartitioning