org.apache.drill.exec.physical.config

Class AbstractDeMuxExchange

java.lang.Object

org.apache.drill.exec.physical.base.AbstractBase

org.apache.drill.exec.physical.base.AbstractSingle

org.apache.drill.exec.physical.base.AbstractExchange

org.apache.drill.exec.physical.config.AbstractDeMuxExchange

All Implemented Interfaces:

Iterable<PhysicalOperator>, GraphValue<PhysicalOperator>, Exchange, PhysicalOperator

Direct Known Subclasses:

UnorderedDeMuxExchange

public abstract class AbstractDeMuxExchange
extends AbstractExchange

DeMuxExchange is opposite of MuxExchange. It is used when the sender has overhead that is proportional to the number of receivers. DeMuxExchange is run one instance per Drillbit endpoint which collects and distributes data belonging to local receiving fragments running on the same Drillbit. Example: On a 3 node cluster, if the sender has 10 receivers on each node each sender requires 30 buffers. By inserting DeMuxExchange, we create one receiver per node which means total of 3 receivers for each sender. If the number of senders is 10, we use 10*3 buffers instead of 10*30. DeMuxExchange has a overhead of buffer space that is equal to number of local receivers. In this case each DeMuxExchange needs 10 buffers, so total of 3*10 buffers.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.drill.exec.physical.base.Exchange

Exchange.ParallelizationDependency

Field Summary

Fields

Modifier and Type	Field and Description
protected LogicalExpression	expr
protected Map<Integer,MinorFragmentEndpoint>	receiverToSenderMapping
protected org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap<Integer,MinorFragmentEndpoint>	senderToReceiversMapping

Fields inherited from class org.apache.drill.exec.physical.base.AbstractExchange

receiverLocations, receiverMajorFragmentId, senderLocations, senderMajorFragmentId

Fields inherited from class org.apache.drill.exec.physical.base.AbstractSingle

child

Fields inherited from class org.apache.drill.exec.physical.base.AbstractBase

INIT_ALLOCATION, initialAllocation, MAX_ALLOCATION, maxAllocation

Constructor Summary

Constructors

Constructor and Description
AbstractDeMuxExchange(PhysicalOperator child, LogicalExpression expr)

Method Summary

Modifier and Type	Method and Description
protected void	createSenderReceiverMapping()
LogicalExpression	getExpression()
Exchange.ParallelizationDependency	getParallelizationDependency() In DeMuxExchange, sender fragment parallelization and endpoint assignment depends on receiver fragment endpoint assignments.
ParallelizationInfo	getReceiverParallelizationInfo(List<CoordinationProtos.DrillbitEndpoint> senderFragmentEndpoints) Default receiver parallelization width range is [1, Integer.MAX_VALUE] and affinity to nodes where sender fragments are running.
Sender	getSender(int minorFragmentId, PhysicalOperator child) Get the Sender associated with the given minorFragmentId.
ParallelizationInfo	getSenderParallelizationInfo(List<CoordinationProtos.DrillbitEndpoint> receiverFragmentEndpoints) Default sender parallelization width range is [1, Integer.MAX_VALUE] and no endpoint affinity

Methods inherited from class org.apache.drill.exec.physical.base.AbstractExchange

accept, getDefaultAffinityMap, getOperatorType, getReceiverMemory, getSenderMemory, isExecutable, setupReceivers, setupReceivers, setupSenders, setupSenders

Methods inherited from class org.apache.drill.exec.physical.base.AbstractSingle

getChild, getNewWithChild, getNewWithChildren, iterator

Methods inherited from class org.apache.drill.exec.physical.base.AbstractBase

accept, getCost, getInitialAllocation, getMaxAllocation, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.physical.base.Exchange

getChild, getReceiver

Methods inherited from interface org.apache.drill.exec.physical.base.PhysicalOperator

getCost, getInitialAllocation, getMaxAllocation, getNewWithChildren, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from interface org.apache.drill.common.graph.GraphValue

accept

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Field Detail

expr

protected final LogicalExpression expr

receiverToSenderMapping

protected Map<Integer,MinorFragmentEndpoint> receiverToSenderMapping

senderToReceiversMapping

protected org.apache.drill.shaded.guava.com.google.common.collect.ArrayListMultimap<Integer,MinorFragmentEndpoint> senderToReceiversMapping

Constructor Detail

AbstractDeMuxExchange

public AbstractDeMuxExchange(PhysicalOperator child,
                             LogicalExpression expr)

Method Detail

getExpression

public LogicalExpression getExpression()

getSenderParallelizationInfo

public ParallelizationInfo getSenderParallelizationInfo(List<CoordinationProtos.DrillbitEndpoint> receiverFragmentEndpoints)

Description copied from class: AbstractExchange

Default sender parallelization width range is [1, Integer.MAX_VALUE] and no endpoint affinity

Specified by:

getSenderParallelizationInfo in interface Exchange

Overrides:

getSenderParallelizationInfo in class AbstractExchange

Parameters:

receiverFragmentEndpoints - Endpoints assigned to receiver fragment if available, otherwise an empty list.

Returns:

Sender ParallelizationInfo.

getReceiverParallelizationInfo

public ParallelizationInfo getReceiverParallelizationInfo(List<CoordinationProtos.DrillbitEndpoint> senderFragmentEndpoints)

Description copied from class: AbstractExchange

Default receiver parallelization width range is [1, Integer.MAX_VALUE] and affinity to nodes where sender fragments are running.

Specified by:

getReceiverParallelizationInfo in interface Exchange

Overrides:

getReceiverParallelizationInfo in class AbstractExchange

Parameters:

senderFragmentEndpoints - Endpoints assigned to receiver fragment if available, otherwise an empty list.

Returns:

Receiver ParallelizationInfo.

getSender

public Sender getSender(int minorFragmentId,
                        PhysicalOperator child)

Description copied from interface: Exchange

Get the Sender associated with the given minorFragmentId. Cannot be called until after setupSenders() and setupReceivers() have been called.

Parameters:

minorFragmentId - The minor fragment id, must be in the range [0, fragment.width).

child - The feeding node for the requested sender.

Returns:

The materialized sender for the given arguments.

getParallelizationDependency

public Exchange.ParallelizationDependency getParallelizationDependency()

In DeMuxExchange, sender fragment parallelization and endpoint assignment depends on receiver fragment endpoint assignments.

Specified by:

getParallelizationDependency in interface Exchange

Overrides:

getParallelizationDependency in class AbstractExchange

createSenderReceiverMapping

protected void createSenderReceiverMapping()