org.apache.drill.exec.work.foreman.rm

Class DistributedQueryQueue

java.lang.Object

org.apache.drill.exec.work.foreman.rm.DistributedQueryQueue

All Implemented Interfaces:

QueryQueue

public class DistributedQueryQueue
extends Object
implements QueryQueue

Distributed query queue which uses a Zookeeper distributed semaphore to control queuing across the cluster. The distributed queue is actually two queues: one for "small" queries, another for "large" queries. Query size is determined by the Planner's estimate of query cost.

This queue is configured using system options:

exec.queue.enable

Set to true to enable the distributed queue.

exec.queue.large

The maximum number of large queries to admit. Additional queries wait in the queue.

exec.queue.small

The maximum number of small queries to admit. Additional queries wait in the queue.

exec.queue.threshold

The cost threshold. Queries below this size are small, at or above this size are large..

exec.queue.timeout_millis

The maximum time (in milliseconds) a query will wait in the queue before failing.

The above values are refreshed every five seconds. This aids performance a bit in systems with very high query arrival rates.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	DistributedQueryQueue.StatusAdapter
static class	DistributedQueryQueue.ZKQueueInfo Exposes a snapshot of internal state information for use in status reporting, such as in the UI.

Nested classes/interfaces inherited from interface org.apache.drill.exec.work.foreman.rm.QueryQueue

QueryQueue.QueryQueueException, QueryQueue.QueueLease, QueryQueue.QueueTimeoutException

Constructor Summary

Constructors

Constructor and Description
DistributedQueryQueue(DrillbitContext context, DistributedQueryQueue.StatusAdapter adapter)

Method Summary

Modifier and Type	Method and Description
void	close()
long	defaultQueryMemoryPerNode(double cost) Return the amount of memory per node when creating a EXPLAIN query plan.
boolean	enabled() Determine if the queue is enabled.
QueryQueue.QueueLease	enqueue(UserBitShared.QueryId queryId, double cost) This limits the number of "small" and "large" queries that a Drill cluster will run simultaneously, if queuing is enabled.
DistributedQueryQueue.ZKQueueInfo	getInfo()
long	minimumOperatorMemory() Optional floor on the amount of memory assigned per operator.
void	setMemoryPerNode(long memoryPerNode)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DistributedQueryQueue

public DistributedQueryQueue(DrillbitContext context,
                             DistributedQueryQueue.StatusAdapter adapter)

Method Detail

setMemoryPerNode

public void setMemoryPerNode(long memoryPerNode)

Specified by:

setMemoryPerNode in interface QueryQueue

defaultQueryMemoryPerNode

public long defaultQueryMemoryPerNode(double cost)

Description copied from interface: QueryQueue

Return the amount of memory per node when creating a EXPLAIN query plan. Plans to be executed should get the query memory from the lease, as the lease may adjust the default amount on a per-query basis. This means that the memory used to execute the query may differ from the amount shown in an EXPLAIN plan.

Specified by:

defaultQueryMemoryPerNode in interface QueryQueue

Returns:

assumed memory per node, in bytes, to use when creating an EXPLAIN plan

minimumOperatorMemory

public long minimumOperatorMemory()

Description copied from interface: QueryQueue

Optional floor on the amount of memory assigned per operator. This ensures that operators receive a certain amount, separate from any memory slicing. This can oversubscribe node memory if used incorrectly.

Specified by:

minimumOperatorMemory in interface QueryQueue

Returns:

minimum per-operator memory, in bytes

enqueue

public QueryQueue.QueueLease enqueue(UserBitShared.QueryId queryId,
                                     double cost)
                              throws QueryQueue.QueryQueueException,
                                     QueryQueue.QueueTimeoutException

This limits the number of "small" and "large" queries that a Drill cluster will run simultaneously, if queuing is enabled. If the query is unable to run, this will block until it can. Beware that this is called under run(), and so will consume a thread while it waits for the required distributed semaphore.

Specified by:

enqueue in interface QueryQueue

Parameters:

queryId - query identifier

cost - the query plan

Returns:

the query lease which must be passed to #release(QueueLease) upon query completion

Throws:

QueryQueueException - if the underlying ZK queuing mechanism fails

QueueTimeoutException - if the query waits too long in the queue

QueryQueue.QueryQueueException

QueryQueue.QueueTimeoutException

enabled

public boolean enabled()

Description copied from interface: QueryQueue

Determine if the queue is enabled.

Specified by:

enabled in interface QueryQueue

Returns:

true if the query is enabled, false otherwise.

getInfo

public DistributedQueryQueue.ZKQueueInfo getInfo()

close

public void close()

Specified by:

close in interface QueryQueue