Package org.apache.drill.exec.util

Class MemoryAllocationUtilities

java.lang.Object
org.apache.drill.exec.util.MemoryAllocationUtilities
public class MemoryAllocationUtilities extends Object

  • Constructor Details

    • MemoryAllocationUtilities

      public MemoryAllocationUtilities()

  • Method Details

    • setupBufferedMemoryAllocations

      public static void setupBufferedMemoryAllocations(PhysicalPlan plan, QueryContext queryContext)

    • getBufferedOperators

      public static List<PhysicalOperator> getBufferedOperators(List<PhysicalOperator> operators, QueryContext queryContext)

    • setupBufferedOpsMemoryAllocations

      public static void setupBufferedOpsMemoryAllocations(boolean planHasMemory, List<PhysicalOperator> bufferedOperators, QueryContext queryContext)
      Helper method to setup Memory Allocations

      Plan the memory for buffered operators (the only ones that can spill in this release) based on assumptions. These assumptions are the amount of memory per node to give to each query and the number of sort operators per node.

      The reason the total memory is an assumption is that we have know knowledge of the number of queries that can run, so we need the user to tell use that information by configuring the amount of memory to be assumed available to each query.

      The number of sorts per node could be calculated, but we instead simply take the worst case: the maximum per-query, per-node parallization and assume that all sorts appear in all fragments — a gross oversimplification, but one that Drill has long made.

      since this method can be used in multiple places adding it in this class rather than keeping it in Foreman

      Parameters:
      planHasMemory - defines the memory planning needs to be done or not. generally skipped when the plan contains memory allocation.
      bufferedOperators - list of buffered operators in the plan.
      queryContext - context of the query.

    • computeOperatorMemory

      public static long computeOperatorMemory(OptionSet optionManager, long maxAllocPerNode, int opCount)
      Compute per-operator memory based on the computed per-node memory, the number of operators, and the computed number of fragments (which house the operators.) Enforces a floor on the amount of memory per operator.
      Parameters:
      optionManager - system option manager
      maxAllocPerNode - computed query memory per node
      opCount - number of buffering operators in this query
      Returns:
      the per-operator memory

    • computeQueryMemory

      public static long computeQueryMemory(DrillConfig config, OptionSet optionManager, long directMemory)
      Per-node memory calculations based on a number of constraints.

      Factored out into a separate method to allow unit testing.

      Parameters:
      config - Drill config
      optionManager - system options
      directMemory - amount of direct memory
      Returns:
      memory per query per node