org.apache.drill.exec.store.dfs.easy

Class EasyGroupScan

java.lang.Object

org.apache.drill.exec.physical.base.AbstractBase

org.apache.drill.exec.physical.base.AbstractGroupScan

org.apache.drill.exec.physical.base.AbstractFileGroupScan

org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata<TableMetadataProvider>

org.apache.drill.exec.store.dfs.easy.EasyGroupScan

All Implemented Interfaces:

Iterable<PhysicalOperator>, GraphValue<PhysicalOperator>, FileGroupScan, FragmentLeaf, GroupScan, HasAffinity, Leaf, PhysicalOperator, Scan

public class EasyGroupScan
extends AbstractGroupScanWithMetadata<TableMetadataProvider>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata

AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B extends AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B>>

Field Summary

Fields inherited from class org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata

columns, files, fileSet, filter, matchAllMetadata, maxRecords, metadataProvider, nonInterestingColumnsMetadata, partitionColumns, partitions, segments, tableMetadata, usedMetastore

Fields inherited from class org.apache.drill.exec.physical.base.AbstractBase

INIT_ALLOCATION, initialAllocation, MAX_ALLOCATION, maxAllocation

Fields inherited from interface org.apache.drill.exec.physical.base.GroupScan

ALL_COLUMNS

Constructor Summary

Constructors

Constructor and Description
EasyGroupScan(String userName, FileSelection selection, EasyFormatPlugin<?> formatPlugin, List<SchemaPath> columns, org.apache.hadoop.fs.Path selectionRoot, int minWidth, MetadataProviderManager metadataProvider)
EasyGroupScan(String userName, FileSelection selection, EasyFormatPlugin<?> formatPlugin, List<SchemaPath> columns, org.apache.hadoop.fs.Path selectionRoot, MetadataProviderManager metadataProviderManager)
EasyGroupScan(String userName, List<org.apache.hadoop.fs.Path> files, StoragePluginConfig storageConfig, FormatPluginConfig formatConfig, StoragePluginRegistry engineRegistry, List<SchemaPath> columns, org.apache.hadoop.fs.Path selectionRoot, TupleMetadata schema)

Method Summary

Modifier and Type	Method and Description
void	applyAssignments(List<CoordinationProtos.DrillbitEndpoint> incomingEndpoints)
boolean	canPushdownProjects(List<SchemaPath> columns) GroupScan should check the list of columns, and see if it could support all the columns in the list.
FileGroupScan	clone(FileSelection selection)
GroupScan	clone(List<SchemaPath> columns) Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .
protected FileTableMetadataProviderBuilder<?>	defaultTableMetadataProviderBuilder(MetadataProviderManager source) Returns TableMetadataProviderBuilder instance which may provide metadata without using Drill Metastore.
AnalyzeInfoProvider	getAnalyzeInfoProvider() Returns AnalyzeInfoProvider instance which will be used when running ANALYZE statement.
List<org.apache.hadoop.fs.Path>	getFiles() Returns a collection of file names associated with this GroupScan.
FileSelection	getFileSelection()
protected AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<?>	getFilterer() Returns holder for metadata values which provides API to filter metadata and build new group scan instance using filtered metadata.
FormatPluginConfig	getFormatConfig()
int	getMaxParallelizationWidth()
int	getMinParallelizationWidth() At minimum, the GroupScan requires these many fragments to run.
PhysicalOperator	getNewWithChildren(List<PhysicalOperator> children) Regenerate with this node with a new set of children.
List<EndpointAffinity>	getOperatorAffinity() Get the list of Endpoints with associated affinities that this operator has preference for.
protected List<String>	getPartitionValues(LocationProvider locationProvider)
ScanStats	getScanStats(PlannerSettings settings)
TupleMetadata	getSchema()
org.apache.hadoop.fs.Path	getSelectionRoot() Returns path to the selection root.
EasySubScan	getSpecificScan(int minorFragmentId)
StoragePluginConfig	getStorageConfig()
Iterable<CompleteFileWork>	getWorkIterable()
boolean	hasFiles() Return true if this GroupScan can return its selection as a list of file names (retrieved by getFiles()).
void	modifyFileSelection(FileSelection selection)
protected boolean	supportsFileImplicitColumns()
boolean	supportsFilterPushDown() Checks whether this group scan supports filter push down.
protected FileTableMetadataProviderBuilder<?>	tableMetadataProviderBuilder(MetadataProviderManager source) Returns TableMetadataProviderBuilder instance based on specified MetadataProviderManager source.
String	toString()

Methods inherited from class org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata

applyFilter, applyLimit, checkMetadataConsistency, getColumns, getColumnValueCount, getDigest, getFileSet, getFilesMetadata, getFilter, getFilterPredicate, getFilterPredicate, getFilterString, getMaxRecords, getMetadataProvider, getNextOrEmpty, getNonInterestingColumnsMetadata, getPartitionColumns, getPartitionsMetadata, getPartitionValue, getScanStats, getSegmentsMetadata, getTableMetadata, getTypeForColumn, init, isAllDataPruned, isGroupScanFullyMatchesFilter, isImplicitOrPartCol, isMatchAllMetadata, limitMetadata, pruneForPartitions, setFilter, setFilterForRuntime, supportsLimitPushdown, usedMetastore

Methods inherited from class org.apache.drill.exec.physical.base.AbstractFileGroupScan

supportsPartitionFilterPushdown

Methods inherited from class org.apache.drill.exec.physical.base.AbstractGroupScan

accept, enforceWidth, getDistributionAffinity, getInitialAllocation, getMaxAllocation, getOperatorType, isDistributed, isExecutable, iterator

Methods inherited from class org.apache.drill.exec.physical.base.AbstractBase

accept, getCost, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.physical.base.GroupScan

enforceWidth, isDistributed

Methods inherited from interface org.apache.drill.exec.physical.base.PhysicalOperator

accept, getCost, getInitialAllocation, getMaxAllocation, getOperatorId, getOperatorType, getSVMode, getUserName, isBufferedOperator, isExecutable, setCost, setMaxAllocation, setOperatorId

Methods inherited from interface org.apache.drill.common.graph.GraphValue

accept

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Methods inherited from interface org.apache.drill.exec.physical.base.HasAffinity

getDistributionAffinity

Constructor Detail

EasyGroupScan

public EasyGroupScan(String userName,
                     List<org.apache.hadoop.fs.Path> files,
                     StoragePluginConfig storageConfig,
                     FormatPluginConfig formatConfig,
                     StoragePluginRegistry engineRegistry,
                     List<SchemaPath> columns,
                     org.apache.hadoop.fs.Path selectionRoot,
                     TupleMetadata schema)
              throws IOException

Throws:

IOException

EasyGroupScan

public EasyGroupScan(String userName,
                     FileSelection selection,
                     EasyFormatPlugin<?> formatPlugin,
                     List<SchemaPath> columns,
                     org.apache.hadoop.fs.Path selectionRoot,
                     MetadataProviderManager metadataProviderManager)
              throws IOException

Throws:

IOException

EasyGroupScan

public EasyGroupScan(String userName,
                     FileSelection selection,
                     EasyFormatPlugin<?> formatPlugin,
                     List<SchemaPath> columns,
                     org.apache.hadoop.fs.Path selectionRoot,
                     int minWidth,
                     MetadataProviderManager metadataProvider)
              throws IOException

Throws:

IOException

Method Detail

getWorkIterable

public Iterable<CompleteFileWork> getWorkIterable()

getSelectionRoot

public org.apache.hadoop.fs.Path getSelectionRoot()

Description copied from interface: GroupScan

Returns path to the selection root. If this GroupScan cannot provide selection root, it returns null.

Specified by:

getSelectionRoot in interface GroupScan

Overrides:

getSelectionRoot in class AbstractGroupScan

Returns:

path to the selection root

getMinParallelizationWidth

public int getMinParallelizationWidth()

Description copied from interface: GroupScan

At minimum, the GroupScan requires these many fragments to run. Currently, this is used in SimpleParallelizer

Specified by:

getMinParallelizationWidth in interface GroupScan

Overrides:

getMinParallelizationWidth in class AbstractGroupScan

Returns:

the minimum number of fragments that should run

getMaxParallelizationWidth

public int getMaxParallelizationWidth()

getScanStats

public ScanStats getScanStats(PlannerSettings settings)

Specified by:

getScanStats in interface GroupScan

Overrides:

getScanStats in class AbstractGroupScan

hasFiles

public boolean hasFiles()

Description copied from interface: GroupScan

Return true if this GroupScan can return its selection as a list of file names (retrieved by getFiles()).

Specified by:

hasFiles in interface GroupScan

Overrides:

hasFiles in class AbstractGroupScanWithMetadata<TableMetadataProvider>

getFiles

public List<org.apache.hadoop.fs.Path> getFiles()

Description copied from interface: GroupScan

Returns a collection of file names associated with this GroupScan. This should be called after checking hasFiles(). If this GroupScan cannot provide file names, it returns null.

Specified by:

getFiles in interface GroupScan

Overrides:

getFiles in class AbstractGroupScanWithMetadata<TableMetadataProvider>

Returns:

collection of files paths

getFileSelection

public FileSelection getFileSelection()

modifyFileSelection

public void modifyFileSelection(FileSelection selection)

Specified by:

modifyFileSelection in interface FileGroupScan

Overrides:

modifyFileSelection in class AbstractGroupScanWithMetadata<TableMetadataProvider>

supportsFileImplicitColumns

protected boolean supportsFileImplicitColumns()

Specified by:

supportsFileImplicitColumns in class AbstractGroupScanWithMetadata<TableMetadataProvider>

supportsFilterPushDown

public boolean supportsFilterPushDown()

Description copied from interface: GroupScan

Checks whether this group scan supports filter push down.

Specified by:

supportsFilterPushDown in interface GroupScan

Overrides:

supportsFilterPushDown in class AbstractGroupScan

Returns:

true if this group scan supports filter push down, false otherwise

getPartitionValues

protected List<String> getPartitionValues(LocationProvider locationProvider)

Specified by:

getPartitionValues in class AbstractGroupScanWithMetadata<TableMetadataProvider>

getNewWithChildren

public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children)

Description copied from interface: PhysicalOperator

Regenerate with this node with a new set of children. This is used in the case of materialization or optimization.

getOperatorAffinity

public List<EndpointAffinity> getOperatorAffinity()

Description copied from interface: HasAffinity

Get the list of Endpoints with associated affinities that this operator has preference for.

Specified by:

getOperatorAffinity in interface HasAffinity

Overrides:

getOperatorAffinity in class AbstractGroupScan

Returns:

List of EndpointAffinity objects.

applyAssignments

public void applyAssignments(List<CoordinationProtos.DrillbitEndpoint> incomingEndpoints)

getSpecificScan

public EasySubScan getSpecificScan(int minorFragmentId)

getStorageConfig

public StoragePluginConfig getStorageConfig()

getFormatConfig

public FormatPluginConfig getFormatConfig()

toString

public String toString()

Overrides:

toString in class Object

clone

public GroupScan clone(List<SchemaPath> columns)

Description copied from interface: GroupScan

Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .

Specified by:

clone in interface GroupScan

Overrides:

clone in class AbstractGroupScan

clone

public FileGroupScan clone(FileSelection selection)
                    throws IOException

Specified by:

clone in interface FileGroupScan

Overrides:

clone in class AbstractFileGroupScan

Throws:

IOException

canPushdownProjects

public boolean canPushdownProjects(List<SchemaPath> columns)

Description copied from interface: GroupScan

GroupScan should check the list of columns, and see if it could support all the columns in the list.

Specified by:

canPushdownProjects in interface GroupScan

Overrides:

canPushdownProjects in class AbstractGroupScan

getSchema

public TupleMetadata getSchema()

Overrides:

getSchema in class AbstractGroupScanWithMetadata<TableMetadataProvider>

getAnalyzeInfoProvider

public AnalyzeInfoProvider getAnalyzeInfoProvider()

Description copied from interface: GroupScan

Returns AnalyzeInfoProvider instance which will be used when running ANALYZE statement.

Specified by:

getAnalyzeInfoProvider in interface GroupScan

Overrides:

getAnalyzeInfoProvider in class AbstractGroupScan

Returns:

AnalyzeInfoProvider instance

getFilterer

protected AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<?> getFilterer()

Description copied from class: AbstractGroupScanWithMetadata

Returns holder for metadata values which provides API to filter metadata and build new group scan instance using filtered metadata.

Specified by:

getFilterer in class AbstractGroupScanWithMetadata<TableMetadataProvider>

tableMetadataProviderBuilder

protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source)

Description copied from class: AbstractGroupScanWithMetadata

Returns TableMetadataProviderBuilder instance based on specified MetadataProviderManager source.

Specified by:

tableMetadataProviderBuilder in class AbstractGroupScanWithMetadata<TableMetadataProvider>

Parameters:

source - metadata provider manager

Returns:

TableMetadataProviderBuilder instance

defaultTableMetadataProviderBuilder

protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source)

Description copied from class: AbstractGroupScanWithMetadata

Returns TableMetadataProviderBuilder instance which may provide metadata without using Drill Metastore.

Specified by:

defaultTableMetadataProviderBuilder in class AbstractGroupScanWithMetadata<TableMetadataProvider>

Parameters:

source - metadata provider manager

Returns:

TableMetadataProviderBuilder instance