org.apache.drill.exec.store.hive

Class HiveDrillNativeParquetScan

java.lang.Object

org.apache.drill.exec.physical.base.AbstractBase

org.apache.drill.exec.physical.base.AbstractGroupScan

org.apache.drill.exec.physical.base.AbstractFileGroupScan

org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata<ParquetMetadataProvider>

org.apache.drill.exec.store.parquet.AbstractParquetGroupScan

org.apache.drill.exec.store.hive.HiveDrillNativeParquetScan

All Implemented Interfaces:

Iterable<PhysicalOperator>, GraphValue<PhysicalOperator>, FileGroupScan, FragmentLeaf, GroupScan, HasAffinity, Leaf, PhysicalOperator, Scan

public class HiveDrillNativeParquetScan
extends AbstractParquetGroupScan

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.exec.store.parquet.AbstractParquetGroupScan

AbstractParquetGroupScan.RowGroupScanFilterer<B extends AbstractParquetGroupScan.RowGroupScanFilterer<B>>

Nested classes/interfaces inherited from class org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata

AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B extends AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer<B>>

Field Summary

Fields inherited from class org.apache.drill.exec.store.parquet.AbstractParquetGroupScan

entries, mappings, readerConfig, rowGroups

Fields inherited from class org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata

columns, files, fileSet, filter, matchAllMetadata, maxRecords, metadataProvider, nonInterestingColumnsMetadata, partitionColumns, partitions, segments, tableMetadata, usedMetastore

Fields inherited from class org.apache.drill.exec.physical.base.AbstractBase

INIT_ALLOCATION, initialAllocation, MAX_ALLOCATION, maxAllocation

Fields inherited from interface org.apache.drill.exec.physical.base.GroupScan

ALL_COLUMNS

Constructor Summary

Constructors

Constructor and Description
HiveDrillNativeParquetScan(StoragePluginRegistry engineRegistry, String userName, HiveStoragePluginConfig hiveStoragePluginConfig, List<SchemaPath> columns, List<ReadEntryWithPath> entries, HivePartitionHolder hivePartitionHolder, Map<String,String> confProperties, ParquetReaderConfig readerConfig, LogicalExpression filter, TupleMetadata schema)
HiveDrillNativeParquetScan(String userName, List<SchemaPath> columns, HiveStoragePlugin hiveStoragePlugin, List<HiveMetadataProvider.LogicalInputSplit> logicalInputSplits, Map<String,String> confProperties, ParquetReaderConfig readerConfig)
HiveDrillNativeParquetScan(String userName, List<SchemaPath> columns, HiveStoragePlugin hiveStoragePlugin, List<HiveMetadataProvider.LogicalInputSplit> logicalInputSplits, Map<String,String> confProperties, ParquetReaderConfig readerConfig, LogicalExpression filter)

Method Summary

Modifier and Type	Method and Description
HiveDrillNativeParquetScan	clone(FileSelection selection)
GroupScan	clone(List<SchemaPath> columns) Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .
protected AbstractParquetGroupScan	cloneWithFileSelection(Collection<org.apache.hadoop.fs.Path> filePaths)
protected HiveParquetTableMetadataProvider.Builder	defaultTableMetadataProviderBuilder(MetadataProviderManager source) Returns TableMetadataProviderBuilder instance which may provide metadata without using Drill Metastore.
Map<String,String>	getConfProperties()
protected Collection<CoordinationProtos.DrillbitEndpoint>	getDrillbits()
protected AbstractParquetGroupScan.RowGroupScanFilterer<?>	getFilterer() Returns holder for metadata values which provides API to filter metadata and build new group scan instance using filtered metadata.
HivePartitionHolder	getHivePartitionHolder()
HiveStoragePluginConfig	getHiveStoragePluginConfig()
PhysicalOperator	getNewWithChildren(List<PhysicalOperator> children) Regenerate with this node with a new set of children.
protected List<String>	getPartitionValues(LocationProvider locationProvider)
SubScan	getSpecificScan(int minorFragmentId)
protected boolean	supportsFileImplicitColumns()
protected HiveParquetTableMetadataProvider.Builder	tableMetadataProviderBuilder(MetadataProviderManager source) Returns TableMetadataProviderBuilder instance based on specified MetadataProviderManager source.
String	toString()

Methods inherited from class org.apache.drill.exec.store.parquet.AbstractParquetGroupScan

applyAssignments, applyFilter, applyLimit, canPushdownProjects, getEntries, getFiles, getMaxParallelizationWidth, getOperatorAffinity, getReadEntries, getReaderConfig, getReaderConfigForSerialization, getRowGroupsMetadata, modifyFileSelection, pruneForPartitions, pruneRowGroupsForFiles, supportsFilterPushDown

Methods inherited from class org.apache.drill.exec.physical.base.AbstractGroupScanWithMetadata

checkMetadataConsistency, getColumns, getColumnValueCount, getDigest, getFileSet, getFilesMetadata, getFilter, getFilterPredicate, getFilterPredicate, getFilterString, getMaxRecords, getMetadataProvider, getNextOrEmpty, getNonInterestingColumnsMetadata, getPartitionColumns, getPartitionsMetadata, getPartitionValue, getScanStats, getSchema, getSegmentsMetadata, getTableMetadata, getTypeForColumn, hasFiles, init, isAllDataPruned, isGroupScanFullyMatchesFilter, isImplicitOrPartCol, isMatchAllMetadata, limitMetadata, pruneForPartitions, setFilter, setFilterForRuntime, supportsLimitPushdown, usedMetastore

Methods inherited from class org.apache.drill.exec.physical.base.AbstractFileGroupScan

supportsPartitionFilterPushdown

Methods inherited from class org.apache.drill.exec.physical.base.AbstractGroupScan

accept, enforceWidth, getAnalyzeInfoProvider, getDistributionAffinity, getInitialAllocation, getMaxAllocation, getMinParallelizationWidth, getOperatorType, getScanStats, getSelectionRoot, isDistributed, isExecutable, iterator

Methods inherited from class org.apache.drill.exec.physical.base.AbstractBase

accept, getCost, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.physical.base.GroupScan

enforceWidth, getAnalyzeInfoProvider, getMinParallelizationWidth, getScanStats, getSelectionRoot, isDistributed

Methods inherited from interface org.apache.drill.exec.physical.base.PhysicalOperator

accept, getCost, getInitialAllocation, getMaxAllocation, getOperatorId, getOperatorType, getSVMode, getUserName, isBufferedOperator, isExecutable, setCost, setMaxAllocation, setOperatorId

Methods inherited from interface org.apache.drill.common.graph.GraphValue

accept

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Methods inherited from interface org.apache.drill.exec.physical.base.HasAffinity

getDistributionAffinity

Constructor Detail

HiveDrillNativeParquetScan

public HiveDrillNativeParquetScan(StoragePluginRegistry engineRegistry,
                                  String userName,
                                  HiveStoragePluginConfig hiveStoragePluginConfig,
                                  List<SchemaPath> columns,
                                  List<ReadEntryWithPath> entries,
                                  HivePartitionHolder hivePartitionHolder,
                                  Map<String,String> confProperties,
                                  ParquetReaderConfig readerConfig,
                                  LogicalExpression filter,
                                  TupleMetadata schema)
                           throws IOException

Throws:

IOException

HiveDrillNativeParquetScan

public HiveDrillNativeParquetScan(String userName,
                                  List<SchemaPath> columns,
                                  HiveStoragePlugin hiveStoragePlugin,
                                  List<HiveMetadataProvider.LogicalInputSplit> logicalInputSplits,
                                  Map<String,String> confProperties,
                                  ParquetReaderConfig readerConfig)
                           throws IOException

Throws:

IOException

HiveDrillNativeParquetScan

public HiveDrillNativeParquetScan(String userName,
                                  List<SchemaPath> columns,
                                  HiveStoragePlugin hiveStoragePlugin,
                                  List<HiveMetadataProvider.LogicalInputSplit> logicalInputSplits,
                                  Map<String,String> confProperties,
                                  ParquetReaderConfig readerConfig,
                                  LogicalExpression filter)
                           throws IOException

Throws:

IOException

Method Detail

getHiveStoragePluginConfig

public HiveStoragePluginConfig getHiveStoragePluginConfig()

getHivePartitionHolder

public HivePartitionHolder getHivePartitionHolder()

getConfProperties

public Map<String,String> getConfProperties()

getSpecificScan

public SubScan getSpecificScan(int minorFragmentId)

getNewWithChildren

public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children)

Description copied from interface: PhysicalOperator

Regenerate with this node with a new set of children. This is used in the case of materialization or optimization.

clone

public HiveDrillNativeParquetScan clone(FileSelection selection)
                                 throws IOException

Specified by:

clone in interface FileGroupScan

Overrides:

clone in class AbstractFileGroupScan

Throws:

IOException

clone

public GroupScan clone(List<SchemaPath> columns)

Description copied from interface: GroupScan

Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .

Specified by:

clone in interface GroupScan

Overrides:

clone in class AbstractGroupScan

toString

public String toString()

Overrides:

toString in class Object

getFilterer

protected AbstractParquetGroupScan.RowGroupScanFilterer<?> getFilterer()

Description copied from class: AbstractGroupScanWithMetadata

Returns holder for metadata values which provides API to filter metadata and build new group scan instance using filtered metadata.

Specified by:

getFilterer in class AbstractParquetGroupScan

getDrillbits

protected Collection<CoordinationProtos.DrillbitEndpoint> getDrillbits()

Specified by:

getDrillbits in class AbstractParquetGroupScan

cloneWithFileSelection

protected AbstractParquetGroupScan cloneWithFileSelection(Collection<org.apache.hadoop.fs.Path> filePaths)
                                                   throws IOException

Specified by:

cloneWithFileSelection in class AbstractParquetGroupScan

Throws:

IOException

supportsFileImplicitColumns

protected boolean supportsFileImplicitColumns()

Specified by:

supportsFileImplicitColumns in class AbstractGroupScanWithMetadata<ParquetMetadataProvider>

getPartitionValues

protected List<String> getPartitionValues(LocationProvider locationProvider)

Specified by:

getPartitionValues in class AbstractGroupScanWithMetadata<ParquetMetadataProvider>

tableMetadataProviderBuilder

protected HiveParquetTableMetadataProvider.Builder tableMetadataProviderBuilder(MetadataProviderManager source)

Description copied from class: AbstractGroupScanWithMetadata

Returns TableMetadataProviderBuilder instance based on specified MetadataProviderManager source.

Specified by:

tableMetadataProviderBuilder in class AbstractGroupScanWithMetadata<ParquetMetadataProvider>

Parameters:

source - metadata provider manager

Returns:

TableMetadataProviderBuilder instance

defaultTableMetadataProviderBuilder

protected HiveParquetTableMetadataProvider.Builder defaultTableMetadataProviderBuilder(MetadataProviderManager source)

Description copied from class: AbstractGroupScanWithMetadata

Returns TableMetadataProviderBuilder instance which may provide metadata without using Drill Metastore.

Specified by:

defaultTableMetadataProviderBuilder in class AbstractParquetGroupScan

Parameters:

source - metadata provider manager

Returns:

TableMetadataProviderBuilder instance