org.apache.drill.exec.store.mapr.db.json

Class JsonTableGroupScan

java.lang.Object

org.apache.drill.exec.physical.base.AbstractBase

org.apache.drill.exec.physical.base.AbstractGroupScan

org.apache.drill.exec.physical.base.AbstractDbGroupScan

org.apache.drill.exec.store.mapr.db.MapRDBGroupScan

org.apache.drill.exec.store.mapr.db.json.JsonTableGroupScan

All Implemented Interfaces:

Iterable<PhysicalOperator>, GraphValue<PhysicalOperator>, DbGroupScan, FragmentLeaf, GroupScan, HasAffinity, IndexGroupScan, Leaf, PhysicalOperator, Scan

Direct Known Subclasses:

RestrictedJsonTableGroupScan

public class JsonTableGroupScan
extends MapRDBGroupScan
implements IndexGroupScan

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<org.apache.calcite.rex.RexNode,Double>	forcedRowCountMap
protected double	fullTableEstimatedSize
protected double	fullTableRowCount
protected int	maxRecordsToRead need only read maxRecordsToRead records.
protected int	parallelizationWidth Forced parallelization width
protected JsonScanSpec	scanSpec
static int	STAR_COLS
protected MapRDBStatistics	stats
static String	TABLE_JSON

Fields inherited from class org.apache.drill.exec.store.mapr.db.MapRDBGroupScan

columns, costFactor, doNotAccessRegionsToScan, endpointFragmentMapping, formatPlugin, formatPluginConfig, storagePlugin

Fields inherited from class org.apache.drill.exec.physical.base.AbstractBase

INIT_ALLOCATION, initialAllocation, MAX_ALLOCATION, maxAllocation

Fields inherited from interface org.apache.drill.exec.physical.base.GroupScan

ALL_COLUMNS

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	JsonTableGroupScan(JsonTableGroupScan that) Private constructor, used for cloning.
	JsonTableGroupScan(String userName, AbstractStoragePlugin storagePlugin, MapRDBFormatPlugin formatPlugin, JsonScanSpec scanSpec, List<SchemaPath> columns, MapRDBStatistics stats, TableMetadataProvider metadataProvider)
	JsonTableGroupScan(String userName, AbstractStoragePlugin storagePlugin, MapRDBFormatPlugin formatPlugin, JsonScanSpec scanSpec, List<SchemaPath> columns, MetadataProviderManager metadataProviderManager)
	JsonTableGroupScan(String userName, JsonScanSpec scanSpec, FileSystemConfig storagePluginConfig, MapRDBFormatPluginConfig formatPluginConfig, List<SchemaPath> columns, TupleMetadata schema, StoragePluginRegistry pluginRegistry)

Method Summary

Modifier and Type	Method and Description
GroupScan	applyLimit(int maxRecords) By default, return null to indicate row count based prune is not supported.
boolean	canPushdownProjects(List<SchemaPath> columns) GroupScan should check the list of columns, and see if it could support all the columns in the list.
GroupScan	clone(JsonScanSpec scanSpec)
GroupScan	clone(List<SchemaPath> columns) Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .
org.ojai.store.QueryCondition	convertToQueryCondition(LogicalExpression condition) Convert a given LogicalExpression condition into a QueryCondition condition
MapRDBStatisticsPayload	getAverageRowSizeStats(IndexDescriptor index) Get the estimated average rowsize.
List<SchemaPath>	getColumns() Returns a list of columns scanned by this group scan
MapRDBStatisticsPayload	getFirstKeyEstimatedStats(org.ojai.store.QueryCondition condition, IndexDescriptor index, org.apache.calcite.rel.RelNode scanRel) Get the estimated statistics after applying the RexNode condition.
com.mapr.db.index.IndexDesc	getIndexDesc()
int	getMaxParallelizationWidth()
PhysicalOperator	getNewWithChildren(List<PhysicalOperator> children) Regenerate with this node with a new set of children.
PartitionFunction	getRangePartitionFunction(List<FieldReference> refList) Get a partition function instance for range based partitioning
protected NavigableMap<TabletFragmentInfo,String>	getRegionsToScan()
protected NavigableMap<TabletFragmentInfo,String>	getRegionsToScan(int scanRangeSizeMB)
RestrictedJsonTableGroupScan	getRestrictedScan(List<SchemaPath> columns) If this DbGroupScan supports restricted scan, create a restricted scan from this DbGroupScan.
double	getRowCount(org.apache.calcite.rex.RexNode condition, org.apache.calcite.rel.RelNode scanRel) Get the row count after applying the RexNode condition
JsonScanSpec	getScanSpec()
ScanStats	getScanStats()
MapRDBSubScan	getSpecificScan(int minorFragmentId)
MapRDBStatistics	getStatistics() Get the statistics for this DbGroupScan
protected MapRDBSubScanSpec	getSubScanSpec(TabletFragmentInfo tfi)
String	getTableName()
boolean	isDisablePushdown()
boolean	isDistributed()
boolean	isIndexScan()
void	setColumns(List<SchemaPath> columns)
void	setParallelizationWidth(int width)
void	setRowCount(org.apache.calcite.rex.RexNode condition, double count, double capRowCount) Set the row count resulting from applying the RexNode condition.
void	setStatistics(Statistics statistics) Set the statistics for IndexGroupScan
boolean	supportsLimitPushdown() Checks if Json table reader supports limit push down.
boolean	supportsRestrictedScan() Whether this DbGroupScan supports creating a restricted (skip) scan
boolean	supportsSecondaryIndex()
String	toString()

Methods inherited from class org.apache.drill.exec.store.mapr.db.MapRDBGroupScan

applyAssignments, getDigest, getFormatPlugin, getIndexHint, getMetadataProvider, getOperatorAffinity, getPluginCostModel, getRowKeyOrdinal, getSchema, getSecondaryIndexCollection, getStorageConfig, getStoragePlugin, getTableMetadata, isFilterPushedDown, resetRegionsToScan, setCostFactor, setFilterPushedDown, setRegionsToScan

Methods inherited from class org.apache.drill.exec.physical.base.AbstractDbGroupScan

getRowKeyName, getRowKeyPath, isRestrictedScan

Methods inherited from class org.apache.drill.exec.physical.base.AbstractGroupScan

accept, applyFilter, enforceWidth, getAnalyzeInfoProvider, getColumnValueCount, getDistributionAffinity, getFiles, getFilter, getInitialAllocation, getMaxAllocation, getMinParallelizationWidth, getOperatorType, getPartitionColumns, getScanStats, getSelectionRoot, hasFiles, isExecutable, iterator, supportsFilterPushDown, supportsPartitionFilterPushdown, usedMetastore

Methods inherited from class org.apache.drill.exec.physical.base.AbstractBase

accept, getCost, getOperatorId, getSVMode, getUserName, isBufferedOperator, setCost, setMaxAllocation, setOperatorId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.physical.base.IndexGroupScan

getRowKeyOrdinal

Methods inherited from interface org.apache.drill.exec.physical.base.GroupScan

applyAssignments, applyFilter, enforceWidth, getAnalyzeInfoProvider, getColumnValueCount, getDigest, getFiles, getFilter, getMetadataProvider, getMinParallelizationWidth, getPartitionColumns, getScanStats, getSelectionRoot, getTableMetadata, hasFiles, supportsFilterPushDown, supportsPartitionFilterPushdown, usedMetastore

Methods inherited from interface org.apache.drill.exec.physical.base.PhysicalOperator

accept, getCost, getInitialAllocation, getMaxAllocation, getOperatorId, getOperatorType, getSVMode, getUserName, isBufferedOperator, isExecutable, setCost, setMaxAllocation, setOperatorId

Methods inherited from interface org.apache.drill.common.graph.GraphValue

accept

Methods inherited from interface java.lang.Iterable

forEach, iterator, spliterator

Methods inherited from interface org.apache.drill.exec.physical.base.HasAffinity

getDistributionAffinity, getOperatorAffinity

Field Detail

STAR_COLS

public static final int STAR_COLS

See Also:

Constant Field Values

TABLE_JSON

public static final String TABLE_JSON

See Also:

Constant Field Values

forcedRowCountMap

protected Map<org.apache.calcite.rex.RexNode,Double> forcedRowCountMap

stats

protected MapRDBStatistics stats

scanSpec

protected JsonScanSpec scanSpec

fullTableRowCount

protected double fullTableRowCount

fullTableEstimatedSize

protected double fullTableEstimatedSize

maxRecordsToRead

protected int maxRecordsToRead

need only read maxRecordsToRead records.

parallelizationWidth

protected int parallelizationWidth

Forced parallelization width

Constructor Detail

JsonTableGroupScan

public JsonTableGroupScan(String userName,
                          JsonScanSpec scanSpec,
                          FileSystemConfig storagePluginConfig,
                          MapRDBFormatPluginConfig formatPluginConfig,
                          List<SchemaPath> columns,
                          TupleMetadata schema,
                          StoragePluginRegistry pluginRegistry)
                   throws ExecutionSetupException

Throws:

ExecutionSetupException

JsonTableGroupScan

public JsonTableGroupScan(String userName,
                          AbstractStoragePlugin storagePlugin,
                          MapRDBFormatPlugin formatPlugin,
                          JsonScanSpec scanSpec,
                          List<SchemaPath> columns,
                          MetadataProviderManager metadataProviderManager)

JsonTableGroupScan

public JsonTableGroupScan(String userName,
                          AbstractStoragePlugin storagePlugin,
                          MapRDBFormatPlugin formatPlugin,
                          JsonScanSpec scanSpec,
                          List<SchemaPath> columns,
                          MapRDBStatistics stats,
                          TableMetadataProvider metadataProvider)

JsonTableGroupScan

protected JsonTableGroupScan(JsonTableGroupScan that)

Private constructor, used for cloning.

Parameters:

that - The HBaseGroupScan to clone

Method Detail

clone

public GroupScan clone(List<SchemaPath> columns)

Description copied from interface: GroupScan

Returns a clone of GroupScan instance, except that the new GroupScan will use the provided list of columns .

Specified by:

clone in interface GroupScan

Overrides:

clone in class AbstractGroupScan

clone

public GroupScan clone(JsonScanSpec scanSpec)

getRegionsToScan

protected NavigableMap<TabletFragmentInfo,String> getRegionsToScan()

Overrides:

getRegionsToScan in class MapRDBGroupScan

getRegionsToScan

protected NavigableMap<TabletFragmentInfo,String> getRegionsToScan(int scanRangeSizeMB)

getSubScanSpec

protected MapRDBSubScanSpec getSubScanSpec(TabletFragmentInfo tfi)

Specified by:

getSubScanSpec in class MapRDBGroupScan

getSpecificScan

public MapRDBSubScan getSpecificScan(int minorFragmentId)

Specified by:

getSpecificScan in interface GroupScan

getScanStats

public ScanStats getScanStats()

Overrides:

getScanStats in class AbstractGroupScan

getNewWithChildren

public PhysicalOperator getNewWithChildren(List<PhysicalOperator> children)

Description copied from interface: PhysicalOperator

Regenerate with this node with a new set of children. This is used in the case of materialization or optimization.

Specified by:

getNewWithChildren in interface PhysicalOperator

getTableName

public String getTableName()

Specified by:

getTableName in class MapRDBGroupScan

getIndexDesc

public com.mapr.db.index.IndexDesc getIndexDesc()

isDisablePushdown

public boolean isDisablePushdown()

canPushdownProjects

public boolean canPushdownProjects(List<SchemaPath> columns)

Description copied from interface: GroupScan

GroupScan should check the list of columns, and see if it could support all the columns in the list.

Specified by:

canPushdownProjects in interface GroupScan

Overrides:

canPushdownProjects in class MapRDBGroupScan

toString

public String toString()

Overrides:

toString in class Object

getScanSpec

public JsonScanSpec getScanSpec()

supportsSecondaryIndex

public boolean supportsSecondaryIndex()

Specified by:

supportsSecondaryIndex in interface DbGroupScan

Overrides:

supportsSecondaryIndex in class AbstractDbGroupScan

isIndexScan

public boolean isIndexScan()

Specified by:

isIndexScan in interface DbGroupScan

supportsRestrictedScan

public boolean supportsRestrictedScan()

Description copied from interface: DbGroupScan

Whether this DbGroupScan supports creating a restricted (skip) scan

Specified by:

supportsRestrictedScan in interface DbGroupScan

Overrides:

supportsRestrictedScan in class AbstractDbGroupScan

Returns:

true if restricted scan is supported, false otherwise

getRestrictedScan

public RestrictedJsonTableGroupScan getRestrictedScan(List<SchemaPath> columns)

Description copied from interface: DbGroupScan

If this DbGroupScan supports restricted scan, create a restricted scan from this DbGroupScan.

Specified by:

getRestrictedScan in interface DbGroupScan

Overrides:

getRestrictedScan in class AbstractDbGroupScan

Parameters:

columns - list of column paths

Returns:

a non-null DbGroupScan if restricted scan is supported, null otherwise

getAverageRowSizeStats

public MapRDBStatisticsPayload getAverageRowSizeStats(IndexDescriptor index)

Get the estimated average rowsize. DO NOT call this API directly. Call the stats API instead which modifies the counts based on preference options.

Parameters:

index, - to use for generating the estimate

Returns:

row count post filtering

getFirstKeyEstimatedStats

public MapRDBStatisticsPayload getFirstKeyEstimatedStats(org.ojai.store.QueryCondition condition,
                                                         IndexDescriptor index,
                                                         org.apache.calcite.rel.RelNode scanRel)

Get the estimated statistics after applying the RexNode condition. DO NOT call this API directly. Call the stats API instead which modifies the counts based on preference options.

Parameters:

condition, - filter to apply

index, - to use for generating the estimate

Returns:

row count post filtering

setRowCount

public void setRowCount(org.apache.calcite.rex.RexNode condition,
                        double count,
                        double capRowCount)

Set the row count resulting from applying the RexNode condition. Forced row counts will take precedence over stats row counts

Specified by:

setRowCount in interface DbGroupScan

Specified by:

setRowCount in interface IndexGroupScan

Parameters:

condition -

count -

capRowCount -

setStatistics

public void setStatistics(Statistics statistics)

Description copied from interface: IndexGroupScan

Set the statistics for IndexGroupScan

Specified by:

setStatistics in interface IndexGroupScan

Parameters:

statistics - which is necessary for index planning

getRowCount

public double getRowCount(org.apache.calcite.rex.RexNode condition,
                          org.apache.calcite.rel.RelNode scanRel)

Get the row count after applying the RexNode condition

Specified by:

getRowCount in interface DbGroupScan

Specified by:

getRowCount in interface IndexGroupScan

Parameters:

condition, - filter to apply

Returns:

row count post filtering

isDistributed

public boolean isDistributed()

Specified by:

isDistributed in interface GroupScan

Overrides:

isDistributed in class AbstractGroupScan

getStatistics

public MapRDBStatistics getStatistics()

Description copied from interface: DbGroupScan

Get the statistics for this DbGroupScan

Specified by:

getStatistics in interface DbGroupScan

Returns:

the Statistics for this Scan

setColumns

public void setColumns(List<SchemaPath> columns)

Specified by:

setColumns in interface IndexGroupScan

getColumns

public List<SchemaPath> getColumns()

Description copied from interface: GroupScan

Returns a list of columns scanned by this group scan

Specified by:

getColumns in interface DbGroupScan

Specified by:

getColumns in interface GroupScan

Specified by:

getColumns in interface IndexGroupScan

Overrides:

getColumns in class MapRDBGroupScan

getRangePartitionFunction

public PartitionFunction getRangePartitionFunction(List<FieldReference> refList)

Description copied from interface: DbGroupScan

Get a partition function instance for range based partitioning

Specified by:

getRangePartitionFunction in interface DbGroupScan

Overrides:

getRangePartitionFunction in class AbstractDbGroupScan

Parameters:

refList - a list of FieldReference exprs that are participating in the range partitioning

Returns:

instance of a partitioning function

convertToQueryCondition

public org.ojai.store.QueryCondition convertToQueryCondition(LogicalExpression condition)

Convert a given LogicalExpression condition into a QueryCondition condition

Parameters:

condition - expressed as a LogicalExpression

Returns:

QueryCondition condition equivalent to the given expression

supportsLimitPushdown

public boolean supportsLimitPushdown()

Checks if Json table reader supports limit push down.

Specified by:

supportsLimitPushdown in interface GroupScan

Overrides:

supportsLimitPushdown in class AbstractGroupScan

Returns:

true if limit push down is supported, false otherwise

applyLimit

public GroupScan applyLimit(int maxRecords)

Description copied from class: AbstractGroupScan

By default, return null to indicate row count based prune is not supported. Each group scan subclass should override, if it supports row count based prune.

Specified by:

applyLimit in interface GroupScan

Overrides:

applyLimit in class AbstractGroupScan

Parameters:

maxRecords - : the number of rows requested from group scan.

Returns:

a new instance of group scan if the prune is successful. null when either if row-based prune is not supported, or if prune is not successful.

getMaxParallelizationWidth

public int getMaxParallelizationWidth()

Specified by:

getMaxParallelizationWidth in interface GroupScan

Overrides:

getMaxParallelizationWidth in class MapRDBGroupScan

setParallelizationWidth

public void setParallelizationWidth(int width)

Specified by:

setParallelizationWidth in interface IndexGroupScan