org.apache.drill.exec.store.log

Class LogFormatPlugin

java.lang.Object

org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin<LogFormatConfig>

org.apache.drill.exec.store.log.LogFormatPlugin

All Implemented Interfaces:

FormatPlugin

public class LogFormatPlugin
extends EasyFormatPlugin<LogFormatConfig>

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

EasyFormatPlugin.EasyFormatConfig, EasyFormatPlugin.EasyFormatConfigBuilder

Field Summary

Fields

Modifier and Type	Field and Description
static String	MAX_ERRORS_PROP
static String	OPERATOR_TYPE
static String	PLUGIN_NAME
static String	PROP_PREFIX
static String	REGEX_PROP

Fields inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

formatConfig

Constructor Summary

Constructors

Constructor and Description
LogFormatPlugin(String name, DrillbitContext context, org.apache.hadoop.conf.Configuration fsConf, StoragePluginConfig storageConfig, LogFormatConfig formatConfig)

Method Summary

Modifier and Type	Method and Description
protected FileScanFramework.FileScanBuilder	frameworkBuilder(OptionManager options, EasySubScan scan) Build a file scan framework for this plugin.
int	maxErrors(TupleMetadata providedSchema)

Methods inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

easyConfig, getConfig, getContext, getFsConf, getGroupScan, getGroupScan, getMatcher, getName, getOptimizerRules, getReaderBatch, getReaderOperatorType, getRecordReader, getRecordWriter, getScanStats, getStatisticsRecordWriter, getStorageConfig, getWriter, getWriterBatch, getWriterOperatorType, initScanBuilder, isBlockSplittable, isCompressible, isStatisticsRecordWriter, newBatchReader, readStatistics, supportsAutoPartitioning, supportsFileImplicitColumns, supportsLimitPushdown, supportsPushDown, supportsRead, supportsStatistics, supportsWrite, useEnhancedScan, writeStatistics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.store.dfs.FormatPlugin

getGroupScan, getGroupScan

Field Detail

PLUGIN_NAME

public static final String PLUGIN_NAME

See Also:

Constant Field Values

PROP_PREFIX

public static final String PROP_PREFIX

REGEX_PROP

public static final String REGEX_PROP

MAX_ERRORS_PROP

public static final String MAX_ERRORS_PROP

OPERATOR_TYPE

public static final String OPERATOR_TYPE

See Also:

Constant Field Values

Constructor Detail

LogFormatPlugin

public LogFormatPlugin(String name,
                       DrillbitContext context,
                       org.apache.hadoop.conf.Configuration fsConf,
                       StoragePluginConfig storageConfig,
                       LogFormatConfig formatConfig)

Method Detail

frameworkBuilder

protected FileScanFramework.FileScanBuilder frameworkBuilder(OptionManager options,
                                                             EasySubScan scan)
                                                      throws ExecutionSetupException

Build a file scan framework for this plugin.

This plugin was created before the concept of "provided schema" was available. This plugin does, however, support a provided schema and table properties. The code here handles the various cases.

For the regex and max errors:

• Use the table property, if a schema is provided and the table property is set.
• Else, use the format config property.

For columns:

• If no schema is provided (or the schema contains only table properties with no columns), use the column names and types from the plugin config.
• If a schema is provided, and the plugin defines columns, use the column types from the provided schema. Columns are matched by name. Provided schema type override any types specified in the plugin.
• If a schema is provided, and the plugin config defines no columns, use the column names and types from the provided schema. The columns are assumed to appear in the same order as regex fields.
• If the regex has more groups than either schema has columns, fill the extras with field_n of type VARCHAR.

Typical use cases:

• Minimum config: only a regex in either plugin config or table properties.
• Plugin config defines regex, field names and types. (The typical approach in Drill 1.16 and before.
• Plugin config defines the regex and field names. The provided schema defines types. (Separates physical and logical table definitions.
• Provided schema defines the regex and columns. May simplify configuration as all table information is in one place. Allows different regex patterns for different tables of the same file suffix.

Overrides:

frameworkBuilder in class EasyFormatPlugin<LogFormatConfig>

scan - the physical operation definition for the scan operation. Contains one or more files to read. (The Easy format plugin works only for files.)

Returns:

the scan framework which orchestrates the scan operation across potentially many files

Throws:

ExecutionSetupException - for all setup failures

maxErrors

public int maxErrors(TupleMetadata providedSchema)