org.apache.drill.exec.store.easy.text

Class TextFormatPlugin

java.lang.Object

org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin<TextFormatPlugin.TextFormatConfig>

org.apache.drill.exec.store.easy.text.TextFormatPlugin

All Implemented Interfaces:

FormatPlugin

public class TextFormatPlugin
extends EasyFormatPlugin<TextFormatPlugin.TextFormatConfig>

Text format plugin for CSV and other delimited text formats. Allows use of a "provided schema", including using table properties on that schema to override "static" ("or default") properties defined in the plugin config. Allows, say, having ".csv" files in which some have no headers (the default) and some do have headers (as specified via table properties in the provided schema.)

Makes use of the scan framework and the result set loader mechanism to allow tight control of the size of produced batches (as well as to support provided schema.)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TextFormatPlugin.TextFormatConfig

Nested classes/interfaces inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

EasyFormatPlugin.EasyFormatConfig, EasyFormatPlugin.EasyFormatConfigBuilder

Field Summary

Fields

Modifier and Type	Field and Description
static String	COMMENT_CHAR_PROP
static String	DELIMITER_PROP
static String	HAS_HEADERS_PROP
static String	LINE_DELIM_PROP
static int	MAX_CHARS_PER_COLUMN
static int	MAXIMUM_NUMBER_COLUMNS
static char	NULL_CHAR
static String	PARSE_UNESCAPED_QUOTES_PROP
static String	QUOTE_ESCAPE_PROP
static String	QUOTE_PROP
static String	SKIP_FIRST_LINE_PROP
static String	TEXT_PREFIX
static String	TRIM_WHITESPACE_PROP
static String	WRITER_OPERATOR_TYPE

Fields inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

formatConfig

Constructor Summary

Constructors

Constructor and Description
TextFormatPlugin(String name, DrillbitContext context, org.apache.hadoop.conf.Configuration fsConf, StoragePluginConfig storageConfig)
TextFormatPlugin(String name, DrillbitContext context, org.apache.hadoop.conf.Configuration fsConf, StoragePluginConfig config, TextFormatPlugin.TextFormatConfig formatPluginConfig)

Method Summary

Modifier and Type	Method and Description
protected FileScanFramework.FileScanBuilder	frameworkBuilder(OptionManager options, EasySubScan scan) Create the plugin-specific framework that manages the scan.
AbstractGroupScan	getGroupScan(String userName, FileSelection selection, List<SchemaPath> columns, MetadataProviderManager metadataProviderManager)
AbstractGroupScan	getGroupScan(String userName, FileSelection selection, List<SchemaPath> columns, OptionManager options, MetadataProviderManager metadataProviderManager)
org.apache.drill.exec.store.RecordWriter	getRecordWriter(FragmentContext context, EasyWriter writer)
protected ScanStats	getScanStats(PlannerSettings settings, EasyGroupScan scan)

Methods inherited from class org.apache.drill.exec.store.dfs.easy.EasyFormatPlugin

easyConfig, getConfig, getContext, getFsConf, getGroupScan, getMatcher, getName, getOptimizerRules, getReaderBatch, getReaderOperatorType, getRecordReader, getStatisticsRecordWriter, getStorageConfig, getWriter, getWriterBatch, getWriterOperatorType, initScanBuilder, isBlockSplittable, isCompressible, isStatisticsRecordWriter, newBatchReader, readStatistics, supportsAutoPartitioning, supportsFileImplicitColumns, supportsLimitPushdown, supportsPushDown, supportsRead, supportsStatistics, supportsWrite, useEnhancedScan, writeStatistics

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.drill.exec.store.dfs.FormatPlugin

getGroupScan

Field Detail

MAXIMUM_NUMBER_COLUMNS

public static final int MAXIMUM_NUMBER_COLUMNS

See Also:

Constant Field Values

MAX_CHARS_PER_COLUMN

public static final int MAX_CHARS_PER_COLUMN

See Also:

Constant Field Values

NULL_CHAR

public static final char NULL_CHAR

See Also:

Constant Field Values

TEXT_PREFIX

public static final String TEXT_PREFIX

HAS_HEADERS_PROP

public static final String HAS_HEADERS_PROP

SKIP_FIRST_LINE_PROP

public static final String SKIP_FIRST_LINE_PROP

DELIMITER_PROP

public static final String DELIMITER_PROP

COMMENT_CHAR_PROP

public static final String COMMENT_CHAR_PROP

QUOTE_PROP

public static final String QUOTE_PROP

QUOTE_ESCAPE_PROP

public static final String QUOTE_ESCAPE_PROP

LINE_DELIM_PROP

public static final String LINE_DELIM_PROP

TRIM_WHITESPACE_PROP

public static final String TRIM_WHITESPACE_PROP

PARSE_UNESCAPED_QUOTES_PROP

public static final String PARSE_UNESCAPED_QUOTES_PROP

WRITER_OPERATOR_TYPE

public static final String WRITER_OPERATOR_TYPE

See Also:

Constant Field Values

Constructor Detail

TextFormatPlugin

public TextFormatPlugin(String name,
                        DrillbitContext context,
                        org.apache.hadoop.conf.Configuration fsConf,
                        StoragePluginConfig storageConfig)

TextFormatPlugin

public TextFormatPlugin(String name,
                        DrillbitContext context,
                        org.apache.hadoop.conf.Configuration fsConf,
                        StoragePluginConfig config,
                        TextFormatPlugin.TextFormatConfig formatPluginConfig)

Method Detail

getGroupScan

public AbstractGroupScan getGroupScan(String userName,
                                      FileSelection selection,
                                      List<SchemaPath> columns,
                                      MetadataProviderManager metadataProviderManager)
                               throws IOException

Specified by:

getGroupScan in interface FormatPlugin

Overrides:

getGroupScan in class EasyFormatPlugin<TextFormatPlugin.TextFormatConfig>

Throws:

IOException

getGroupScan

public AbstractGroupScan getGroupScan(String userName,
                                      FileSelection selection,
                                      List<SchemaPath> columns,
                                      OptionManager options,
                                      MetadataProviderManager metadataProviderManager)
                               throws IOException

Throws:

IOException

frameworkBuilder

protected FileScanFramework.FileScanBuilder frameworkBuilder(OptionManager options,
                                                             EasySubScan scan)
                                                      throws ExecutionSetupException

Description copied from class: EasyFormatPlugin

Create the plugin-specific framework that manages the scan. The framework creates batch readers one by one for each file or block. It defines semantic rules for projection. It handles "early" or "late" schema readers. A typical framework builds on standardized frameworks for files in general or text files in particular.

Overrides:

frameworkBuilder in class EasyFormatPlugin<TextFormatPlugin.TextFormatConfig>

scan - the physical operation definition for the scan operation. Contains one or more files to read. (The Easy format plugin works only for files.)

Returns:

the scan framework which orchestrates the scan operation across potentially many files

Throws:

ExecutionSetupException - for all setup failures

getRecordWriter

public org.apache.drill.exec.store.RecordWriter getRecordWriter(FragmentContext context,
                                                                EasyWriter writer)
                                                         throws IOException

Overrides:

getRecordWriter in class EasyFormatPlugin<TextFormatPlugin.TextFormatConfig>

Throws:

IOException

getScanStats

protected ScanStats getScanStats(PlannerSettings settings,
                                 EasyGroupScan scan)

Overrides:

getScanStats in class EasyFormatPlugin<TextFormatPlugin.TextFormatConfig>