org.apache.drill.exec.store.easy.text.compliant

Class TestCsvWithHeaders

java.lang.Object

org.apache.drill.test.BaseTest

org.apache.drill.test.DrillTest

org.apache.drill.test.ClusterTest

org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

org.apache.drill.exec.store.easy.text.compliant.TestCsvWithHeaders

public class TestCsvWithHeaders
extends BaseCsvTest

Sanity test of CSV files with headers.

Open issues:

• DRILL-7080: A query like SELECT *, dir0 produces the result schema of (dir0, a, b, ...) in V2 and (a, b, ... dir0, dir00) in V3. This seems to be a bug in the Project operator.

The tests assume that the "early schema" mechanism is disabled: that the first batch either contains data, or that the first batch is empty only if there is no data at all to be read.

See Also:

TestHeaderBuilder

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.DrillTest

DrillTest.MemWatcher

Field Summary

Fields inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

BIG_COL_SIZE, EMPTY_FILE, FILE_N_NAME, NESTED_DIR, NESTED_FILE, PART_DIR, ROOT_FILE, SCHEMA_BATCH_ENABLED, secondFile, testDir, validHeaders

Fields inherited from class org.apache.drill.test.ClusterTest

client, cluster, dirTestWatcher

Fields inherited from class org.apache.drill.test.DrillTest

logOutcome, objectMapper, REPEAT_RULE, thrownException, TIMEOUT

Constructor Summary

Constructors

Constructor and Description
TestCsvWithHeaders()

Method Summary

Modifier and Type	Method and Description
void	doTestExplicitPartitionsMultiFiles() Test using partition columns with partitioned files in V3.
static void	setup()
void	testColsWithWildcard()
void	testColumnsCol() The column name `columns` is treated as a plain old column when using column headers.
void	testColumnsIndex() The column name `columns` is treated as a plain old column when using column headers.
void	testColumnsIndexMissing() If columns[x] is used, then this can't possibly match a valid text reader column, so raise an error instead.
void	testColumnsMissing()
void	testCsvHeadersCaseInsensitive()
void	testDataNoNewline()
void	testDupColumn()
void	testEmptyBody() A file with a header has a schema, but has no rows.
void	testEmptyCsvHeaders() Trivial case: empty header.
void	testEmptyFile() An empty file with schema is invalid: there is no header line and so there is no schema.
void	testHeadersNoNewline()
void	testHeadersOnly()
void	testHugeColumn()
void	testImplicitColsExplicitSelect() Verify that implicit columns are recognized and populated.
void	testImplicitColWildcard() Verify that implicit columns are recognized and populated.
void	testInvalidCsvHeaders()
void	testPartitionColsExplicit() V3 allows the use of partition columns, even for a non-partitioned file.
void	testPartitionExpansion() Test partition expansion.
void	testRaggedRows() Test that ragged rows result in the "missing" columns being filled in with the moral equivalent of a null column for CSV: a blank string.
void	testValidCsvHeaders()
void	testWildcard() Verify that the wildcard expands columns to the header names, including case
void	testWildcardAndPartitionsMultiFiles() Test the use of partition columns with the wildcard.

Methods inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

buildBigColFile, buildFile, buildFile, buildNestedTable, buildTable, enableMultiScan, enableSchema, enableSchemaSupport, resetMultiScan, resetSchema, resetSchemaSupport, setup, setup

Methods inherited from class org.apache.drill.test.ClusterTest

getFile, queryBuilder, run, runAndLog, runAndPrint, runAndPrint, shutdown, startCluster, testBuilder

Methods inherited from class org.apache.drill.test.DrillTest

escapeJsonString, finishDrillTest, initDrillTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestCsvWithHeaders

public TestCsvWithHeaders()

Method Detail

setup

public static void setup()
                  throws Exception

Throws:

Exception

testEmptyFile

public void testEmptyFile()
                   throws IOException

An empty file with schema is invalid: there is no header line and so there is no schema. It is probably not helpful to return a batch with an empty schema; doing so would simply conflict with the schema of a non-empty file. Also, there is no reason to throw an error; this is not a problem serious enough to fail the query. Instead, we elect to simply return no results at all: no schema and no data.

Prior research revealed that most DB engines can handle a null empty result set: no schema, no rows. For example:
SELECT * FROM VALUES ();
The implementation tested here follows that pattern.

Throws:

IOException

See Also:

TestCsvWithoutHeaders.testEmptyFile()

testEmptyCsvHeaders

public void testEmptyCsvHeaders()
                         throws IOException

Trivial case: empty header. This case should fail.

Throws:

IOException

testHeadersNoNewline

public void testHeadersNoNewline()
                          throws IOException

Throws:

IOException

testEmptyBody

public void testEmptyBody()
                   throws IOException

A file with a header has a schema, but has no rows. This is different than the empty file case because we do, in fact, know the schema.

Throws:

IOException

testValidCsvHeaders

public void testValidCsvHeaders()
                         throws IOException

Throws:

IOException

testInvalidCsvHeaders

public void testInvalidCsvHeaders()
                           throws IOException

Throws:

IOException

testCsvHeadersCaseInsensitive

public void testCsvHeadersCaseInsensitive()
                                   throws IOException

Throws:

IOException

testWildcard

public void testWildcard()
                  throws IOException

Verify that the wildcard expands columns to the header names, including case

Throws:

IOException

testDataNoNewline

public void testDataNoNewline()
                       throws IOException

Throws:

IOException

testImplicitColsExplicitSelect

public void testImplicitColsExplicitSelect()
                                    throws IOException

Verify that implicit columns are recognized and populated. Sanity test of just one implicit column. V3 uses non-nullable VARCHAR for file metadata columns.

Throws:

IOException

testImplicitColWildcard

public void testImplicitColWildcard()
                             throws IOException

Verify that implicit columns are recognized and populated. Sanity test of just one implicit column. V3 uses non-nullable VARCHAR for file metadata columns.

Throws:

IOException

testColsWithWildcard

public void testColsWithWildcard()
                          throws IOException

Throws:

IOException

testPartitionColsExplicit

public void testPartitionColsExplicit()
                               throws IOException

V3 allows the use of partition columns, even for a non-partitioned file. The columns are null of type Nullable VARCHAR. This is area of Drill is a bit murky: it seems reasonable to support partition columns consistently rather than conditionally based on the structure of the input.

Throws:

IOException

testDupColumn

public void testDupColumn()
                   throws IOException

Throws:

IOException

testRaggedRows

public void testRaggedRows()
                    throws IOException

Test that ragged rows result in the "missing" columns being filled in with the moral equivalent of a null column for CSV: a blank string.

Throws:

IOException

testPartitionExpansion

public void testPartitionExpansion()

Test partition expansion.

This test is tricky because it will return two data batches (preceded by an empty schema batch.) File read order is random so we have to expect the files in either order.

V3 puts partition columns after data columns (so that data columns don't shift positions if files are nested to another level.)

testWildcardAndPartitionsMultiFiles

public void testWildcardAndPartitionsMultiFiles()

Test the use of partition columns with the wildcard. This works for file metadata columns, but confuses the project operator when used for partition columns. DRILL-7080. Still broken in V3 because this appears to be a Project operator issue, not reader issue. Not that the partition column moves after data columns.

doTestExplicitPartitionsMultiFiles

public void doTestExplicitPartitionsMultiFiles()

Test using partition columns with partitioned files in V3. Although the file is nested to one level, both dir0 and dir1 are nullable VARCHAR. See TestPartitionRace to show that the types and schemas are consistent even when used across multiple scans.

testColumnsCol

public void testColumnsCol()
                    throws IOException

The column name `columns` is treated as a plain old column when using column headers.

Throws:

IOException

testColumnsIndex

public void testColumnsIndex()
                      throws Exception

The column name `columns` is treated as a plain old column when using column headers. If used with an index, validation will fail because the VarChar column is not an array

Throws:

Exception

testColumnsMissing

public void testColumnsMissing()
                        throws IOException

Throws:

IOException

testColumnsIndexMissing

public void testColumnsIndexMissing()
                             throws Exception

If columns[x] is used, then this can't possibly match a valid text reader column, so raise an error instead.

Throws:

Exception

testHugeColumn

public void testHugeColumn()
                    throws IOException

Throws:

IOException

testHeadersOnly

public void testHeadersOnly()
                     throws Exception

Throws:

Exception