org.apache.drill.exec.store.easy.text.compliant

Class TestCsvTableProperties

java.lang.Object

org.apache.drill.test.BaseTest

org.apache.drill.test.DrillTest

org.apache.drill.test.ClusterTest

org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

org.apache.drill.exec.store.easy.text.compliant.TestCsvTableProperties

public class TestCsvTableProperties
extends BaseCsvTest

Test table properties with the compliant text reader. The table properties override selected properties in the format plugin config. The tests here start with a "stock" CSV format plugin config without headers. We then use table properties to vary the table format: without headers, skip first row, with headers.

The tests also verify that, without headers, if a schema is provided, the text format plugin will create columns using that schema rather than using the "columns" array column.

See Also:

for similar tests using table properties within SQL

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.DrillTest

DrillTest.MemWatcher

Field Summary

Fields inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

BIG_COL_SIZE, EMPTY_FILE, FILE_N_NAME, NESTED_DIR, NESTED_FILE, PART_DIR, ROOT_FILE, SCHEMA_BATCH_ENABLED, secondFile, testDir, validHeaders

Fields inherited from class org.apache.drill.test.ClusterTest

client, cluster, dirTestWatcher

Fields inherited from class org.apache.drill.test.DrillTest

logOutcome, objectMapper, REPEAT_RULE, thrownException, TIMEOUT

Constructor Summary

Constructors

Constructor and Description
TestCsvTableProperties()

Method Summary

Modifier and Type	Method and Description
static void	setup()
void	testComment()
void	testDelimiter()
void	testDoubleQuoteChars() Test that the quote escape can be the quote character itself.
void	testHeadersWithoutSchema()
void	testHeadersWithSchema()
void	testKeepWitespace() Trim leading and trailing whitespace.
void	testMessyQuotes() The legacy "V2" text reader had special handling for quotes that appear inside fields.
void	testNewlineProp() Verify that a custom newline character works, and that the symbol '\n' can be used in SQL and is stored properly in the schema file.
void	testNoComment() Users have complained about the comment character.
void	testNoHeadersWithoutSchema()
void	testNoHeadersWithSchema()
void	testNoHeadersWithSchemaExtraCols()
void	testQuoteChars() Test quote and quote escape
void	testQuotesAndCustomNewLine()
void	testSkipHeadersWithoutSchema()
void	testSkipHeadersWithSchema()
void	testSpecialChars() End-to-end test of special characters for delimiter (a control character, ASCII 0x01) and quote (same as the SQL quote.)
void	testTrimWitespace() Trim leading and trailing whitespace.

Methods inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

buildBigColFile, buildFile, buildFile, buildNestedTable, buildTable, enableMultiScan, enableSchema, enableSchemaSupport, resetMultiScan, resetSchema, resetSchemaSupport, setup, setup

Methods inherited from class org.apache.drill.test.ClusterTest

getFile, queryBuilder, run, runAndLog, runAndPrint, runAndPrint, shutdown, startCluster, testBuilder

Methods inherited from class org.apache.drill.test.DrillTest

escapeJsonString, finishDrillTest, initDrillTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestCsvTableProperties

public TestCsvTableProperties()

Method Detail

setup

public static void setup()
                  throws Exception

Throws:

Exception

testNoHeadersWithSchema

public void testNoHeadersWithSchema()
                             throws Exception

Throws:

Exception

testNoHeadersWithoutSchema

public void testNoHeadersWithoutSchema()
                                throws Exception

Throws:

Exception

testNoHeadersWithSchemaExtraCols

public void testNoHeadersWithSchemaExtraCols()
                                      throws Exception

Throws:

Exception

testSkipHeadersWithSchema

public void testSkipHeadersWithSchema()
                               throws Exception

Throws:

Exception

testSkipHeadersWithoutSchema

public void testSkipHeadersWithoutSchema()
                                  throws Exception

Throws:

Exception

testHeadersWithSchema

public void testHeadersWithSchema()
                           throws Exception

Throws:

Exception

testHeadersWithoutSchema

public void testHeadersWithoutSchema()
                              throws Exception

Throws:

Exception

testDelimiter

public void testDelimiter()
                   throws Exception

Throws:

Exception

testComment

public void testComment()
                 throws Exception

Throws:

Exception

testNoComment

public void testNoComment()
                   throws Exception

Users have complained about the comment character. We usually suggest they change it to some other character. This test verifies that the plugin will choose the ASCII NUL (0) character if the comment property is set to a blank string. Since NUL never occurs in the input, the result is to essentially disable comment support.

Throws:

Exception

testQuoteChars

public void testQuoteChars()
                    throws Exception

Test quote and quote escape

Throws:

Exception

testDoubleQuoteChars

public void testDoubleQuoteChars()
                          throws Exception

Test that the quote escape can be the quote character itself. In this case, <escape>& is the same as <quote><quote> and is considered to be an escaped quote. There is no "orphan" escape case.

Throws:

Exception

testQuotesAndCustomNewLine

public void testQuotesAndCustomNewLine()
                                throws Exception

Throws:

Exception

testSpecialChars

public void testSpecialChars()
                      throws Exception

End-to-end test of special characters for delimiter (a control character, ASCII 0x01) and quote (same as the SQL quote.)

Throws:

Exception

testNewlineProp

public void testNewlineProp()
                     throws Exception

Verify that a custom newline character works, and that the symbol '\n' can be used in SQL and is stored properly in the schema file.

Throws:

Exception

testMessyQuotes

public void testMessyQuotes()
                     throws Exception

The legacy "V2" text reader had special handling for quotes that appear inside fields. Example:

 first"field"here,another "field

Since behavior in this case is ill-defined, the reader apparently treated quotes as normal characters unless the field started with a quote. There is an option in the UniVocity code to set this behavior, but it is not exposed in Drill. So, this test verifies the non-customizable messy quote handling logic.

If a field starts with a quote, quoting rules kick in, including the quote escape, which is, by default, itself a quote. So
"foo""bar"
is read as
foo"bar
But, for fields not starting with a quote, the quote escape is ignored, so:
foo""bar
is read as
foo""bar
This seems more like a bug than a feature, but it does appear to be how the "new" text reader always worked, so the behavior is preserved.

Also, seems that the text reader supported embedded newlines, even though such behavior will not work if the embedded newline occurs near a split. In this case, the reader will scan forward to find a record delimiter (a newline by default), will find the embedded newline, and will read a partial first record. Again, this appears to be legacy behavior, and so is preserved, even if broken.

The key thing is that if the CSV is well-formed (no messy quotes, properly quoted fields with proper escapes, no embedded newlines) then things will work OK.

Throws:

Exception

testKeepWitespace

public void testKeepWitespace()
                       throws Exception

Trim leading and trailing whitespace. This setting is currently only available via table properties.

Throws:

Exception

testTrimWitespace

public void testTrimWitespace()
                       throws Exception

Trim leading and trailing whitespace. This setting is currently only available via table properties.

Throws:

Exception