org.apache.drill.exec.store.easy.text.compliant

Class TestCsvWithSchema

java.lang.Object

org.apache.drill.test.BaseTest

org.apache.drill.test.DrillTest

org.apache.drill.test.ClusterTest

org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

org.apache.drill.exec.store.easy.text.compliant.TestCsvWithSchema

public class TestCsvWithSchema
extends BaseCsvTest

Test the schema mechanism in the context of a simple CSV tables. The focus here is on the schema mechanism, which can be explored with simple tables of just a few rows.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.DrillTest

DrillTest.MemWatcher

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	FILE1_NAME

Fields inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

BIG_COL_SIZE, EMPTY_FILE, FILE_N_NAME, NESTED_DIR, NESTED_FILE, PART_DIR, ROOT_FILE, SCHEMA_BATCH_ENABLED, secondFile, testDir, validHeaders

Fields inherited from class org.apache.drill.test.ClusterTest

client, cluster, dirTestWatcher

Fields inherited from class org.apache.drill.test.DrillTest

logOutcome, objectMapper, REPEAT_RULE, thrownException, TIMEOUT

Constructor Summary

Constructors

Constructor and Description
TestCsvWithSchema()

Method Summary

Modifier and Type	Method and Description
static void	setup()
void	testBasicSchema() Test the simplest possible case: a table with one file: Column in projection, table, and schema Column in projection and table but not in schema. Column in projection and schema, but not in table. Column in projection, but not in schema or table.
void	testBlankCols() Demonstrate that CSV uses a comma to introduce a column, even if that column has no data.
void	testBlankColsWithNoDefaultValue() As above, but with a non-nullable numeric column with a default value.
void	testBlankColsWithNullableSchema() As above, but with a nullable numeric column.
void	testBlankColsWithSchema() Use the same data set as above tests, but use a schema to do type conversion.
void	testBool() Test the many ways to specify True for boolean columns.
void	testDateColDefault() Use a user-provided default value for a missing required column.
void	testDecimal() Basic decimal sanity test showing rounding, using default values, and so on.
void	testDecimalDefaultOverflow() Test proper handling of overflow for a default value.
void	testDecimalDefaultRound() Verify that the decimal default value is rounded according to the scale specified in the decimal type.
void	testDecimalNoPrecOrScale() Verify that a decimal type without precision or scale defaults to precision of 38, scale of 0.
void	testDecimalNoScale() Verify that a decimal type with no scale defaults to a scale of 0.
void	testDecimalOverflow() Test decimal overflow during data reads.
void	testExplicitSort() Because V3 uses VARCHAR for missing columns, and handles ragged rows, there is no vector corruption, and the sort operator sees a uniform schema, even without a schema.
void	testInvalidDecimalSchema()
void	testMissingCols()
void	testMissingColsNullable() Demonstrate that CSV works for a schema with nullable types when columns are missing (there is no comma to introduce an empty field in the data.)
void	testMissingColsReq() Verify that, if a schema is provided, a column is missing, and there is no default, that the mode is left at required and the column is filled with zeros.
void	testMissingColsReqDefault() Verify the behavior of missing columns, not null mode, with a default value.
void	testMissingRequiredCol() Show that the projection framework generates a reasonable default value if told to create a required column that does not exist.
void	testMultiFileSchema() Use a schema with explicit projection to get a consistent view of the table schema, even if columns are missing, rows are ragged, and column order changes.
void	testMultiFileSchemaMissingCol() Test the case that a file does not contain a required column (in this case, id in the third file.) There are two choices.
void	testMultiFragmentStrictSchema() Test a strict schema where it is needed most: in a scan with multiple fragments, each of which sees a different reader schema.
void	testRequiredColDefault() Use a user-provided default value for a missing required column.
void	testSchemaExplicitSort() Adding a schema makes the data types more uniform, and fills in defaults for missing columns.
void	testWildcardLenientSchema() Test lenient wildcard projection.
void	testWildcardSortFailure() Show that, without schema, the hard schema change for the "missing" gender column causes an error in the sort operator when presented with one batch in which gender is VARCHAR, another in which it is Nullable INT.
void	testWildcardStrictSchema() Test wildcard projection with a strict schema: only schema columns are projected.

Methods inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

buildBigColFile, buildFile, buildFile, buildNestedTable, buildTable, enableMultiScan, enableSchema, enableSchemaSupport, resetMultiScan, resetSchema, resetSchemaSupport, setup, setup

Methods inherited from class org.apache.drill.test.ClusterTest

getFile, queryBuilder, run, runAndLog, runAndPrint, runAndPrint, shutdown, startCluster, testBuilder

Methods inherited from class org.apache.drill.test.DrillTest

escapeJsonString, finishDrillTest, initDrillTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

FILE1_NAME

protected static final String FILE1_NAME

See Also:

Constant Field Values

Constructor Detail

TestCsvWithSchema

public TestCsvWithSchema()

Method Detail

setup

public static void setup()
                  throws Exception

Throws:

Exception

testBasicSchema

public void testBasicSchema()
                     throws Exception

Test the simplest possible case: a table with one file:

• Column in projection, table, and schema
• Column in projection and table but not in schema.
• Column in projection and schema, but not in table.
• Column in projection, but not in schema or table.

Throws:

Exception

testMissingRequiredCol

public void testMissingRequiredCol()
                            throws Exception

Show that the projection framework generates a reasonable default value if told to create a required column that does not exist. In this case, the default default [sic] value for an INT column is 0. There is no "default" value set in the schema, so we use a "default default" instead.)

Throws:

Exception

testRequiredColDefault

public void testRequiredColDefault()
                            throws Exception

Use a user-provided default value for a missing required column.

Throws:

Exception

testDateColDefault

public void testDateColDefault()
                        throws Exception

Use a user-provided default value for a missing required column.

Throws:

Exception

testMultiFileSchema

public void testMultiFileSchema()
                         throws Exception

Use a schema with explicit projection to get a consistent view of the table schema, even if columns are missing, rows are ragged, and column order changes.

Force the scans to occur in distinct fragments so the order of the file batches is random.

Throws:

Exception

testWildcardSortFailure

public void testWildcardSortFailure()
                             throws Exception

Show that, without schema, the hard schema change for the "missing" gender column causes an error in the sort operator when presented with one batch in which gender is VARCHAR, another in which it is Nullable INT. This is a consequence of using SELECT * on a distributed scan.

Throws:

Exception

testExplicitSort

public void testExplicitSort()
                      throws Exception

Because V3 uses VARCHAR for missing columns, and handles ragged rows, there is no vector corruption, and the sort operator sees a uniform schema, even without a schema.

This and other tests enable multiple scan fragments, even for small files, then run the test multiple times to generate the result set in different orders (file1 first sometimes, file2 other times.)

Throws:

Exception

testSchemaExplicitSort

public void testSchemaExplicitSort()
                            throws Exception

Adding a schema makes the data types more uniform, and fills in defaults for missing columns. Note that the default is not applied (at least at present) if a column is missing within a file that says it has the column.

Throws:

Exception

testMultiFileSchemaMissingCol

public void testMultiFileSchemaMissingCol()
                                   throws Exception

Test the case that a file does not contain a required column (in this case, id in the third file.) There are two choices. 1) we could fail the query, 2) we can muddle through as best we can. The scan framework chooses to muddle through by assuming a default value of 0 for the missing int column.

Inserts an ORDER BY to force a single batch in a known order. Assumes the other ORDER BY tests pass.

This test shows that having consistent types is sufficient for the sort operator to work; the DAG will include a project operator that reorders the columns when produced by readers in different orders. (Column ordering is more an abstract concept anyway in a columnar system such as Drill.)

Throws:

Exception

testWildcardLenientSchema

public void testWildcardLenientSchema()
                               throws Exception

Test lenient wildcard projection. The schema contains all columns in the table; the schema ensures a consistent schema regardless of file shape or read order. The sort operator works because it sees the consistent schema, despite great variation in inputs.

Throws:

Exception

testWildcardStrictSchema

public void testWildcardStrictSchema()
                              throws Exception

Test wildcard projection with a strict schema: only schema columns are projected.

Throws:

Exception

testMultiFragmentStrictSchema

public void testMultiFragmentStrictSchema()
                                   throws Exception

Test a strict schema where it is needed most: in a scan with multiple fragments, each of which sees a different reader schema. The output schema ensures that each scan independently reports the same schema, so that the downstream sort operator gets a single consistent batch schema.

Throws:

Exception

testBool

public void testBool()
              throws Exception

Test the many ways to specify True for boolean columns. Anything that is not true is false.

Throws:

Exception

testDecimal

public void testDecimal()
                 throws Exception

Basic decimal sanity test showing rounding, using default values, and so on.

Throws:

Exception

testDecimalNoPrecOrScale

public void testDecimalNoPrecOrScale()
                              throws Exception

Verify that a decimal type without precision or scale defaults to precision of 38, scale of 0.

Throws:

Exception

testDecimalNoScale

public void testDecimalNoScale()
                        throws Exception

Verify that a decimal type with no scale defaults to a scale of 0.

Throws:

Exception

testDecimalDefaultRound

public void testDecimalDefaultRound()
                             throws Exception

Verify that the decimal default value is rounded according to the scale specified in the decimal type.

Throws:

Exception

testDecimalOverflow

public void testDecimalOverflow()
                         throws Exception

Test decimal overflow during data reads. The overflow occurs after rounding the data value of 99999.9 to 100000.

Throws:

Exception

testDecimalDefaultOverflow

public void testDecimalDefaultOverflow()
                                throws Exception

Test proper handling of overflow for a default value. In this case, the overflow occurs after rounding.

Throws:

Exception

testInvalidDecimalSchema

public void testInvalidDecimalSchema()
                              throws Exception

Throws:

Exception

testMissingCols

public void testMissingCols()
                     throws Exception

Throws:

Exception

testMissingColsReq

public void testMissingColsReq()
                        throws Exception

Verify that, if a schema is provided, a column is missing, and there is no default, that the mode is left at required and the column is filled with zeros. Note that this behavior is specific to the text readers: if have no schema, even an missing VARCHAR column will be REQUIRED and set to an empty string (reason: if the column does appear it will be a required VARCHAR, so, to be consistent, missing columns are also required.)

Throws:

Exception

testMissingColsReqDefault

public void testMissingColsReqDefault()
                               throws Exception

Verify the behavior of missing columns, not null mode, with a default value.

Throws:

Exception

testMissingColsNullable

public void testMissingColsNullable()
                             throws Exception

Demonstrate that CSV works for a schema with nullable types when columns are missing (there is no comma to introduce an empty field in the data.)

Throws:

Exception

testBlankCols

public void testBlankCols()
                   throws Exception

Demonstrate that CSV uses a comma to introduce a column, even if that column has no data. In this case, CSV assumes the value of the column is a blank string.

Such a schema cannot be converted to a number or date column, even nullable, because a blank string is neither a valid number nor a valid date.

Throws:

Exception

testBlankColsWithSchema

public void testBlankColsWithSchema()
                             throws Exception

Use the same data set as above tests, but use a schema to do type conversion. Blank columns become 0 for numeric non-nullable, nulls for nullable non-numeric.

Throws:

Exception

testBlankColsWithNullableSchema

public void testBlankColsWithNullableSchema()
                                     throws Exception

As above, but with a nullable numeric column. Here, by default, blank values become nulls.

Throws:

Exception

testBlankColsWithNoDefaultValue

public void testBlankColsWithNoDefaultValue()
                                     throws Exception

As above, but with a non-nullable numeric column with a default value.

Throws:

Exception