org.apache.drill.exec.physical.impl.scan.project

Class TestSchemaSmoothing

java.lang.Object

org.apache.drill.test.BaseTest

org.apache.drill.test.DrillTest

org.apache.drill.test.SubOperatorTest

org.apache.drill.exec.physical.impl.scan.project.TestSchemaSmoothing

public class TestSchemaSmoothing
extends SubOperatorTest

Tests schema smoothing at the schema projection level. This level handles reusing prior types when filling null values. But, because no actual vectors are involved, it does not handle the schema chosen for a table ahead of time, only the schema as it is merged with prior schema to detect missing columns.

Focuses on the SmoothingProjection class itself.

Note that, at present, schema smoothing does not work for entire maps. That is, if file 1 has, say {a: {b: 10, c: "foo"}} and file 2 has, say, {a: null}, then schema smoothing does not currently know how to recreate the map. The same is true of lists and unions. Handling such cases is complex and is probably better handled via a system that allows the user to specify their intent by providing a schema to apply to the two files.

Note that schema smoothing itself is an experimental work-around to a fundamental limitation in Drill:

• Drill cannot predict the future: each file (or batch) may have a different schema.
• Drill does not know about these differences until they occur.
• The scan operator is obligated to return the same schema (and same vectors) from one file to the next, else a "hard" schema change is sent down stream.

The problem is actually intractable. The schema smoother handles the cases that can be handled, such as required --> nullable, a column disappearing, etc. This whole mechanism should be scrapped if/when Drill can work with schemas. Or, keep this to handle, as best we can, odd schemas, but insist on a schema to resolve issues that this mechanism cannot handle (and that, indeed, no algorithm could handle because such an algorithm would require time-travel: looking into the future to know what data will be scanned.)

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.DrillTest

DrillTest.MemWatcher

Field Summary

Fields inherited from class org.apache.drill.test.SubOperatorTest

dirTestWatcher, fixture

Fields inherited from class org.apache.drill.test.DrillTest

logOutcome, objectMapper, REPEAT_RULE, thrownException, TIMEOUT

Constructor Summary

Constructors

Constructor and Description
TestSchemaSmoothing()

Method Summary

Modifier and Type	Method and Description
void	testDifferentCase() The prior and table schemas are identical, but the cases of names differ.
void	testDifferentTypes() Column names match, but types differ.
void	testDiscrete() Sanity test for the simple, discrete case.
void	testDisjoint() Case in which the table schema and prior are disjoint sets.
void	testLongerPartitionLength() If using the legacy wildcard expansion, we are able to use the same schema even if the new partition path is longer than the previous.
void	testMissingNullableColumns() Preserve the prior schema if table is a subset and missing columns are nullable or repeated.
void	testReordering() Preserve the prior schema if table is a subset.
void	testRequired() Can't preserve the prior schema if it had required columns where the new schema has no columns.
void	testSamePartitionLength() If using the legacy wildcard expansion, reuse schema if partition paths are the same length.
void	testSameSchemas() The prior and table schemas are identical.
void	testShorterPartitionLength() If using the legacy wildcard expansion, reuse schema if the new partition path is shorter than the previous.
void	testSmaller() Case in which the table schema is a superset of the prior schema.
void	testSmoothableSchemaBatches() Integrated test across multiple schemas at the batch level.
void	testSmoothingProjection() Low-level test of the smoothing projection, including the exceptions it throws when things are not going its way.
void	testWildcardSmoothing() A SELECT * query uses the schema of the table as the output schema.

Methods inherited from class org.apache.drill.test.SubOperatorTest

classSetup, classTeardown

Methods inherited from class org.apache.drill.test.DrillTest

escapeJsonString, finishDrillTest, initDrillTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestSchemaSmoothing

public TestSchemaSmoothing()

Method Detail

testDiscrete

public void testDiscrete()

Sanity test for the simple, discrete case. The purpose of discrete is just to run the basic lifecycle in a way that is compatible with the schema-persistence version.

testSmoothingProjection

public void testSmoothingProjection()

Low-level test of the smoothing projection, including the exceptions it throws when things are not going its way.

testSmaller

public void testSmaller()

Case in which the table schema is a superset of the prior schema. Discard prior schema. Turn off auto expansion of metadata for a simpler test.

testDisjoint

public void testDisjoint()

Case in which the table schema and prior are disjoint sets. Discard the prior schema.

testDifferentTypes

public void testDifferentTypes()

Column names match, but types differ. Discard the prior schema.

testSameSchemas

public void testSameSchemas()

The prior and table schemas are identical. Preserve the prior schema (though, the output is no different than if we discarded the prior schema...)

testDifferentCase

public void testDifferentCase()

The prior and table schemas are identical, but the cases of names differ. Preserve the case of the first schema.

testRequired

public void testRequired()

Can't preserve the prior schema if it had required columns where the new schema has no columns.

testMissingNullableColumns

public void testMissingNullableColumns()

Preserve the prior schema if table is a subset and missing columns are nullable or repeated.

testReordering

public void testReordering()

Preserve the prior schema if table is a subset. Map the table columns to the output using the prior schema ordering.

testSamePartitionLength

public void testSamePartitionLength()

If using the legacy wildcard expansion, reuse schema if partition paths are the same length.

testShorterPartitionLength

public void testShorterPartitionLength()

If using the legacy wildcard expansion, reuse schema if the new partition path is shorter than the previous. (Unneeded partitions will be set to null by the scan projector.)

testLongerPartitionLength

public void testLongerPartitionLength()

If using the legacy wildcard expansion, we are able to use the same schema even if the new partition path is longer than the previous. Because all file names are provided up front.

testSmoothableSchemaBatches

public void testSmoothableSchemaBatches()

Integrated test across multiple schemas at the batch level.

testWildcardSmoothing

public void testWildcardSmoothing()

A SELECT * query uses the schema of the table as the output schema. This is trivial when the scanner has one table. But, if two or more tables occur, then things get interesting. The first table sets the schema. The second table then has:

• The same schema, trivial case.
• A subset of the first table. The type of the "missing" column from the first table is used for a null column in the second table.
• A superset or disjoint set of the first schema. This triggers a hard schema change.

It is an open question whether previous columns should be preserved on a hard reset. For now, the code implements, and this test verifies, that a hard reset clears the "memory" of prior schemas.