org.apache.drill.exec.store.easy.text.compliant

Class TestPartitionRace

java.lang.Object

org.apache.drill.test.BaseTest

org.apache.drill.test.DrillTest

org.apache.drill.test.ClusterTest

org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

org.apache.drill.exec.store.easy.text.compliant.TestPartitionRace

public class TestPartitionRace
extends BaseCsvTest

Demonstrates a race condition inherent in the way that partition columns are currently implemented. Two files: one at the root directory, one down one level. Parallelization is forced to two. (Most tests use small files and both files end up being read in the same scanner, which masks the problem shown here.)

Depending on which file is read first, the output row may start with or without the partition column. Once the column occurs, it will persist.

The solution is to figure out the max partition depth in the EasySubScan rather than in each scan operator, which is done in the current "V3" version. The tests here verify this behavior.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.DrillTest

DrillTest.MemWatcher

Field Summary

Fields inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

BIG_COL_SIZE, EMPTY_FILE, FILE_N_NAME, NESTED_DIR, NESTED_FILE, PART_DIR, ROOT_FILE, SCHEMA_BATCH_ENABLED, secondFile, testDir, validHeaders

Fields inherited from class org.apache.drill.test.ClusterTest

client, cluster, dirTestWatcher

Fields inherited from class org.apache.drill.test.DrillTest

logOutcome, objectMapper, REPEAT_RULE, thrownException, TIMEOUT

Constructor Summary

Constructors

Constructor and Description
TestPartitionRace()

Method Summary

Modifier and Type	Method and Description
static void	setup()
void	testNoRace() V3 computes partition depth in the group scan (which sees all files), and so the partition column count does not vary across scans.
void	testSingleScan() Oddly, when run in a single fragment, the files occur in a stable order, the partition always appears, and it appears in the first column position.

Methods inherited from class org.apache.drill.exec.store.easy.text.compliant.BaseCsvTest

buildBigColFile, buildFile, buildFile, buildNestedTable, buildTable, enableMultiScan, enableSchema, enableSchemaSupport, resetMultiScan, resetSchema, resetSchemaSupport, setup, setup

Methods inherited from class org.apache.drill.test.ClusterTest

getFile, queryBuilder, run, runAndLog, runAndPrint, runAndPrint, shutdown, startCluster, testBuilder

Methods inherited from class org.apache.drill.test.DrillTest

escapeJsonString, finishDrillTest, initDrillTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestPartitionRace

public TestPartitionRace()

Method Detail

setup

public static void setup()
                  throws Exception

Throws:

Exception

testSingleScan

public void testSingleScan()
                    throws IOException

Oddly, when run in a single fragment, the files occur in a stable order, the partition always appears, and it appears in the first column position.

Throws:

IOException

testNoRace

public void testNoRace()
                throws IOException

V3 computes partition depth in the group scan (which sees all files), and so the partition column count does not vary across scans. Also, V3 puts partition columns at the end of the row so that data columns don't "jump around" when files are shifted to a new partition depth.

Throws:

IOException