org.apache.drill.exec.physical.impl.writer

Class TestCorruptParquetDateCorrection

java.lang.Object

org.apache.drill.test.BaseTest

org.apache.drill.test.DrillTest

org.apache.drill.exec.ExecTest

org.apache.drill.test.BaseTestQuery

org.apache.drill.PlanTestBase

org.apache.drill.exec.physical.impl.writer.TestCorruptParquetDateCorrection

public class TestCorruptParquetDateCorrection
extends PlanTestBase

Tests for compatibility reading old parquet files after date corruption issue was fixed in DRILL-4203. Drill could write non-standard dates into parquet files. This issue is related to all drill releases where ParquetRecordWriter.WRITER_VERSION_PROPERTY < ParquetReaderUtility.DRILL_WRITER_VERSION_STD_DATE_FORMAT The values have been read correctly by Drill, but external tools like Spark reading the files will see corrupted values for all dates that have been written by Drill. This change corrects the behavior of the Drill parquet writer to correctly store dates in the format given in the parquet specification. To maintain compatibility with old files, the parquet reader code has been updated to check for the old format and automatically shift the corrupted values into corrected ones automatically. The test cases included here should ensure that all files produced by historical versions of Drill will continue to return the same values they had in previous releases. For compatibility with external tools, any old files with corrupted dates can be re-written using the CREATE TABLE AS command (as the writer will now only produce the specification-compliant values, even if after reading out of older corrupt files). While the old behavior was a consistent shift into an unlikely range to be used in a modern database (over 10,000 years in the future), these are still valid date values. In the case where these may have been written into files intentionally, an option is included to turn off the auto-correction. Use of this option is assumed to be extremely unlikely, but it is included for completeness.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.drill.test.BaseTestQuery

BaseTestQuery.ClassicTestServices, BaseTestQuery.SilentListener, BaseTestQuery.TestResultSet

Nested classes/interfaces inherited from class org.apache.drill.test.DrillTest

DrillTest.MemWatcher

Field Summary

Fields inherited from class org.apache.drill.PlanTestBase

EXPECTED_NOT_FOUND, JSON_FORMAT, OPTIQ_FORMAT, UNEXPECTED_FOUND

Fields inherited from class org.apache.drill.test.BaseTestQuery

allocator, bits, client, config, serviceSet

Fields inherited from class org.apache.drill.exec.ExecTest

c, dirTestWatcher, optionManager

Fields inherited from class org.apache.drill.test.DrillTest

logOutcome, objectMapper, REPEAT_RULE, thrownException, TIMEOUT

Constructor Summary

Constructors

Constructor and Description
TestCorruptParquetDateCorrection()

Method Summary

Modifier and Type	Method and Description
static void	initFs()
void	testCorrectDatesAndExceptionWhileParsingCreatedBy()
void	testCorrectDateValuesGeneratedByOldVersionOfDrill()
void	testCorruptValueDetectionDuringPruning()
void	testDatePartitionedReadWithCorruption()
void	testQueryWithCorruptedDates()
void	testReadCorruptDatesWithNullFilledColumns() To fix some of the corrupted dates fixed as part of DRILL-4203 it requires actually looking at the values stored in the file.
void	testReadMixedOldAndNewBothReaders() Test reading a directory full of parquet files with dates, some of which have corrupted values due to DRILL-4203.
void	testReadNewMetadataCacheFileOverOldAndNewFiles()
void	testReadOldMetadataCacheFile()
void	testReadOldMetadataCacheFileWithPruning()
void	testReadPartitionedOnCorrectDates() Test reading a directory full of partitioned parquet files with dates, these files have a drill version number of "1.9.0-SNAPSHOT" and parquet-writer version number of "2" in their footers, so we can be certain they do not have corruption.
void	testUserOverrideDateCorrection()
void	testVarcharPartitionedReadWithCorruption()

Methods inherited from class org.apache.drill.PlanTestBase

checkForMetadataFile, createMetadataDir, getPhysicalJsonPlan, getPlanInString, testPhysicalPlan, testPhysicalPlanExecutionBasedOnQuery, testPlanMatchingPatterns, testPlanMatchingPatterns, testPlanMatchingPatterns, testPlanMatchingPatterns, testPlanOneExcludedPattern, testPlanOneExpectedPattern, testPlanOneExpectedPatternOneExcluded, testPlanSubstrPatterns, testPlanWithAttributesMatchingPatterns, testRelLogicalJoinOrder, testRelLogicalPlanLevDigest, testRelLogicalPlanLevExplain, testRelPhysicalJoinOrder, testRelPhysicalPlanLevDigest, testRelPhysicalPlanLevExplain

Methods inherited from class org.apache.drill.test.BaseTestQuery

alterSession, cloneDefaultTestConfigProperties, closeClient, errorMsgTestHelper, getAllocator, getDrillbitContext, getFile, getPhysicalFileFromResource, getResultString, getUserPort, logResult, newTest, parseErrorHelper, printResult, resetAllSessionOptions, resetDrillbitCount, resetSessionOption, runSQL, setColumnWidth, setColumnWidths, setSessionOption, setSessionOption, setSessionOption, setSessionOption, setupDefaultTestCluster, test, test, testBuilder, testLogicalWithResults, testNoResult, testNoResult, testPhysical, testPhysicalFromFile, testPhysicalWithResults, testPreparedStatement, testRunAndPrint, testRunAndReturn, testSql, testSqlWithResults, testWithListener, updateClient, updateClient, updateClient, updateTestCluster, updateTestCluster

Methods inherited from class org.apache.drill.exec.ExecTest

clear, getLocalFileSystem, mockDrillbitContext, mockUsDateFormatSymbols, mockUtcDateTimeZone, parseExpr, setupOptionManager

Methods inherited from class org.apache.drill.test.DrillTest

escapeJsonString, finishDrillTest, initDrillTest

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestCorruptParquetDateCorrection

public TestCorruptParquetDateCorrection()

Method Detail

initFs

public static void initFs()
                   throws Exception

Throws:

Exception

testReadPartitionedOnCorrectDates

public void testReadPartitionedOnCorrectDates()
                                       throws Exception

Test reading a directory full of partitioned parquet files with dates, these files have a drill version number of "1.9.0-SNAPSHOT" and parquet-writer version number of "2" in their footers, so we can be certain they do not have corruption. The option to disable the correction is passed, but it will not change the result in the case where we are certain correction is NOT needed. For more info see DRILL-4203.

Throws:

Exception

testVarcharPartitionedReadWithCorruption

public void testVarcharPartitionedReadWithCorruption()
                                              throws Exception

Throws:

Exception

testDatePartitionedReadWithCorruption

public void testDatePartitionedReadWithCorruption()
                                           throws Exception

Throws:

Exception

testCorrectDatesAndExceptionWhileParsingCreatedBy

public void testCorrectDatesAndExceptionWhileParsingCreatedBy()
                                                       throws Exception

Throws:

Exception

testQueryWithCorruptedDates

public void testQueryWithCorruptedDates()
                                 throws Exception

Throws:

Exception

testCorruptValueDetectionDuringPruning

public void testCorruptValueDetectionDuringPruning()
                                            throws Exception

Throws:

Exception

testReadCorruptDatesWithNullFilledColumns

public void testReadCorruptDatesWithNullFilledColumns()
                                               throws Exception

To fix some of the corrupted dates fixed as part of DRILL-4203 it requires actually looking at the values stored in the file. A column with date values actually stored must be located to check a value. Just because we find one column where the all values are null does not mean we can safely avoid reading date columns with auto-correction, although null values do not need fixing, other columns may contain actual corrupt date values. This test checks the case where the first columns in the file are all null filled and a later column must be found to identify that the file is corrupt.

Throws:

Exception

testUserOverrideDateCorrection

public void testUserOverrideDateCorrection()
                                    throws Exception

Throws:

Exception

testReadMixedOldAndNewBothReaders

public void testReadMixedOldAndNewBothReaders()
                                       throws Exception

Test reading a directory full of parquet files with dates, some of which have corrupted values due to DRILL-4203. Tests reading the files with both the vectorized and complex parquet readers.

Throws:

Exception

testReadOldMetadataCacheFile

public void testReadOldMetadataCacheFile()
                                  throws Exception

Throws:

Exception

testReadOldMetadataCacheFileWithPruning

public void testReadOldMetadataCacheFileWithPruning()
                                             throws Exception

Throws:

Exception

testReadNewMetadataCacheFileOverOldAndNewFiles

public void testReadNewMetadataCacheFileOverOldAndNewFiles()
                                                    throws Exception

Throws:

Exception

testCorrectDateValuesGeneratedByOldVersionOfDrill

public void testCorrectDateValuesGeneratedByOldVersionOfDrill()
                                                       throws Exception

Throws:

Exception