WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content

Conversation

@kabo87777
Copy link

Fix flaky tests in BatchedAlignedSeriesReadChunkCompactionTest

Description

This PR fixes 12 flaky tests in BatchedAlignedSeriesReadChunkCompactionTest that were failing intermittently due to non-deterministic iteration order of collections.

Root Cause

The getPaths() method in AbstractCompactionTest.java had two sources of non-determinism:

  1. HashSet usage: The method used HashSet<IFullPath> which does not guarantee iteration order
  2. Unsorted schemaMap.values(): The measurementSchemas list was created directly from schemaMap.values() without sorting, and since schemaMap is backed by a ConcurrentHashMap, the iteration order is non-deterministic

This caused AlignedFullPath objects to be created with measurements in different orders across test runs, leading to comparison failures when validating compaction results.

Fix

  1. Changed HashSet to LinkedHashSet to preserve insertion order
  2. Sorted measurementSchemas by measurement name before creating the list

Changes

File: iotdb-core/datanode/src/test/java/org/apache/iotdb/db/storageengine/dataregion/compaction/AbstractCompactionTest.java

// Before (non-deterministic):
Set<IFullPath> paths = new HashSet<>();
// ...
List<IMeasurementSchema> measurementSchemas = new ArrayList<>(schemaMap.values());

// After (deterministic):
Set<IFullPath> paths = new LinkedHashSet<>();
// ...
List<IMeasurementSchema> measurementSchemas =
    schemaMap.values().stream()
        .sorted(Comparator.comparing(IMeasurementSchema::getMeasurementName))
        .collect(Collectors.toList());

Tests Fixed

  • BatchedAlignedSeriesReadChunkCompactionTest#testCompactionWithDifferentCompressionTypeOrEncoding
  • BatchedAlignedSeriesReadChunkCompactionTest#testCompactionWithEmptyBatch
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByFlushPage
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionByWritePoint
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithAllDeletedColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithAllDeletedPageByFlushPage
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNotExistColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNullColumn
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithNullColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithPartialDeletedColumnByFlushChunk
  • BatchedAlignedSeriesReadChunkCompactionTest#testSimpleCompactionWithPartialDeletedPageByWritePoint

Verification

Verified using NonDex with 3 different random seeds - all tests pass consistently.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant