Skip to content

fix: preserve partition types after source columns are dropped - #999

Open
manuzhang wants to merge 1 commit into
apache:mainfrom
manuzhang:codex/fix-partition-types-dropped-source
Open

manuzhang wants to merge 1 commit into
apache:mainfrom
manuzhang:codex/fix-partition-types-dropped-source

Conversation

@manuzhang

Copy link
Copy Markdown
Member

When a source column has been dropped from the current schema, PartitionSpec::PartitionType currently assigns unknown to all partition fields referencing it. This loses the known result types of fixed-result transforms and prevents older manifest bounds from being decoded correctly.

Always ask the transform for its result type, passing unknown when the source is absent. Bucket and year fields retain int32, and day fields retain date; identity fields remain unknown. Extend PartitionTypeMissingSource to cover dropped bucket, day, year and identity sources.

Extracted from #876 as an independent prerequisite for its metadata-table work.

Validation: pre-commit and git diff --check passed; the standalone schema_test suite passed all 561 tests. The expanded PartitionTypeMissingSource regression fails against the previous implementation for dropped bucket/day/year fields and passes with the fix.

Copilot AI balanced review requested due to automatic review settings October 10, 2026 16:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The implementation matches transform result-type contracts and has focused regression coverage.

0 open findings

What changed in this PR

Preserves fixed-result partition types when source columns have been dropped, enabling historical partition data to be decoded correctly.

Changes:

  • Derives partition result types using unknown() for missing sources.
  • Adds regression coverage for dropped bucket, day, year, and identity sources.
File Description
src/​iceberg/​partition_spec.cc Preserves transform-derived partition types for missing sources.
src/​iceberg/​test/​partition_spec_test.cc Expands missing-source partition type tests.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@manuzhang
manuzhang requested a review from wgtmac October 10, 2026 16:53

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants