feat: support read for nested type sub column

[lucasfang]

Title: feat: support nested struct sub-column recall and map selected-keys recall

Purpose

Linked issue: close #xxx

This PR introduces end-to-end support for read-side projection improvements focused on two capabilities:

1. Nested struct sub-column recall

• Support reading partial nested fields from STRUCT/LIST/MAP hierarchies.
• Add recursive nested projection utilities at framework level, including schema-level detection for nested sub-field projection.
• Keep format-specific behavior in format layer:
  • Avro: fail fast for unsupported nested sub-field projection with explicit error.
  • Lance: add the same fail-fast protection for unsupported nested sub-field projection.

2. Map selected-keys recall

• Support MAP_SELECTED_KEYS metadata driven key filtering during read.
• Define and implement empty-string semantic for MAP_SELECTED_KEYS:
  • metadata key present with empty value means filter all keys.
  • empty tokens in comma-separated values are ignored.
• Add utility and integration coverage for selected-keys behavior.

Additional alignment/fixes included in this commit range:

• Refactor and consolidate nested projection detection logic into NestedProjectionUtils and simplify format readers.
• Parquet read path fix for out-of-order projected schema (align output to read schema order).
• Field mapping relaxations to avoid false mismatches for valid nested/projection scenarios.
• Extend integration and unit tests around nested pruning and map selected-keys.

Tests

Unit tests:

• NestedProjectionUtilsTest.HasNestedSubfieldProjection_NoProjection
• NestedProjectionUtilsTest.HasNestedSubfieldProjection_WithProjection
• NestedProjectionUtils map selected-keys related tests, including:
  • GetMapSelectedKeys_EmptyString
  • GetMapSelectedKeys_ContainsEmptyToken
  • FilterMapArrayBySelectedKeys_EmptyKeyMeansFilterAll

Format tests:

• AvroFileBatchReaderTest.TestSetReadSchemaRejectNestedSubFieldProjection
• Compression/AvroFileFormatTest.TestNestedMap/*

Integration tests:

• FileFormats/NestedColumnPruningInteTest.MapSelectedKeysEmptyStringMeansFilterAll/*
• Nested column pruning scenarios for parquet/orc matrix (struct/deep nested/special fields/map selected-keys)

Note:

• Lance nested sub-field projection fail-fast test was added in source and should be executed in an environment built with PAIMON_ENABLE_LANCE=ON.

API and Format

• API impact:

  • Read projection behavior is enhanced for nested recall and map selected-keys filtering.
  • No incompatible public API removal.

• Storage/protocol impact:

  • No on-disk format or protocol change.
  • Behavior change is in reader projection and validation paths.

Documentation

• No new external feature switch introduced.
• Existing behavior is clarified by tests for nested projection and map selected-keys semantics.

Generative AI tooling

Generated-by: GitHub Copilot (GPT-5.3-Codex)

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 5 comments.