Skip to content

fix: reject malformed Arrow field IDs #845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
fallintoplace wants to merge 1 commit into
base: main
Choose a base branch
from
+29 −5
Open

fix: reject malformed Arrow field IDs #845

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions src/iceberg/schema_internal.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ Status ToArrowSchema(const Schema& schema, ArrowSchema* out) {

namespace {

int32_t GetFieldId(const ArrowSchema& schema) {
Result<int32_t> GetFieldId(const ArrowSchema& schema) {
if (schema.metadata == nullptr) {
return kUnknownFieldId;

@evindj evindj Jul 25, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be changed? It is not clear to me what is the difference between unknowFieldId (returning the constant) and InvalidFieldId (returning in a error)

@fallintoplace fallintoplace Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are intentionally different. If the PARQUET:field_id metadata is missing, we keep the existing behavior and use the unknown field ID sentinel. If the key is present but the value is malformed or outside int32_t, we return an invalid schema error instead.

}
Expand All @@ -240,9 +240,14 @@ int32_t GetFieldId(const ArrowSchema& schema) {
return kUnknownFieldId;
}

int32_t field_id = kUnknownFieldId;
std::from_chars(field_id_value.data, field_id_value.data + field_id_value.size_bytes,
field_id);
int32_t field_id = 0;
const auto* end = field_id_value.data + field_id_value.size_bytes;
const auto [ptr, ec] = std::from_chars(field_id_value.data, end, field_id);
if (ec != std::errc{} || ptr != end) {
return InvalidSchema(
"Invalid Arrow field ID: '{}'",
std::string_view(field_id_value.data, field_id_value.size_bytes));
}

return field_id;
}
Expand All @@ -252,7 +257,7 @@ Result<std::shared_ptr<Type>> FromArrowSchema(const ArrowSchema& schema) {
[](const ArrowSchema& schema) -> Result<std::unique_ptr<SchemaField>> {
ICEBERG_ASSIGN_OR_RAISE(auto field_type, FromArrowSchema(schema));

auto field_id = GetFieldId(schema);
ICEBERG_ASSIGN_OR_RAISE(auto field_id, GetFieldId(schema));
bool is_optional = (schema.flags & ARROW_FLAG_NULLABLE) != 0;
if (field_type->type_id() == TypeId::kUnknown && !is_optional) {
return InvalidSchema("Arrow null field '{}' must be nullable", schema.name);
Expand Down
19 changes: 19 additions & 0 deletions src/iceberg/test/arrow_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -371,6 +371,25 @@ TEST_P(FromArrowSchemaTest, PrimitiveType) {
ASSERT_EQ(*field.type(), *param.iceberg_type);
}

TEST(FromArrowSchemaTest, RejectMalformedFieldIdMetadata) {
for (const auto& field_id : {"1x", "2147483648", ""}) {
auto metadata =
::arrow::key_value_metadata(std::unordered_map<std::string, std::string>{
{std::string(kParquetFieldIdKey), field_id}});
auto arrow_schema = ::arrow::schema({::arrow::field(
"foo", ::arrow::int32(), /*nullable=*/true, std::move(metadata))});
ArrowSchema exported_schema;
ASSERT_TRUE(::arrow::ExportSchema(*arrow_schema, &exported_schema).ok());

auto result = FromArrowSchema(exported_schema, /*schema_id=*/1);
ArrowSchemaRelease(&exported_schema);

EXPECT_THAT(result, IsError(ErrorKind::kInvalidSchema));
EXPECT_THAT(result,
HasErrorMessage(std::format("Invalid Arrow field ID: '{}'", field_id)));
}
}

INSTANTIATE_TEST_SUITE_P(
SchemaConversion, FromArrowSchemaTest,
::testing::Values(
Expand Down
Loading