Surface corrupt Iceberg metadata as 422 instead of masking as table-not-found by dushyantk1509 · Pull Request #541 · linkedin/openhouse

dushyantk1509 · 2026-04-12T06:54:44Z

Summary

A table had corrupt metadata — current-schema-id=6 but schema 6 was missing from the schemas array. Both Spark and Trino reported "Table does not exist", making the issue impossible to diagnose without digging through tables-service logs.

The reason behind Spark and Trino reporting "Table does not exist" is -

Iceberg's TableMetadataParser.fromJson() throws IllegalArgumentException when parsing corrupt metadata.
OpenHouseExceptionHandler maps IllegalArgumentException → HTTP 400 (Bad Request).
Catalog client (OpenHouseTableOperations.doRefresh()) silently swallows 400 responses via .onErrorResume(BadRequest.class, e -> Mono.empty()), converting them to NoSuchTableException → "Table does not exist".

There are couple of issues in this code path

Mapping of IllegalArgumentException → HTTP 400 (Bad Request) is done assuming that user side validation will be mapped to IllegalArgumentException. It's done correctly in certain places like like OpenHouseInternalTableOperations.rebuildSortOrder but seems messed up in certain places like StorageManager where it indicates server side issue. This is something can be corrected in follow up PR(s).
Catalog client silently swallows 400 responses doesn't seem correct as bad request doesn't mean table doesn't exist. Such issues should be surfaced to users. This is something can be corrected in follow up PR(s).

Keeping the scope only to surfacing iceberg metadata corruption issue, this PR adds InvalidTableMetadataException (extends UnprocessableEntityException → HTTP 422) and catch metadata corruption exceptions in OpenHouseInternalTableOperations.refreshMetadata(). The catch covers the full range of Iceberg metadata failures from refreshFromMetadataLocation():

IllegalArgumentException: corrupt content — malformed JSON fields, missing schema IDs, invalid partition specs, sort orders, snapshots (~60+ throw sites across TableMetadataParser, SchemaParser, PartitionSpecParser, SortOrderParser, SnapshotParser, JsonUtil, and TableMetadata constructor)
IllegalStateException: UUID mismatch — refreshed metadata has a different table UUID than expected.
NotFoundException: metadata file missing from storage — HouseTable record exists but the metadata file it points to has been deleted (dangling pointer)
ValidationException: snapshot/partition spec inconsistencies — e.g. snapshot sequence number exceeds last sequence number.

RuntimeIOException and UncheckedIOException are intentionally not caught as they also represent transient infrastructure failures (HDFS down, network timeout), not necessarily metadata corruption. So, 500 server error is fine.

Changes

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

Manually Tested on local docker setup. Please include commands ran, and their output.
Added new tests for the changes made.
Updated existing tests to reflect the changes made.
No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
Some other form of testing like staging or soak time in production. Please explain.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

Breaking Changes
Deprecations
Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

…ot-found IllegalArgumentException thrown during metadata parsing (e.g. missing schema ID) was mapped to HTTP 400 by the global exception handler, which the catalog client silently swallowed, causing Spark/Trino to report "Table does not exist" instead of the real error. Fix: catch metadata corruption exceptions in refreshMetadata() and wrap them in InvalidTableMetadataException (extends UnprocessableEntityException → 422), so the actual cause surfaces to users. Caught exceptions cover the full range of Iceberg metadata failures: - IllegalArgumentException: corrupt content (malformed JSON, missing schema IDs) - IllegalStateException: UUID mismatch between refreshes - NotFoundException: metadata file missing from storage (dangling pointer) - ValidationException: snapshot/partition spec inconsistencies I/O exceptions (RuntimeIOException, UncheckedIOException) are intentionally NOT caught here as they represent transient infrastructure failures, not metadata corruption. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

sumedhsakdeo · 2026-04-13T04:20:14Z

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java

+          metadataLoc,
+          System.currentTimeMillis() - startTime,
+          e);
+      throw new InvalidTableMetadataException(


It seems not intuitive to have user specified invalid or illegal argument throw an invalid table metadata exception

This is not an user specified invalid or illegal argument. Here, iceberg is throwing IllegalArgumentException because of corrupted metadata. More specifically, the example I've see is - json file is referring to current-schema-id=6 which doesn't exist.

sumedhsakdeo · 2026-04-13T04:21:38Z

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java

          metadataLoc,
          System.currentTimeMillis() - startTime);
+    } catch (IllegalArgumentException
+        | IllegalStateException


What is the iceberg client side behavior you expect? Cleanup the written commit or leave it there like how it is handled in CUSE?

I just looked at the read path while making this change but in the write path - commit step, refresh is done before commit. So, if refreshMetadata() fails, it fails before any new metadata is written — nothing to clean up.

maluchari

422 implies request was not processible. I am inclined towards 500 for this usecase. 500 does not promise transience - IMO, it means server failed to fulfill a valid request. It could cause retries but I would also say that implies retry for 500 errors should be revisited. This is a server side issue and hence we should not categorize it as a client class error.

dushyantk1509 · 2026-04-14T05:26:43Z

422 implies request was not processible. I am inclined towards 500 for this usecase. 500 does not promise transience - IMO, it means server failed to fulfill a valid request. It could cause retries but I would also say that implies retry for 500 errors should be revisited. This is a server side issue and hence we should not categorize it as a client class error.

yes, I agree it's not a client issue. We can go with 500 with clear error message as it falls under a generic server side issue.

abhisheknath2011 · 2026-04-14T05:36:22Z

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java

          System.currentTimeMillis() - startTime);
+    } catch (IllegalArgumentException
+        | IllegalStateException
+        | NotFoundException


We could separate out NotFoundException and ValidationException to clearly categorize them as 4xx error.

In this context, both NotFoundException and ValidationException are server-side data integrity issues, not client errors:

NotFoundException: the HouseTable record exists but the metadata file it points to is missing from HDFS — a dangling pointer.

ValidationException: snapshot sequence number inconsistency in stored metadata.

abhisheknath2011 · 2026-04-14T05:37:51Z

.../src/main/java/com/linkedin/openhouse/internal/catalog/OpenHouseInternalTableOperations.java

+          metadataLoc,
+          System.currentTimeMillis() - startTime,
+          e);
+      throw new InvalidTableMetadataException(


This may not be right exception as we are considering handling 4 types of exception within the catch block.

Seems related to above comment.

Per review feedback: 422 is a 4xx client error class, but corrupt metadata is a server-side issue. 500 correctly signals that the server failed to fulfill a valid request. - InvalidTableMetadataException now extends RuntimeException (not UnprocessableEntityException) - Added dedicated handler in OpenHouseExceptionHandler mapping to 500 - Added controller test verifying the 500 status code mapping Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dushyantk1509 marked this pull request as ready for review April 13, 2026 02:18

sumedhsakdeo reviewed Apr 13, 2026

View reviewed changes

maluchari reviewed Apr 13, 2026

View reviewed changes

abhisheknath2011 reviewed Apr 14, 2026

View reviewed changes

dushyantk1509 force-pushed the dushyantk1509/fix-corrupt-metadata-error-surfacing branch from 6928731 to 7fa2e59 Compare April 14, 2026 05:57

dushyantk1509 force-pushed the dushyantk1509/fix-corrupt-metadata-error-surfacing branch from 7fa2e59 to 80e7c6d Compare April 14, 2026 06:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Surface corrupt Iceberg metadata as 422 instead of masking as table-not-found#541

Surface corrupt Iceberg metadata as 422 instead of masking as table-not-found#541
dushyantk1509 wants to merge 2 commits intolinkedin:mainfrom
dushyantk1509:dushyantk1509/fix-corrupt-metadata-error-surfacing

dushyantk1509 commented Apr 12, 2026 •

edited

Loading

Uh oh!

sumedhsakdeo Apr 13, 2026

Uh oh!

dushyantk1509 Apr 13, 2026

Uh oh!

sumedhsakdeo Apr 13, 2026

Uh oh!

dushyantk1509 Apr 13, 2026

Uh oh!

maluchari left a comment

Uh oh!

dushyantk1509 commented Apr 14, 2026

Uh oh!

abhisheknath2011 Apr 14, 2026

Uh oh!

dushyantk1509 Apr 14, 2026

Uh oh!

abhisheknath2011 Apr 14, 2026

Uh oh!

dushyantk1509 Apr 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dushyantk1509 commented Apr 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing Done

Additional Information

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maluchari left a comment

Choose a reason for hiding this comment

Uh oh!

dushyantk1509 commented Apr 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dushyantk1509 commented Apr 12, 2026 •

edited

Loading