Skip to content

HDDS-15000. Improve S3 audit log stack traces#10097

Open
adoroszlai wants to merge 8 commits intoapache:masterfrom
adoroszlai:HDDS-15000
Open

HDDS-15000. Improve S3 audit log stack traces#10097
adoroszlai wants to merge 8 commits intoapache:masterfrom
adoroszlai:HDDS-15000

Conversation

@adoroszlai
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Audit log shows only the last two elements of stack traces. For several failed S3 Gateway requests it is always the same:

... | ERROR | S3GAudit | ... | op=GET_BUCKET ... | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:192)
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:179)

due to two levels of newError calls.

This PR changes code to avod chained newError calls by creating OS3Exception in various newError implementations. Related changes to avoid duplication:

  • Simplify OS3Exception creation:
    • pass error code (S3ErrorTable) instead of separate variables
    • also pass cause exception and resource
    • remove unused constructors
    • log from constructor instead of newError
  • Create few more variants of newError to reduce null arguments in callers.
  • Move BucketEndpoint#translateException to S3ErrorTable (renamed to translateResultCode), add more cases to switch. Change ObjectEndpoint to rely on result code translation in newError.

Related minor fix:

  • Deleting non-existent keys returns success response (204), not an error. However, the audit log shows ERROR instead of SUCCESS, because NOT_FOUND error is handled in the outer delete method, not handleDeleteRequest. This is now moved to let audit log reflect the correct result.

https://issues.apache.org/jira/browse/HDDS-15000

How was this patch tested?

Before:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=GET_BUCKET {"bucket":"[bucket-etqvtcowkf]","list-type":"[2]","max-keys":"[-1]","encoding-type":"[url]","x-amz-request-id":"1f772aef-ed39-41a6-b470-5fff0962422e","x-amz-id-2":"bn6THRxxaQso"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:192)
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:179)

After:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=GET_BUCKET {"bucket":"[bucket-mpqrnkuavm]","list-type":"[2]","max-keys":"[-1]","encoding-type":"[url]","x-amz-request-id":"28a04087-212e-4e82-bc3b-60976b690f86","x-amz-id-2":"0mVf1nwDd"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception: Invalid Argument
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:261)
	at org.apache.hadoop.ozone.s3.endpoint.BucketEndpoint.validateMaxKeys(BucketEndpoint.java:257)

Before:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=INIT_MULTIPART_UPLOAD {"bucket":"[bucket-etqvtcowkf]","path":"[ozone-test-6052266241/mpuWithLargeMetadata]","x-amz-request-id":"0e1c28c9-3451-498b-933e-2b668991216f","x-amz-id-2":"5kEMzckEG19","uploads":"[]"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:192)
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:179)

After:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=INIT_MULTIPART_UPLOAD {"bucket":"[bucket-mpqrnkuavm]","path":"[ozone-test-5385731147/mpuWithLargeMetadata]","x-amz-request-id":"41329a9a-7ab3-4fcb-8f41-06e130631a2e","x-amz-id-2":"S6IY69aIGOpGP","uploads":"[]"} | ret=FAILURE | org.apache.hadoop.ozone.s3.exception.OS3Exception: Illegal user defined metadata. Combined size exceeds the maximum allowed metadata size of 2KB
	at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:261)
	at org.apache.hadoop.ozone.s3.endpoint.EndpointBase.getCustomMetadataFromHeaders(EndpointBase.java:315)

Before:

| ERROR | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=DELETE_KEY {"bucket":"[bucket-etqvtcowkf]","path":"[thereisnosuchfile]",...} | ret=FAILURE | KEY_NOT_FOUND org.apache.hadoop.ozone.om.exceptions.OMException: Key not found
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.handleError(OzoneManagerProtocolClientSideTranslatorPB.java:766)
	at org.apache.hadoop.ozone.om.protocolPB.OzoneManagerProtocolClientSideTranslatorPB.deleteKey(OzoneManagerProtocolClientSideTranslatorPB.java:965)

After:

| INFO  | S3GAudit | user=dlfknslnfslf | ip=172.18.0.6 | op=DELETE_KEY {"bucket":"[bucket-mpqrnkuavm]","path":"[thereisnosuchfile]",...} | ret=SUCCESS |  

https://github.com/adoroszlai/ozone/actions/runs/24665673476

@adoroszlai adoroszlai self-assigned this Apr 20, 2026
@adoroszlai adoroszlai added the s3 S3 Gateway label Apr 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

s3 S3 Gateway

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant