Botocore instrumentation incorrectly marks S3 304 Not Modified as span error

[apinkney97]

Describe your environment

OS: Ubuntu
Python version: Python 3.11.10
Package version: opentelemetry-instrumentation-botocore==0.62b1

What happened?

When using conditional S3 requests (eg using IfNoneMatch or IfModifiedSince parameters), a 304 Not Modified response is returned by botocore as a ClientError exception. The botocore instrumentor treats all ClientError exceptions as errors and sets StatusCode.ERROR on the span, even though a 304 is a normal expected
response and not a true error.

This causes S3.GetObject spans (and potentially other operations that return 3xx responses) to be spuriously marked as errors in tracing backends, polluting error rates and alerting.

Steps to Reproduce

import boto3

s3 = boto3.client("s3")
etag = s3.head_object(Bucket="my-bucket", Key="my-key")["ETag"]

# This raises ClientError with a 304 status - the span is incorrectly marked as error.
s3.get_object(Bucket="my-bucket", Key="my-key", IfNoneMatch=etag)

Expected Result

Spans for responses with HTTP 3xx status codes should be marked OK, not ERROR.

Actual Result

Any ClientError, regardless of HTTP status code, results in StatusCode.ERROR on the span.

Additional context

No response

Would you like to implement a fix?

Yes

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[apinkney97]

FYI I have a potential fix; will try to submit a PR tomorrow some time.

[apinkney97]

This will be delayed slightly while I wade through some corporate red tape to get this signed off properly by my employer. Another PR will follow from my work GitHub account. (This may take a while while I get signoff...)