-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(tracing): strip inline media from messages #18413
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
feat(tracing): strip inline media from messages #18413
Conversation
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. Re JS-1002 Re GH-17810
7c5aad2 to
3512099
Compare
size-limit report 📦
|
node-overhead report 🧳Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.
|
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. Re JS-1002 Re GH-17810
3512099 to
5d7bbc8
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. If any truncation occurs, then a `client.recordDroppedEvent()` message is fired, with the reason `before_send`, category `attachment`, and the count of messages affected. Re JS-1002 Re GH-17810
5d7bbc8 to
3e58d4b
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. If any truncation occurs, then a `client.recordDroppedEvent()` message is fired, with the reason `before_send`, category `attachment`, and the count of messages affected. Re JS-1002 Re GH-17810
3e58d4b to
687dd68
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice work. I left a comment can you please resolve it before merging? and i recommend adding an integration test for this after fixing lint issues.
| return []; | ||
| } | ||
|
|
||
| const REMOVED_STRING = '<removed>'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@RulaKhaled how do we usually mark redacted data in the JS SDK? I couldn't find other uses of this anywhere, just want to make sure we're consistent in how we mark this type of thing - especially across SDKs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Had a quick check with @sentrivana - there is no such string in python usually, but they do add a _meta entry when removing entire values (see here). Sensitive data is marked using [Filtered].
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. If any truncation occurs, then a `client.recordDroppedEvent()` message is fired, with the reason `before_send`, category `attachment`, and the count of messages affected. Re JS-1002 Re GH-17810
687dd68 to
0b22140
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. If any truncation occurs, then a `client.recordDroppedEvent()` message is fired, with the reason `before_send`, category `attachment`, and the count of messages affected. Re JS-1002 Re GH-17810
0b22140 to
6692634
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. If any truncation occurs, then a `client.recordDroppedEvent()` message is fired, with the reason `before_send`, category `attachment`, and the count of messages affected. Re JS-1002 Re GH-17810
6692634 to
f3ccd12
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. Re JS-1002 Re GH-17810
f3ccd12 to
ba36a66
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. Re JS-1002 Re GH-17810
ba36a66 to
816ad3e
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. Re JS-1002 Re GH-17810
816ad3e to
22a1a57
Compare
This is the functional portion addressing JS-1002. Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data. If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large. Re JS-1002 Re GH-17810
22a1a57 to
1095669
Compare
| } | ||
| if (isContentMedia(message)) { | ||
| newMessage = stripInlineMediaFromSingleMessage(message); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bug: Media message overwrite discards prior content stripping
When a message matches both isContentArrayMessage (or isPartsMessage) AND isContentMedia at the top level, the stripInlineMediaFromSingleMessage call at line 363 uses the original message instead of the already-processed newMessage. This overwrites prior content/parts stripping. Additionally, since MEDIA_FIELDS includes 'content', the original content array would be replaced with '[Filtered]' string, losing all message content structure. This occurs for messages that have inline media fields (like inlineData, b64_json, or type: 'base64') at the top level alongside a content or parts array.
This is the functional portion addressing JS-1002.
Prior to truncating text messages for their byte length, any inline base64-encoded media properties are filtered out. This allows the message to possibly be included in the span, indicating to the user that a media object was present, without overflowing the allotted buffer for sending data.
If a media message is not removed, the fallback is still to simply remove it if its overhead grows too large.
Re JS-1002
Re GH-17810
Message truncation (for text length and inline media) still needs to be added to the docs.