bug: self-hosted webapp cannot obtain IRSA credentials — fast-xml-parser 5.7.x rejects @aws-sdk/xml-builder's addEntity("#xD")

[vlakshmanan-silkline]

Summary

On a self-hosted deployment on EKS, annotating the trigger-webapp service account for IRSA makes every AWS call from the webapp fail during credential resolution. The pinned fast-xml-parser@5.7.1 throws while the AWS SDK configures its XML parser, so the STS AssumeRoleWithWebIdentity response can never be deserialized.

The most visible symptom is that magic-link login email fails every time — which locks users out completely when email is the only configured auth provider.

Error

Failed to send email to <redacted>, Magic sign-in link for Trigger.dev.
Error Error: [EntityReplacer] Invalid character '#' in entity name: "#xD"
  Deserialization error: to see the raw response, inspect the hidden field {error}.$response on this object.
{"message":"Error sending magic link email","level":"error"}

Stack:

at fast-xml-parser@5.7.1 … setExternalEntities
at fast-xml-parser@5.7.1 … parse
at @aws-sdk/xml-builder@3.930.0/dist-cjs/xml-parser.js:17 … parseXML
at @aws-sdk/core@3.940.0 … XmlShapeDeserializer.parseXml
at @aws-sdk/core@3.940.0 … AwsQueryProtocol.deserializeResponse

The STS response itself is HTTP 200 and well formed — it contains no character references and the string #xD appears nowhere in it. The throw happens while configuring the parser, not while reading the document, so it is entirely content-independent.

Cause

@aws-sdk/xml-builder@3.930.0/dist-cjs/xml-parser.js registers two character references as entities:

parser.addEntity("#xD", "\r");
parser.addEntity("#10", "\n");

fast-xml-parser 5.7.0 tightened entity-name validation to reject #, so setExternalEntities throws on every parse() call.

Minimal reproduction (no AWS credentials, no network)

const { XMLParser } = require('fast-xml-parser');
const parser = new XMLParser({ htmlEntities: true });
parser.addEntity('#xD', '\r');   // exactly what @aws-sdk/xml-builder does
parser.parse('<a>1</a>', true);  // throws on 5.7.0 and 5.7.1

Affected versions

fast-xml-parser	Result
4.5.6	ok
5.6.0	ok
5.7.0	throws
5.7.1	throws
5.8.0	ok
5.9.3 / 5.10.1 / 5.11.0 / 5.11.1	ok

Broken only in 5.7.0–5.7.1, fixed in 5.8.0.

Why this affects every current release

pnpm-lock.yaml pins fast-xml-parser@5.7.1 on v4.5.9, v4.5.11, v4.5.13–v4.5.16, v4.6.0–v4.6.3 and main. No published image avoids the broken window, so operators cannot work around it by moving image tags.

The repo also resolves fast-xml-parser@4.5.6 for the older @aws-sdk/xml-builder@3.821.0. Which copy a given client resolves to decides whether it works: the ECR push path keeps working, while @aws-sdk/client-sesv2@3.940.0 (which pulls xml-builder@3.930.0) does not. That makes the failure look service-specific when it is really protocol-specific — every awsQuery/XML service, STS included, is affected.

Impact

Any self-hosted deployment using IRSA. Because STS uses the awsQuery (XML) protocol, credential resolution itself fails, so this is not limited to email — it breaks every AWS call made through the affected SDK copy.

Deployments using node/instance credentials are unaffected, since IMDS returns JSON and never touches the XML parser. That is why the problem appears only once a pod is switched to IRSA, and why it can sit latent for a long time before surfacing.

Suggested fix

Force a non-affected version:

// package.json
"pnpm": { "overrides": { "fast-xml-parser": "^5.8.0" } }

Environment

• ghcr.io/triggerdotdev/trigger.dev:v4.6.0, Helm chart, self-hosted on EKS
• Node 24
• EMAIL_TRANSPORT=aws-ses
• Credentials via IRSA (service account annotated with eks.amazonaws.com/role-arn)

Possibly related: #3364 (service-account annotation support for IRSA).