timestamp_format is ignored

[davidjmemmett]

Describe the bug
Given standard high-precision syslog format and config: "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f+00:00" and "trim_timestamp": true, the timestamp in CloudWatch Logs differs from the timestamp in the log message, and the timestamp isn't trimmed.

Steps to reproduce

                    {
                        "file_path": "/var/log/syslog",
                        "log_group_class": "STANDARD",
                        "log_group_name": "syslog",
                        "log_stream_name": "{instance_id}",
                        "retention_in_days": 30,
                        "timestamp_format": "%Y-%m-%dT%H:%M:%S.%f+00:00",
                        "trim_timestamp": true
                    },

What did you expect to see?
No timestamp in the Message field and the correct timestamp in the Timestamp field

What did you see instead?
The correct timestamp in the Message field, and an incorrect timestamp in the Timestamp field

What version did you use?
CWAgent/1.300064.0b1337 (go1.25.5; linux; arm64)

What config did you use?
As above

Environment
Ubuntu 24.04.2 LTS

Additional context

[davidjmemmett]

To confirm, I ensured that the time format is correct, as per the docs and the standard strptime/strftime functions:

>>> datetime.strptime("2026-02-04T11:27:44.945314+00:00", "%Y-%m-%dT%H:%M:%S.%f+00:00")
datetime.datetime(2026, 2, 4, 11, 27, 44, 945314)

[davidjmemmett]

The generated TOML config file contains the following lines, which appears to miss the dot before the high-precision timestamp in the timestamp_regex:

      timestamp_layout = ["2006-01-_2T15:04:05.000+00:00", "2006-1-_2T15:04:05.000+00:00"]
      timestamp_regex = "(\\d{4}-\\s{0,1}\\d{1,2}-\\s{0,1}\\d{1,2}T\\d{2}:\\d{2}:\\d{2}(\\d{1,9})+00:00)"

[davidjmemmett]

After some messing about, I've tweaked the expression in RegEx101 in order to make it match: https://regex101.com/r/QI1Xe8/1

The new expression: (\d{4}-\s{0,1}\d{1,2}-\s{0,1}\d{1,2}T\d{2}:\d{2}:\d{2}\.(\d{1,9})\+00:00)

So I believe the original expression is missing the \. between the seconds and the microseconds, as well as escaping on the \+ before the hard-coded timezone.

[varunch77]

Hi David, thanks for deep diving on this issue and bringing this to our attention. I will bring this up to my team for further discussion.

[jefchien]

Tested the configuration and it gets generated with the escaped .. The issue is that the agent doesn't escape the +.

      timestamp_layout = ["2006-01-_2T15:04:05..000+00:00", "2006-1-_2T15:04:05..000+00:00"]
      timestamp_regex = "(\\d{4}-\\s{0,1}\\d{1,2}-\\s{0,1}\\d{1,2}T\\d{2}:\\d{2}:\\d{2}\\.(\\d{1,9})+00:00)"

[davidjmemmett]

Hi @jefchien @varunch77 ,
I've got AWS CloudWatch Agent installed via a AWS EC2 ImageBuilder recipe - is there an easy way for me to update it to an alpha build containing this fix to verify?
Cheers,
David

[davidjmemmett]

Any progress @jefchien @varunch77 ?

[jefchien]

Hi @davidjmemmett,

Sorry for the delay (need to look at GitHub notifications more). We don't have an alpha build that is publicly available. How do you install the CloudWatch agent in your ImageBuilder recipe? Is it from S3 or the Amazon Linux yum repos?

[davidjmemmett]

Hi @davidjmemmett,

Sorry for the delay (need to look at GitHub notifications more). We don't have an alpha build that is publicly available. How do you install the CloudWatch agent in your ImageBuilder recipe? Is it from S3 or the Amazon Linux yum repos?

It's AWS' recipe & it's on Ubuntu, so I've no idea what it does under the hood.

[jefchien]

Took a look at the EC2 Image builder component (amazon-cloudwatch-agent-linux). Looks like it downloads the CloudWatch agent from the regional S3 bucket.

https://s3.$REGION.amazonaws.com/amazoncloudwatch-agent-$REGION/$OS/$TYPE/latest/amazon-cloudwatch-agent.$EXTENSION

[davidjmemmett]

Took a look at the EC2 Image builder component (amazon-cloudwatch-agent-linux). Looks like it downloads the CloudWatch agent from the regional S3 bucket.

https://s3.$REGION.amazonaws.com/amazoncloudwatch-agent-$REGION/$OS/$TYPE/latest/amazon-cloudwatch-agent.$EXTENSION

I'm just including the component ARN: arn:aws:imagebuilder:eu-west-2:aws:component/amazon-cloudwatch-agent-linux/1.0.1/1

[davidjmemmett]

Hi @jefchien,
Any idea when I can use a newer build?

[jefchien]

Hi @davidjmemmett,

The CloudWatch Agent follows a regular release cycle. You can check the release tags (https://github.com/aws/amazon-cloudwatch-agent/tags) to confirm which release includes the fix.

Once a release is built, it is rolled out progressively across all distribution channels (S3, SSM, container images, Amazon Linux repos, EKS add-on, etc.), which can take several weeks to complete across all regions and channels. This means there can be a gap between when a release is tagged and when updated artifacts are available.

We don't publish fixed timelines for individual patches.