[Bug] apm-toolkit-log4j-2.x jar has been missing Log4j2Plugins.dat since 9.5.0, breaking %traceId, %sw_ctx and GRPCLogClientAppender

[ElliotXue-crypto]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

Since 9.5.0, the released apm-toolkit-log4j-2.x jar no longer contains the Log4j2 plugin descriptor:

META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat

Without that descriptor Log4j2 cannot discover TraceIdConverter, so %traceId in a PatternLayout is greedily matched as the built-in %t (thread name) followed by the literal raceId:

ERROR [main] [mainraceId] Foo: message     <- actual
ERROR [main] [TID: N/A]   Foo: message     <- expected

Verified directly against Maven Central:

Version	Log4j2Plugins.dat
9.0.0 – 9.4.0	present
9.5.0 – 9.7.0	missing

Only apm-toolkit-log4j-2.x is affected. apm-toolkit-log4j-1.x and apm-toolkit-logback-1.x never shipped this file, as they do not use the Log4j2 plugin mechanism.

Root cause

The toolkit module itself did not change:

• apm-toolkit-log4j-2.x/pom.xml is identical between 9.4.0 and 9.5.0 apart from the parent <version>.
• Every converter/appender class is byte-identical (sha1, first 12 chars):

class	9.4.0	9.6.0
TraceIdConverter	76bf70b1d17c	76bf70b1d17c
SkyWalkingContextConverter	c5481fdab754	c5481fdab754
Log4j2OutputAppender	79bc1667fbac	79bc1667fbac
Log4j2SkyWalkingContextOutputAppender	cbe0b6076cee	cbe0b6076cee

The change is in the java-agent root pom. 9.5.0 added an annotationProcessorPaths block to maven-compiler-plugin:

<annotationProcessorPaths>
    <path>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <version>${lombok.version}</version>
    </path>
</annotationProcessorPaths>

Once annotationProcessorPaths is declared explicitly, javac stops discovering annotation processors from the compile classpath. apm-toolkit-log4j-2.x declares log4j-core as provided, and it is Log4j2's PluginProcessor — previously picked up from that classpath — that generates Log4j2Plugins.dat. With the processor list narrowed to Lombok only, PluginProcessor never runs and the descriptor is silently dropped.

This matches the 9.5.0 changelog entry:

Centralized plugin version management in the root POM and remove redundant declarations

so this looks like an unintended side effect of a build refactor rather than a deliberate removal.

What you expected to happen

The released jar should contain Log4j2Plugins.dat, so that %traceId and %sw_ctx resolve automatically as documented, without users having to declare packages in log4j2.xml.

How to reproduce

Confirm the descriptor is missing:

for v in 9.4.0 9.5.0 9.6.0 9.7.0; do
  curl -sfLO "https://repo1.maven.org/maven2/org/apache/skywalking/apm-toolkit-log4j-2.x/$v/apm-toolkit-log4j-2.x-$v.jar"
  printf "%-8s dat=%s\n" "$v" "$(unzip -l apm-toolkit-log4j-2.x-$v.jar | grep -c Log4j2Plugins.dat)"
done

Output:

9.4.0    dat=1
9.5.0    dat=0
9.6.0    dat=0
9.7.0    dat=0

End-to-end, with apm-toolkit-log4j-2.x:9.6.0 and log4j-core:2.24.3 on the classpath and this log4j2.xml:

<Configuration status="warn">
  <Appenders>
    <Console name="Console" target="SYSTEM_OUT">
      <PatternLayout pattern="%p [%t] [%traceId] %c{1.}: %m%n"/>
    </Console>
  </Appenders>
  <Loggers>
    <Root level="DEBUG"><AppenderRef ref="Console"/></Root>
  </Loggers>
</Configuration>

a single log.error("MARKER") prints ERROR [main] [mainraceId] .... Swapping the toolkit to 9.4.0, with no other change, prints ERROR [main] [TID: N/A] ....

Anything else

Impact. Every project on 9.5.0+ that uses %traceId or %sw_ctx in log4j2.xml. The failure is silent — logging keeps working and only the trace id is replaced by a thread name, so it is easy to ship without noticing. This has been broken across three releases (9.5.0, 9.6.0, 9.7.0).

Workarounds, both verified locally:

1. Declare the package explicitly. Note packages is deprecated in Log4j2 and removed in Log4j 3:

   <Configuration packages="org.apache.skywalking.apm.toolkit.log.log4j.v2.x">

2. Pin the toolkit back to 9.4.0. Since all classes are byte-identical, this is functionally equivalent to 9.6.0 plus the missing descriptor.

Suggested fix. Add log4j-core to the processor paths for this module. Because annotationProcessorPaths replaces rather than appends when overriding the parent, the Lombok entry has to be repeated:

<!-- apm-toolkit-log4j-2.x/pom.xml -->
<build>
  <plugins>
    <plugin>
      <artifactId>maven-compiler-plugin</artifactId>
      <configuration>
        <annotationProcessorPaths>
          <path>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>${lombok.version}</version>
          </path>
          <path>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j-core.version}</version>
          </path>
        </annotationProcessorPaths>
      </configuration>
    </plugin>
  </plugins>
</build>

It would also be worth adding a build-time check that the descriptor exists in the packaged jar, so a future refactor cannot drop it silently again.

Disclosure. The root-cause analysis above was done with the help of an AI assistant (Claude). Every claim in it — the version boundary, the class checksums, the pom diff, and both workarounds — was verified against artifacts published on Maven Central and can be reproduced with the commands above.

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Thanks for finding this and reporting this. This is an issue.