Skip to content

tables: expose loggers actuator endpoint for dynamic log level control#508

Merged
cbb330 merged 1 commit intolinkedin:mainfrom
cbb330:chbush/tables-actuator-loggers-endpoint
Mar 24, 2026
Merged

tables: expose loggers actuator endpoint for dynamic log level control#508
cbb330 merged 1 commit intolinkedin:mainfrom
cbb330:chbush/tables-actuator-loggers-endpoint

Conversation

@cbb330
Copy link
Copy Markdown
Collaborator

@cbb330 cbb330 commented Mar 21, 2026

Summary

Adds the Spring Boot Actuator /actuator/loggers endpoint to the tables-service, enabling dynamic log level changes at runtime without pod restarts or deployment changes.

Changes

  • Client-facing API Changes
  • Internal API Changes
  • Bug Fixes
  • New Features
  • Performance Improvements
  • Code Style
  • Refactoring
  • Documentation
  • Tests

Added to services/tables/src/main/resources/application.properties:

management.endpoints.web.exposure.include=health, shutdown, prometheus, beans, loggers
management.endpoint.loggers.enabled=true

This exposes POST /actuator/loggers/{name} to set any logger to DEBUG (or reset it) on a live pod without restarting. Intended for incident use to activate targeted diagnostic logging — for example the loggers in application-hdfs-diagnostics.properties — on a single replica without modifying the deployment or replicaset.

Testing Done

  • Manually Tested on local docker setup. Please include commands ran, and their output.
  • Added new tests for the changes made.
  • Updated existing tests to reflect the changes made.
  • No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
  • Some other form of testing like staging or soak time in production. Please explain.

Local bootRun:

./gradlew :services:tables:bootRun --args='--server.port=18081'
# Endpoint appears in actuator index
GET http://localhost:18081/actuator
→ includes "loggers" and "loggers-name" hrefs

# Set a logger to DEBUG
POST http://localhost:18081/actuator/loggers/org.apache.hadoop.hdfs.DFSClient
Content-Type: application/json
{"configuredLevel": "DEBUG"}
→ HTTP 204

# Verify it took effect
GET http://localhost:18081/actuator/loggers/org.apache.hadoop.hdfs.DFSClient
→ {"configuredLevel": "DEBUG", "effectiveLevel": "DEBUG"}

Test cluster:

Deployed to the test cluster via internal build. Verified from inside the running pod:

kubectl exec <pod> -c tables-service -- \
  curl -s http://localhost:8080/actuator | python3 -m json.tool
{
    "loggers-name": {"href": "http://localhost:8080/actuator/loggers/{name}", "templated": true},
    "loggers":      {"href": "http://localhost:8080/actuator/loggers", "templated": false}
}

All log levels (OFF, FATAL, ERROR, WARN, INFO, DEBUG, TRACE) available via GET /actuator/loggers.

Why no new tests: Configuration-only change (application.properties). The actuator auto-configuration is covered by Spring Boot's own test suite; no functional code paths were modified.

Additional Information

  • Breaking Changes
  • Deprecations
  • Large PR broken into smaller PRs, and PR plan linked in the description.

Adds the Spring Boot Actuator /actuator/loggers endpoint to the tables-service,
enabling dynamic log level changes at runtime without pod restarts or deployment
changes.

This supports the hdfs-diagnostics tooling: enable-hdfs-debug.sh uses this
endpoint to activate DEBUG logging on a single pod during incident windows,
mirroring the loggers configured in application-hdfs-diagnostics.properties.
@cbb330
Copy link
Copy Markdown
Collaborator Author

cbb330 commented Mar 23, 2026

Note: the management.endpoints.web.exposure.include change from this PR is included in #511 (which also does the broader SLF4J/log4j unification needed to make the loggers endpoint actually work end-to-end). If #511 merges first, this PR will need to drop the application.properties change to avoid a conflict.

@cbb330 cbb330 merged commit 6aa7cbb into linkedin:main Mar 24, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants