feat(etcd): periodic auth token refresh to prevent expiry

[okaybase]

Summary

The etcd auth token acquired at connect time was never refreshed. Once it expired (default 300s --auth-token-ttl), every subsequent Admin API write and etcd watch operation failed with etcdserver: invalid auth token.

Changes

• aisix-etcd: add start_token_refresh_task — a background Tokio task that calls Client::refresh_token on a configurable interval (default 240s, leaving a 60s margin below etcd's default 300s TTL). No-ops when no credentials are configured, or when called outside a Tokio runtime.
• aisix-core: add EtcdConfig.auth_token_refresh_secs field (default 240). Documented in config.example.yaml and config.managed.yaml.
• aisix-server: thread auth_token_refresh_secs from config through EtcdConfigProvider::connect and EtcdConfigStore::new.
• aisix-admin: EtcdConfigStore::new now spawns the refresh task for the admin write-path client, preventing Admin API writes from failing after token expiry.
• Dependency: bump etcd-client from 0.14 to 0.19 to pick up Client::refresh_token and the merged WatchStream (no longer requires a separate Watcher handle).
• Tests: add admin_write_survives_token_expiry and token_refresh_prevents_auth_expiry regression tests against an auth-enabled etcd with a 5s TTL, exercising the refresh loop.
• CI: add etcd-auth service container (short TTL) and "Enable auth on etcd-auth" step to run the token-refresh tests.

Verification

• cargo test --workspace passes
• cargo clippy --workspace -- -D warnings clean
• CI rust-unit job runs the new token-refresh tests against the auth-enabled etcd container
• Manual: start gateway with etcd.user + password_env, verify Admin API writes still succeed after 5+ minutes

Summary by CodeRabbit

• New Features

  • Added configurable etcd authentication-token refresh intervals.
  • Tokens refresh automatically to keep authenticated connections and watch operations available beyond token expiry.
  • The default refresh interval is 240 seconds and can be overridden in configuration.
  • Added retry handling for temporary token-refresh failures.

• Bug Fixes

  • Improved reliability for authenticated admin requests and watch operations.

• Tests

  • Added integration coverage for token refresh, authentication expiry, and authenticated watch events.

[CLAassistant]

All committers have signed the CLA.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The change adds configurable etcd authentication-token refresh, updates provider and application wiring, upgrades etcd-client, adds authenticated integration tests, and provisions an auth-enabled etcd service in CI.

Changes

etcd authentication refresh

Layer / File(s)	Summary
Refresh configuration contract Cargo.toml, crates/aisix-core/src/config.rs, config*.yaml	Adds auth_token_refresh_secs with a default of 240 seconds, YAML override coverage, documented examples, and updates etcd-client to 0.19.
Provider refresh and watch behavior crates/aisix-etcd/src/etcd_provider.rs, crates/aisix-etcd/src/lib.rs, crates/aisix-etcd/tests/watch_integration.rs	Adds configurable background token refresh, updates provider signatures and exports, adjusts watch-stream ownership, and tests refresh across token expiry.
Application and admin-store wiring crates/aisix-server/src/main.rs, crates/aisix-server/src/export/mod.rs, crates/aisix-admin/src/etcd_store.rs, crates/aisix-admin/tests/etcd_integration.rs, crates/aisix-admin/Cargo.toml	Passes refresh configuration through server and admin-store construction and verifies authenticated admin writes and watches after token expiry.
Authenticated etcd CI environment .github/workflows/ci.yml	Adds an auth-enabled etcd service, credentials and endpoints, and authentication bootstrap commands for integration tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Server
  participant EtcdConfigProvider
  participant EtcdClient
  participant AdminStore
  Server->>EtcdConfigProvider: connect with refresh interval
  EtcdConfigProvider->>EtcdClient: authenticate and start refresh task
  EtcdConfigProvider->>AdminStore: provide connected client
  AdminStore->>EtcdClient: write admin configuration
  EtcdConfigProvider->>EtcdClient: refresh token periodically

Loading

🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
E2e Test Quality Review	⚠️ Warning	The real-etcd tests cover Admin writes, reads, and watches, but the merged tree leaves Cargo.toml at etcd-client 0.19 while Cargo.lock remains at 0.14.1.	Regenerate and commit Cargo.lock for etcd-client 0.19, then rerun the locked workspace build and the authenticated integration tests.

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: periodic etcd authentication token refresh to prevent token expiry.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Security Check	✅ Passed	Changed code refreshes in-memory etcd tokens without logging credentials or persisting new secrets; it does not alter authorization, ownership, TLS verification, isolation, or secret resolution.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[moonming]

Thanks for working on this — the underlying issue is real, and periodically refreshing credential-based etcd auth is valuable. The overall direction looks reasonable.

Before this is ready to merge, please address the following blocking items:

1. Sync with the latest main and restore workspace compilation.
   The current PR changes EtcdConfigProvider::connect_with_policy to take a fifth refresh_secs argument. After combining the PR with the latest main, crates/aisix-server/src/export/mod.rs still calls it with four arguments, and cargo check --workspace fails with E0061. Please update that call (the export path should likely pass None) or preserve the existing API and add a separate refresh-aware method.

2. Retry promptly after a token-refresh failure.
   The current loop logs a failed refresh and then waits until the next full interval. With the default 240s refresh interval and a 300s token TTL, a transient failure at the first refresh can let the current token expire and leave the process unauthenticated until the next tick (up to roughly 180s after expiry). Please add a short bounded retry/backoff after failures, then return to the normal interval after a successful refresh.

3. Exercise the authenticated watch path in the regression test.
   token_refresh_prevents_auth_expiry currently waits past the TTL and calls load_all(), so it only verifies a range/get request. Please create/recreate provider.watch(...) after the TTL window, write an event, and assert that the stream receives it. This is especially important because the PR also changes the WatchStream ownership/API while claiming to fix watch operations.

After updating the branch, please ensure the new fork workflow run is approved and actually executes all jobs. The current workflow runs are action_required with no jobs, so CodeRabbit and CLA are currently the only reported checks. At minimum, the updated branch should pass:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
cargo check --workspace

Once these items are addressed, this PR should be worth proceeding with.

[okaybase]

@moonming Thanks for the detailed review! All blocking items have been addressed — compilation fix, retry logic, watch path coverage, and sync with main. Please take another look.