Release 3.9.0

CHANGELOG.md

@@ -11,6 +11,54 @@ target.
 
 ## [Unreleased]
 
+## [3.9.0] — 2026-10-20
+
+The Add to ThinkWatch Lite link from a rotated key now carries the key's
+request limits. A database failure during authentication answers 503 or
+500 with the usual JSON error instead of an empty 500, and password
+hashing no longer runs on the request workers. Monitoring that matches
+only on 500 sees a database outage during authentication as a 503 from
+this release on. The thinkwatch-core crates stay at v0.77.0.
+
+No setting, schema or API route changes.
+
+### Added
+
+- **The rotated-key dialog's Add to ThinkWatch Lite link carries the key's
+  request limits.** The key's own enabled, unexpired AI-gateway rules that
+  count requests over a window of whole minutes up to 60, or whole hours
+  up to 24 (the console's 1 minute, 5 minutes, 1 hour, 5 hours and 24
+  hours among them), go into the link's `limits` parameter as Lite's
+  rolling windows of the same length; of two rules on one window the
+  smaller count goes, and at most 16 windows, those allowing the fewest
+  requests per second. Token rules stay out, since they count weighted
+  tokens and Lite counts plain ones; so do sliding 7-day rules (Lite's
+  week is a calendar week), budgets, MCP rules and the owner's limits.
+  A link with limits needs ThinkWatch Lite 2026.10.26 or later, which the
+  dialog states under the button. Reading the rules needs
+  `rate_limits:read` in a scope covering the key; without it, or when the
+  rules cannot be read, the link is built without limits as before. A new
+  key has no limits of its own yet, so its link carries none.
+
+### Fixed
+
+- **A database outage during authentication is a 503 with an error body,
+  not an empty 500.** When the session or API-key check could not reach
+  Postgres (most often a connection-pool timeout), the console and the
+  gateway answered `500 Internal Server Error` with no body and, for the
+  permission lookup, no log line. These lookups now answer like every
+  handler does: `503 Service Unavailable` with the usual JSON error when
+  the database is unavailable, `500` with it for any other database
+  error, and an error log naming the lookup that failed.
+- **Password hashing no longer holds up other requests.** Logins,
+  password changes and resets, new users, turning off two-factor
+  authentication and first-run setup ran Argon2 (tens of milliseconds of
+  CPU by design) on the server's async workers, so every other request
+  scheduled on that worker waited for it. They now run it on the blocking
+  thread pool, at most one run per CPU at a time as before, so a burst of
+  logins holds no more memory than it did (19 MiB per run). The hashes
+  and the constant-time login are unchanged.
+
 ## [3.8.3] — 2026-10-20
 
 Tool-call inspection in streamed answers now decides only on arguments
@@ -2489,7 +2537,8 @@ unreleased builds should: stop the gateway, run `db/schema.sql`
 against PostgreSQL, restart against this tag. The schema is
 idempotent end-to-end, so the apply is safe to repeat.
 
-[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.8.3...HEAD
+[Unreleased]: https://github.com/ThinkWatchProject/ThinkWatch/compare/v3.9.0...HEAD
+[3.9.0]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.9.0
 [3.8.3]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.3
 [3.8.2]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.2
 [3.8.1]: https://github.com/ThinkWatchProject/ThinkWatch/releases/tag/v3.8.1

Cargo.toml

@@ -10,7 +10,7 @@ members = [
 ]
 
 [workspace.package]
-version = "3.8.3"
+version = "3.9.0"
 edition = "2024"
 # The MSRV: the newest `rust-version` among the locked dependencies
 # (sqlx 0.9 and the aws-smithy crates under aws-sigv4). Without it,

crates/auth/src/password.rs

@@ -1,7 +1,10 @@
+use std::sync::{Arc, LazyLock};
+
 use argon2::{
     Argon2,
     password_hash::{PasswordHasher, PasswordVerifier, phc::PasswordHash},
 };
+use tokio::sync::Semaphore;
 
 const RANDOM_PASSWORD_LEN: usize = 16;
 const CHARSET: &[u8] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!@#$%&*";
@@ -52,10 +55,105 @@ pub fn verify_password_or_dummy(password: &str, hash: Option<&str>) -> bool {
     }
 }
 
+// Argon2 costs tens of milliseconds of CPU on purpose. Called from a
+// request handler it would hold an async worker for that long, stalling
+// every other request scheduled on that thread, so the handlers use
+// these versions, which run it on the blocking pool.
+
+/// How many Argon2 runs go at once: one per CPU, as many as the async
+/// workers allowed when they ran it. Each run holds 19 MiB (`m=19456`)
+/// while it lasts and the blocking pool grows to hundreds of threads, so
+/// without this a burst of logins would hold gigabytes at once.
+static ARGON2_PERMITS: LazyLock<Arc<Semaphore>> = LazyLock::new(|| {
+    Arc::new(Semaphore::new(
+        std::thread::available_parallelism().map_or(1, |n| n.get()),
+    ))
+});
+
+/// `f` on the blocking pool once an [`ARGON2_PERMITS`] place is free. The
+/// task keeps the place until it ends, even when the caller has gone.
+async fn on_blocking_pool<T: Send + 'static>(
+    f: impl FnOnce() -> T + Send + 'static,
+) -> Result<T, tokio::task::JoinError> {
+    // Never closed, so the acquire cannot fail; the permit inside the
+    // result is held all the same.
+    let permit = Arc::clone(&ARGON2_PERMITS).acquire_owned().await;
+    tokio::task::spawn_blocking(move || {
+        let _permit = permit;
+        f()
+    })
+    .await
+}
+
+/// [`hash_password`] on the blocking pool.
+pub async fn hash_password_off_runtime(password: String) -> anyhow::Result<String> {
+    on_blocking_pool(move || hash_password(&password))
+        .await
+        .map_err(|e| anyhow::anyhow!("Password hashing task failed: {e}"))?
+}
+
+/// [`verify_password`] on the blocking pool.
+pub async fn verify_password_off_runtime(password: String, hash: String) -> anyhow::Result<bool> {
+    on_blocking_pool(move || verify_password(&password, &hash))
+        .await
+        .map_err(|e| anyhow::anyhow!("Password verification task failed: {e}"))?
+}
+
+/// [`verify_password_or_dummy`] on the blocking pool. A task that cannot
+/// run counts as a mismatch, like an unreadable hash.
+pub async fn verify_password_or_dummy_off_runtime(password: String, hash: Option<String>) -> bool {
+    on_blocking_pool(move || verify_password_or_dummy(&password, hash.as_deref()))
+        .await
+        .unwrap_or(false)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
+    #[tokio::test]
+    async fn the_off_runtime_versions_agree_with_the_blocking_ones() {
+        let hash = hash_password_off_runtime("pw".into()).await.unwrap();
+        assert!(
+            verify_password_off_runtime("pw".into(), hash.clone())
+                .await
+                .unwrap()
+        );
+        assert!(
+            !verify_password_off_runtime("nope".into(), hash.clone())
+                .await
+                .unwrap()
+        );
+        assert!(verify_password_or_dummy_off_runtime("pw".into(), Some(hash)).await);
+        assert!(!verify_password_or_dummy_off_runtime("pw".into(), None).await);
+    }
+
+    #[tokio::test(flavor = "multi_thread")]
+    async fn no_more_runs_at_once_than_there_are_permits() {
+        use std::sync::atomic::{AtomicUsize, Ordering};
+        static RUNNING: AtomicUsize = AtomicUsize::new(0);
+        static PEAK: AtomicUsize = AtomicUsize::new(0);
+        let permits = std::thread::available_parallelism().map_or(1, |n| n.get());
+        let tasks: Vec<_> = (0..permits * 4)
+            .map(|_| {
+                tokio::spawn(on_blocking_pool(|| {
+                    let now = RUNNING.fetch_add(1, Ordering::SeqCst) + 1;
+                    PEAK.fetch_max(now, Ordering::SeqCst);
+                    std::thread::sleep(std::time::Duration::from_millis(20));
+                    RUNNING.fetch_sub(1, Ordering::SeqCst);
+                }))
+            })
+            .collect();
+        for task in tasks {
+            task.await.unwrap().unwrap();
+        }
+        let peak = PEAK.load(Ordering::SeqCst);
+        assert!(
+            (1..=permits).contains(&peak),
+            "peak {peak}, permits {permits}"
+        );
+    }
+
     /// A hash written by argon2 0.5.3 (`Argon2::default()`, fixed salt).
     /// Stored hashes outlive crate upgrades: they must keep verifying.
     const KAT_HASH: &str = "$argon2id$v=19$m=19456,t=2,p=1$dGhpbmt3YXRjaC1rYXQtc2FsdA$6xffcgfh1b9KJMWg4mBS4qOWsK5MNUM1asLNHvgV0v8";

crates/server/src/handlers/admin/users.rs

@@ -458,7 +458,7 @@ pub async fn create_user(
         return Err(AppError::Conflict("Email already registered".into()));
     }
 
-    let password_hash = password::hash_password(&raw_password)?;
+    let password_hash = password::hash_password_off_runtime(raw_password.clone()).await?;
 
     let mut tx = state.db.begin().await?;
 
@@ -920,7 +920,7 @@ pub async fn reset_user_password(
     }
 
     let new_password = password::generate_random_password();
-    let hash = password::hash_password(&new_password)?;
+    let hash = password::hash_password_off_runtime(new_password.clone()).await?;
 
     user_repository::update_password_hash(&state.db, user_id, &hash, true).await?;
 

crates/server/src/handlers/api_keys.rs

@@ -927,11 +927,10 @@ pub async fn rotate_key(
         .await;
     let grace_period_ends_at = chrono::Utc::now() + chrono::Duration::hours(grace_hours);
 
-    // Generate new key. Rate-limit / budget rules attached to the
-    // OLD key id are NOT copied here — they live in `rate_limit_rules`
-    // / `budget_caps` keyed by api_key UUID. The follow-up phase
-    // ("limits attached to api_keys") will add an explicit copy step
-    // when rotation happens.
+    // Generate new key. Nothing to copy for limits: the key's
+    // rate-limit rules and budgets are attached to its `lineage_id`
+    // (`api_key_lineage` subjects, see `think_watch_common::limits`),
+    // and the new key keeps that lineage, so they carry over as is.
     let generated = api_key::generate_api_key();
 
     // The new key carries the old one's name verbatim — the chain is

crates/server/src/handlers/auth.rs

@@ -497,10 +497,11 @@ pub async fn login(
     // Constant-time login: always run Argon2, against a dummy hash when
     // there is no account or it has no password, so the response time does
     // not tell which addresses have accounts.
-    let password_valid = password::verify_password_or_dummy(
-        &req.password,
-        user.as_ref().and_then(|u| u.password_hash.as_deref()),
-    );
+    let password_valid = password::verify_password_or_dummy_off_runtime(
+        req.password.clone(),
+        user.as_ref().and_then(|u| u.password_hash.clone()),
+    )
+    .await;
 
     if !password_valid || user.is_none() {
         // Per-email cross-IP failure counter. The `count` above is keyed
@@ -1053,7 +1054,7 @@ pub async fn register(
     let email = normalize_email(&req.email);
     validate_email(&email)?;
 
-    let password_hash = password::hash_password(&req.password)?;
+    let password_hash = password::hash_password_off_runtime(req.password.clone()).await?;
 
     // Use a transaction to ensure user creation + role assignment are atomic
     let mut tx = state.db.begin().await?;
@@ -1467,7 +1468,9 @@ pub async fn change_password(
         return Err(AppError::Unauthorized);
     }
 
-    if !password::verify_password(&req.old_password, current_hash)? {
+    if !password::verify_password_off_runtime(req.old_password.clone(), current_hash.to_owned())
+        .await?
+    {
         let fails =
             crate::services::auth_lockout::record_failure(&state.redis, &pwchange_fail_key).await?;
         if let Some(secs) = crate::services::auth_lockout::ladder_secs(fails) {
@@ -1481,7 +1484,7 @@ pub async fn change_password(
     crate::services::auth_lockout::clear(&state.redis, &[&pwchange_fail_key, &pwchange_locked_key])
         .await;
 
-    let new_hash = password::hash_password(&req.new_password)?;
+    let new_hash = password::hash_password_off_runtime(req.new_password.clone()).await?;
     repo::set_own_password(&state.db, user.id, &new_hash).await?;
 
     // Revoke all signing public keys for this user (invalidates sessions)
@@ -1832,7 +1835,7 @@ pub async fn totp_disable(
     let hash = user.password_hash.as_ref().ok_or(AppError::BadRequest(
         "SSO accounts cannot manage TOTP here".into(),
     ))?;
-    if !password::verify_password(&req.old_password, hash)? {
+    if !password::verify_password_off_runtime(req.old_password.clone(), hash.to_owned()).await? {
         return Err(AppError::Unauthorized);
     }
 

crates/server/src/handlers/setup.rs

@@ -145,7 +145,7 @@ pub async fn setup_initialize(
 
     // Create the super_admin user with the first API key, and mark
     // setup done.
-    let password_hash = password::hash_password(&req.admin.password)?;
+    let password_hash = password::hash_password_off_runtime(req.admin.password.clone()).await?;
     let generated = api_key::generate_api_key();
     let site_name = req.site_name.as_deref().unwrap_or("ThinkWatch");
     let admin_user = repo::create_first_admin(

crates/server/src/middleware/api_key_auth.rs

@@ -365,9 +365,16 @@ fn authenticate(
             .bind(&key_hash)
             .bind(global_inactivity_days)
             .fetch_optional(&state.db)
-            .await
-            .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?
-            .ok_or(StatusCode::UNAUTHORIZED)?;
+            .await;
+            let row = match row {
+                Ok(row) => row.ok_or(StatusCode::UNAUTHORIZED)?,
+                Err(e) => {
+                    return Ok(super::auth_guard::db_failure_response(
+                        "Gateway API key lookup failed",
+                        e,
+                    ));
+                }
+            };
 
             // Surface gate. Even a valid key gets rejected if its
             // `surfaces` list doesn't include the gateway being called.
@@ -433,9 +440,15 @@ fn authenticate(
                 // the aggregated `surface_constraints` JSON so the gateway
                 // hot path has rate limits + budgets without further lookups.
                 let (role_limits, user_roles, loaded_limits) = if let Some(uid) = row.user_id {
-                    let limits = rbac::compute_user_resource_limits(&state.db, uid)
-                        .await
-                        .map_err(|_| StatusCode::INTERNAL_SERVER_ERROR)?;
+                    let limits = match rbac::compute_user_resource_limits(&state.db, uid).await {
+                        Ok(limits) => limits,
+                        Err(e) => {
+                            return Ok(super::auth_guard::db_failure_response(
+                                "Loading the key owner's resource limits failed",
+                                e,
+                            ));
+                        }
+                    };
                     let names = rbac::load_user_role_names(&state.db, uid)
                         .await
                         .unwrap_or_default();