Worker thread crashes (all API requests timeout) after disk has been full

[ErikBjare]

My system drive got full the other day, and today I noticed aw-server-rust wasn't responding to API requests (but was happily serving the web UI).

After a while I realized what may have caused it, and indeed found in the logs:

thread '<unnamed>' panicked at 'Failed to commit datastore transaction! database or disk is full', aw-datastore/src/worker.rs:182:29

So, when the disk gets full, the DB worker thread panics, and future DB requests will just fail to respond (since the worker is dead).

[johan-bjareholt]

I've seen this a few times before as well, have forgotten to document it.

What do you think the correct behaviour should be?
The options i see are:

• Shut down aw-server-rust completely so the user gets notified by aw-qt
• Drop all transactions until the disk is no longer full, but print warnings in the log.

[ErikBjare]

I think the server should respond with HTTP code 503, with a suitable error in the log. That way, watchers can queue events in memory, as appropriate.

…

On Mon, Nov 15, 2021 at 10:54 PM Johan Bjäreholt ***@***.***> wrote: I've seen this a few times before as well, have forgotten to document it. What do you think the correct behaviour should be? The options i see are: - Shut down aw-server-rust completely so the user gets notified by aw-qt - Drop all transactions until the disk is no longer full, but print warnings in the log. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#256 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKXDOTMBMVHWTQZ6E6K6XDUMF6RNANCNFSM5ICMDH7A> .

[TimeToBuildBob]

Triage: Root Cause Analysis

The crash happens because of two panic!() calls in the worker thread's commit path (aw-datastore/src/worker.rs):

1. Transaction start failure (partially handled)

// work_loop(), line ~130 — this one retries after 1s, so it survives transient failures
let mut tx = match conn.transaction_with_behavior(TransactionBehavior::Immediate) {
    Ok(tx) => tx,
    Err(err) => { error!("Unable to start transaction! {:?}", err); continue; }
};

2. Transaction commit failure (panics)

// work_loop(), line ~158 — this is the crash site
match tx.commit() {
    Ok(_) => (),
    Err(err) => panic!("Failed to commit datastore transaction! {err}"),
}

When the disk is full, SQLite returns SQLITE_FULL (error 13) on commit. The worker thread panics, which kills the thread. Since the Datastore struct communicates with the worker via an mpsc channel, all subsequent requests from the HTTP layer get an MpscError → HTTP 500. The server process stays alive but the datastore is permanently broken until restart.

The same panic pattern exists in the legacy import commit path (~line 120), but that only runs at startup.

Proposed Fix

Replace the panic! on commit failure with graceful error propagation:

1. In work_loop(): When tx.commit() fails, log the error and continue the outer loop (the transaction gets rolled back via Drop). The next iteration will start a fresh transaction. This matches what already happens for transaction-start failures.

2. Propagate write errors to callers: For the in-flight request that triggered the commit, the response has already been sent (optimistic). To properly surface the error, either:

   • (Simple) Set tx.set_drop_behavior(DropBehavior::Rollback) instead of Commit, and only explicitly commit. This way failed commits don't lose data silently — the caller gets their response, and the next heartbeat/insert re-sends the data.
   • (Better) Track whether the last commit succeeded, and if not, return a new DatastoreError::DiskFull variant that maps to HTTP 503 in util.rs. This lets watchers know to queue events client-side, as Erik suggested.

3. Add a DiskFull variant to DatastoreError that maps to Status::ServiceUnavailable (503) in the From<DatastoreError> for HttpErrorJson impl. This gives watchers the signal to back off and retry.

The minimal safe fix is just changing the panic to error! + continue in the commit path — that alone prevents the crash and lets the server recover once disk space is freed.

[TimeToBuildBob]

Fix submitted in PR #572 — replaces the panic\!() calls in the commit path with graceful error handling and transaction retry logic.