Skip to content

Look up accounting reservations by primary key from the usage trigger - #41

Merged
ancongui merged 1 commit into
mainfrom
fix/reservation-collation
Oct 13, 2026
Merged

ancongui merged 1 commit into
mainfrom
fix/reservation-collation

Conversation

@ancongui

Copy link
Copy Markdown
Contributor

Summary

Starting a run got slower with every run ever started: the usage accounting trigger looked up its reservation by scanning every row of operation_reservations (36–90 ms per inserted run at 524,000 reservations).

Cause. weave_usage_charge passes TG_TABLE_NAME (type name, collation "C") down the call chain, so PL/pgSQL gives every text parameter and variable of weave_reservation_touch, weave_reserve_refresh and weave_allocation_delta collation "C". Compared with the default-collation key columns, that rules out the primary key. auto_explain inside the trigger shows three affected statements:

Function Statement Before
weave_reserve_refresh SELECT amount,outbox_slots FROM operation_reservations WHERE kind=kind_name AND resource_id=identifier Seq Scan, kind = 'run'::text COLLATE "C"
weave_reserve_refresh EXISTS(... operation_control_context ... AND kind=kind_name ...) COLLATE "C" key condition
weave_allocation_delta SELECT/DELETE ... operation_allocations WHERE relation=relation_name AND row_key=identity_key Seq Scan, both columns COLLATE "C"

weave_control_begin/weave_control_finish receive literals or table-read kind values and are not affected.

Fix. Forward migration 0031_operations_key_collation patches both shared functions in place (pg_get_functiondef + replace, as in 0021/0025), comparing in the default collation. It fails closed unless each pattern occurs exactly twice; CREATE OR REPLACE keeps owner, grants and SET options. No table, data or role change. SCHEMA_VERSION moves to the new head.

Tests

  • New tests/integration/test_reservation_lookup.py: seeds 6,000 reservations and 6,000 allocations in another project, inserts a run through the trigger with auto_explain, and asserts index scans on operation_reservations_pkey / operation_allocations_pkey with both key columns as index conditions, no COLLATE "C" condition on any of the three tables, and unchanged usage accounting (assert_usage before and after).
  • Fails without the migration (Seq Scan), passes with it.
  • tests/integration/operations_support.py carries recompute_usage/assert_usage verbatim from feat/operations-observability, which is not merged yet; that branch's file is a superset.
  • make check: all requested checks passed.
  • Integration on real PostgreSQL 17 (Docker context colima-weave-tests): test_reservation_lookup, test_operations, test_large_row_accounting, test_run_lifecycle, test_outbox, test_terminal_capacity, test_boot, test_incident_migration, test_schedule_migration, test_runs, test_waits: 115 passed.

Upgrade note

Operators must run the explicit migration before restarting the server; CHANGELOG (Unreleased) and docs/operations/upgrades.md are updated.

Coordination

feat/operations-observability also adds a 0031_* migration. Whichever merges second renumbers to 0032_* and chains its down_revision.

The usage trigger passes TG_TABLE_NAME (collation "C") down to
weave_reserve_refresh and weave_allocation_delta, so their kind and
relation comparisons could not use the default-collation primary keys
and every charged write scanned all reservations and allocations.
Migration 0031_operations_key_collation patches both shared functions
to compare in the default collation; an auto_explain integration test
pins the primary-key plans and unchanged usage accounting.
@ancongui
ancongui merged commit 9c28f47 into main Oct 13, 2026
12 checks passed
@ancongui
ancongui deleted the fix/reservation-collation branch October 13, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant