helios/deploy failed in CI: collections.len() != 1

[iximeow]

This test failed on a CI run on #10560 (commit 07578fd)

https://github.com/oxidecomputer/omicron/pull/10560/checks?check_run_id=80211588124

i've rerun this, so i'm not sure if the check still has anything useful to offer.

Log showing the specific test failure:

https://buildomat.eng.oxide.computer/wg/0/details/01KTMPA8HJ9Y5XAQ7306QTTZCZ/x4GX4ri3hooCo1wJ54Vy7za25Fw0K5BLFxddqidfDabDXZlK/01KTMPB4PMV7V2KYAEK7PCYZ1Q#S1838

Excerpt from the log showing the failure:

1849  2026-06-09T02:21:18.549Z  ++ pfexec zlogin oxz_switch /opt/oxide/omdb/bin/omdb db zpool list -i
1850  2026-06-09T02:21:18.549Z  ++ wc -l
1851  2026-06-09T02:21:18.580Z  note: database URL not specified.  Will search DNS.
1852  2026-06-09T02:21:18.580Z  note: (override with --db-url or OMDB_DB_URL)
1853  2026-06-09T02:21:18.580Z  note: using DNS from system config (typically /etc/resolv.conf)
1854  2026-06-09T02:21:18.580Z  note: (if this is not right, use --dns-server to specify an alternate DNS server)
1855  2026-06-09T02:21:18.605Z  note: using database URL postgresql://root@[fd00:1122:3344:101::5]:32221,[fd00:1122:3344:101::6]:32221,[fd00:1122:3344:101::7]:32221,[fd00:1122:3344:101::4]:32221,[fd00:1122:3344:101::3]:32221/omicron?sslmode=disable
1856  2026-06-09T02:21:18.630Z  note: database schema version matches expected (264.0.0)
1857  2026-06-09T02:21:21.517Z  Error: Internal Error: failed runtime check: "collections.len() == 1"
1858  2026-06-09T02:21:21.542Z  + ACTUAL_ZPOOL_COUNT='       0'
1859  2026-06-09T02:21:21.542Z  + _exit_trap
1860  2026-06-09T02:21:21.543Z  + local status=1
1861  2026-06-09T02:21:21.543Z  + set +o errexit
1862  2026-06-09T02:21:21.543Z  + [[ x != \x ]]
1863  2026-06-09T02:21:21.543Z  + pfexec svcadm -Z disable -s cron
1864  2026-06-09T02:21:24.330Z  + [[ 1 -eq 0 ]]
1865  2026-06-09T02:21:24.330Z  + set -o xtrace
1866  2026-06-09T02:21:24.330Z  + banner evidence

i've not yet looked at what might have collections.len() == 1'd to have an assertion failure, but i cannot imagine this is related to a new CPU platform...

[iximeow]

this assert was presumably the one in nexus/db-queries/src/db/datastore/inventory.rs:

        // Read the top-level collection metadata last. We do this at the end
        // (rather than the beginning) so that if a concurrent delete operation
        // has started, we will observe that the top-level collection record is
        // missing and return an error. This prevents returning a partially-torn
        // inventory collection where child rows have been deleted but we still
        // return an incomplete result.
        //
        // The inventory insert and delete operations are transactional, so if
        // this read succeeds, we know the collection exists and hasn't been
        // deleted.
        let (time_started, time_done, collector) = {
            use nexus_db_schema::schema::inv_collection::dsl;

            let collections = dsl::inv_collection
                .filter(dsl::id.eq(db_id))
                .limit(2)
                .select(InvCollection::as_select())
                .load_async(&*conn)
                .await
                .map_err(|e| {
                    public_error_from_diesel(e, ErrorHandler::Server)
                })?; 
            bail_unless!(collections.len() == 1);
            let collection = collections.into_iter().next().unwrap();
            (    
                collection.time_started,
                collection.time_done,
                collection.collector,
            )    
        };

[jgallagher]

I agree that's the error we hit - IIUC the only way for this bail_unless!() to fire is if the inventory collection was pruned while it was being read. That's more or less a normal "inherently racy" failure mode that the client needs to retry, but I'm not sure where that retry ought to happen. The deploy script failed trying to run omdb db zpool list, so the levels to choose from are:

1. The deploy script could retry the omdb command
2. omdb could retry if inventory_get_latest_collection() fails
3. inventory_get_latest_collection() could retry if inventory_collection_read() fails

If we expect the race here to be rare, maybe it should be 3? inventory_get_latest_collection() reads the latest ID then tries to read the collection, all without transactions, so arguably it should do its own optimistic concurrency and retry on specifically the error of "the collection that was the latest was deleted while you were reading it and now there's a new latest"?

[davepacheco]

Makes sense to me. Seems like the code above should return a 404 in this case.

See also: #8336.