Skip to content

Add a cost keyword to @testitem and schedule by it - #242

Open
jey wants to merge 1 commit into
JuliaTesting:mainfrom
jey:testitem-cost
Open

Add a cost keyword to @testitem and schedule by it#242
jey wants to merge 1 commit into
JuliaTesting:mainfrom
jey:testitem-cost

Conversation

@jey

@jey jey commented Aug 5, 2026

Copy link
Copy Markdown

Close #241

Adds a cost keyword to @testitem declaring roughly how long the item takes to run, in nominal seconds, and schedules by it: items that declare a cost are claimed before those that don't, most expensive first, so a long item can't start near the end of the run and leave the other workers idle waiting for it.

@testitem "slow integration test" cost=450 begin
    @test long_running_thing()
end

Details:

  • The queue is sorted by (status_when_last_seen, -cost, number); number is unique, so the order is deterministic regardless of sort stability. This also makes the existing failures_first ordering deterministic, which it previously was not. failures_first takes precedence over cost within the key.
  • Items with no declared cost sort after all items that do, in the order they would otherwise have run in. A run with no costs and no cached failure status takes the exact same path as before (unsorted queue, evenly spaced worker starts).
  • When the queue is sorted, workers start at the front rather than at evenly spaced positions, so the most expensive items start immediately.
  • A cost can also be a function of the run configuration, for items whose duration depends on it, e.g. cost=(cfg -> 15 + 450 / cfg.nworker_threads). It receives (; nworkers::Int, nworker_threads::Int) and is called exactly once per item, in the coordinator, before any item runs; the returned number replaces the function, so no user function is ever serialized to a worker. nworker_threads is the thread count items will actually run with (the setting, or Threads.nthreads() when serial).
  • Costs are validated to be finite and non-negative (Bool rejected); a function must return a Real or nothing; errors while evaluating a cost name the test item.

Tests cover the macro parsing/validation, the sort (including ties, precedence with failures_first, and worker starting positions), cost-function resolution, and end-to-end run order via runtests with nworkers 0/1/2. README and docstring updated; version bumped to 1.36.0.

Questions:

  • Happy to rename the keyword if cost isn't the right word.
  • Should there be a run-level switch on runtests to ignore declared costs (keyword + env var, like failures_first)? As is, declaring one cost opts the whole suite into the sorted-queue regime on every run.

Test items that declare a cost are claimed before those that don't, most
expensive first, and workers start at the front of the queue instead of
at evenly spaced positions. A cost is a number of nominal seconds, or a
function of the run configuration called once in the coordinator. The
unique tie-break in the new sort key also makes the `failures_first`
ordering deterministic, which it previously was not.

Co-Authored-By: Claude <noreply@anthropic.com>
@jey
jey marked this pull request as ready for review August 5, 2026 21:48
@jey

jey commented Aug 5, 2026

Copy link
Copy Markdown
Author

Marking this ready for review since the two CI test failures appear to be spurious.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Schedule long-running test items first via a cost keyword on @testitem

1 participant