Add a cost keyword to @testitem and schedule by it - #242
Open
jey wants to merge 1 commit into
Open
Conversation
Test items that declare a cost are claimed before those that don't, most expensive first, and workers start at the front of the queue instead of at evenly spaced positions. A cost is a number of nominal seconds, or a function of the run configuration called once in the coordinator. The unique tie-break in the new sort key also makes the `failures_first` ordering deterministic, which it previously was not. Co-Authored-By: Claude <noreply@anthropic.com>
jey
marked this pull request as ready for review
August 5, 2026 21:48
Author
|
Marking this ready for review since the two CI test failures appear to be spurious. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close #241
Adds a
costkeyword to@testitemdeclaring roughly how long the item takes to run, in nominal seconds, and schedules by it: items that declare a cost are claimed before those that don't, most expensive first, so a long item can't start near the end of the run and leave the other workers idle waiting for it.Details:
(status_when_last_seen, -cost, number);numberis unique, so the order is deterministic regardless of sort stability. This also makes the existingfailures_firstordering deterministic, which it previously was not.failures_firsttakes precedence over cost within the key.cost=(cfg -> 15 + 450 / cfg.nworker_threads). It receives(; nworkers::Int, nworker_threads::Int)and is called exactly once per item, in the coordinator, before any item runs; the returned number replaces the function, so no user function is ever serialized to a worker.nworker_threadsis the thread count items will actually run with (the setting, orThreads.nthreads()when serial).Boolrejected); a function must return aRealornothing; errors while evaluating a cost name the test item.Tests cover the macro parsing/validation, the sort (including ties, precedence with
failures_first, and worker starting positions), cost-function resolution, and end-to-end run order viaruntestswithnworkers0/1/2. README and docstring updated; version bumped to 1.36.0.Questions:
costisn't the right word.runteststo ignore declared costs (keyword + env var, likefailures_first)? As is, declaring one cost opts the whole suite into the sorted-queue regime on every run.