Skip to content

Judge tuning (3/5): Preserve native token usage - #140

Open
ErlisLushtaku wants to merge 9 commits into
erlislushtaku/feat/judge-tuning-02-neps-configfrom
erlislushtaku/feat/cache-native-usage
Open

ErlisLushtaku wants to merge 9 commits into
erlislushtaku/feat/judge-tuning-02-neps-configfrom
erlislushtaku/feat/cache-native-usage

Conversation

@ErlisLushtaku

@ErlisLushtaku ErlisLushtaku commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

Keeps the token counts the model reports, so judging cost can be computed from what the judge actually generated, including cached judgements.

  • Usage survives inference caching: usage_json is a new column in the completion and judgement caches, restored on cache hits and also written to the meta-eval annotation artifacts together with the new error field.
  • Only tokens are cached, not dollar charges. Prices differ per run, so each run applies its own, and a charge stored in an older row is dropped on read.
  • Existing databases get the column when they are opened. merge_from keeps the stored usage when an incoming row for the same output has none, so syncing an older cache does not erase counts.

The cost objective in #133 needs this. NePS does not own the cache.

Merge order: #138 → #135 → #140 → #133 → #136.

@ErlisLushtaku ErlisLushtaku changed the title WIP: Judge tuning (3/6): Preserve native usage in caches and annotations WIP: Judge tuning (3/5): Preserve native token usage Oct 13, 2026
@ErlisLushtaku ErlisLushtaku changed the title WIP: Judge tuning (3/5): Preserve native token usage Judge tuning (3/5): Preserve native token usage Oct 13, 2026
@ErlisLushtaku
ErlisLushtaku marked this pull request as ready for review October 13, 2026 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant