[Vedaant-J]

Series: kumo-engine MLOps series: every snapshot, training table, model and prediction run carries a self-contained _provenance.json; with kumo.mlops(...) stores each artifact under one artifact root, tracks fits and prediction runs in MLflow or Snowflake and registers models; volumes and stages can hold files; prediction runs are monitored for drift and realized performance.
Stack: #53 → #112 → #50 → #60 → #64; #114 on #112; #115 on #64
Merge order: one linear stack; merge bottom to top (#53 first); #114 sits on #112 and merges any time after it; #115 sits on #64 and merges after it

Summary

This PR makes TrainingResult.holdout_path the holdout of the model that was published, or None when that model has no holdout. The answer no longer depends on whether explanations are on.

The holdout is the test-split predictions fit() writes to test_df.parquet. holdout_path points at that file and holdout_df() reads it. Teams use it to check the calibration and score distribution of the model they are about to ship.

One fit can train up to three models from the best trial, and the bundle ships the one trained on the most data:

Fit mode	Trained on	Has a holdout
fit	train	yes
refit_trainval (training_job.refit_trainval)	train + val	yes
refit_full (training_job.refit_full)	train + val + test	no: it saw the test split

Usage

plan = ModelPlan()
plan.training_job.refit_trainval = True
result = kumo.Trainer(model_plan=plan).fit(training_table, snapshot=snapshot)

result.holdout_path   # .../version_1_refit_trainval/test_df.parquet: the shipped model's
result.holdout_df()   # its test-split predictions

With refit_full on, holdout_path is None, holdout_df() raises FileNotFoundError, and the log says: The published model is refit on the full data, including the test split, so it has no holdout predictions.

Production questions answered

• Are the holdout predictions I am checking from the model that shipped? Yes, always.
• Does turning explanations on or off change which holdout I get? No.
• Why is there no holdout for my refit_full model? It trained on the test split; the log says so.

How it works

fit() asks kumo-core which fit mode the bundle publishes (Trainer.published_fit_mode(): refit_full, else refit_trainval, else fit) and reads test_df.parquet only from that model's run directory. For refit_full it returns None. Prediction and the holdout lookup share this one definition of the published model.

Run directory of a CPU fit on a small synthetic graph (users, items, orders) with both refits on; the best trial is version_1:

<scratch>/trainers/<trainer>/runs/<run>/
  published-model/                     # the bundle; holds the refit_full model
  models/<run>/automl/
    version_0/  version_2/  version_3/  # other trials
    version_1/                         # best trial, fit mode 'fit'
      test_df.parquet  test_df_meta.json  test_result.csv  checkpoint/  ...
    version_1_refit_trainval/          # train + val, evaluated on test
      test_df.parquet  test_df_meta.json  test_result.csv  checkpoint/  ...
    version_1_refit_full/              # train + val + test, never evaluated
      checkpoint/  tensorboard/  train_progress.json  train_result.json

Which file holdout_path returns, before and after:

refit_trainval	refit_full	explain	before	after
no	no	off/on	version_N/	version_N/
yes	no	off/on	version_N_refit_trainval/	version_N_refit_trainval/
no	yes	off	None	None
no	yes	on	version_N/ (not the shipped model)	None
yes	yes	off	None	None
yes	yes	on	version_N_refit_trainval/ (not the shipped model)	None

Before, the lookup followed whichever fit mode the trainer object was last left in, and explain() changed that mode as a side effect. kumo-core gains Trainer.published_fit_mode() and an optional fit_mode argument on Trainer.get_log_dir, so the lookup reads another mode's directory without changing the trainer or creating empty directories.

Back-compat

Only the choice of file changes. No fields, checks or errors are added, and the run directory layout is unchanged.

Situation	Behaviour
No refit, or refit_trainval only	Same holdout_path as before, explanations on or off.
refit_full, explanations off	None, as before.
refit_full, explanations on	None. It used to return an earlier model's holdout without saying so.
No holdout written (e.g. ranking tasks)	None, as before.
Result of Trainer.load()	None, as before.

Testing

• test_holdout_df.py: real CPU fits with no refit (file under version_N/), refit_trainval with explanations on (under version_N_refit_trainval/), and both refits (None); the holdout is never inside the bundle.
• test_training_explanations.py, test_semantic_carry_forward.py: fake trainers answer published_fit_mode() and get_log_dir(fit_mode=).
• kumo-core test_artifacts.py, training and checkpoint suites pass with the new get_log_dir argument.
• The three engine files above: 35 passed on CPU.

Out of scope

• A holdout for a refit_full model. The fit and refit_trainval test predictions stay in the run directory shown above; they are not returned as the shipped model's.

Change map: see the kumo-pr-change-map comment.

🤖 Generated with Claude Code

[Vedaant-J]

Change map

File	Function / scope	What changed	Why
packages/kumo-core/src/kumo_core/train/trainer.py	new: published_fit_mode; get_log_dir	Names the fit mode the bundle publishes; get_log_dir takes an optional fit_mode	Callers find the published model's directory
packages/kumo-core/tests/test_artifacts.py	fake SourceTrainer	Exposes published_fit_mode	Matches the fit-mode setter's new call
packages/kumo-engine/kumo/engine/trainer/trainer.py	new: _published_holdout; fit, TrainingResult.holdout_path	Reads the published fit mode's holdout, or None for refit_full	Holdout matches the published model
packages/kumo-engine/test/test_holdout_df.py	_fit; new test_holdout_is_the_published_models	Real fits for no refit, refit_trainval with explain, and both refits	Covers each published fit mode
packages/kumo-engine/test/test_semantic_carry_forward.py	_best_trainer fake	Answers published_fit_mode() and get_log_dir(fit_mode)	Matches the trainer's new calls
packages/kumo-engine/test/test_training_explanations.py	fake best trainer	Answers published_fit_mode() and get_log_dir(fit_mode)	Matches the trainer's new calls

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

• Last reviewed commit: ea7119d
• Latest commit, not reviewed: 9d82a62

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review.

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 02f9dcd3-edf8-40f6-b115-ff21b676c041

📥 Commits

Reviewing files that changed from the base of the PR and between 0c37212 and ea7119d.

📒 Files selected for processing: 5

• packages/kumo-core/src/kumo_core/train/trainer.py
• packages/kumo-engine/kumo/engine/trainer/trainer.py
• packages/kumo-engine/test/test_holdout_df.py
• packages/kumo-engine/test/test_semantic_carry_forward.py
• packages/kumo-engine/test/test_training_explanations.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.

Walkthrough

Trainer now resolves the highest completed fit mode through published_fit_mode. get_log_dir can use an explicit fit mode. Trainer.fit uses the published mode to locate holdout artifacts and returns no holdout path for full-data refits. Tests cover standard fitting, train-validation refitting, full-data refitting, explanation settings, path naming, and updated trainer interfaces.

Sequence Diagram(s)

sequenceDiagram
  participant Trainer.fit
  participant published_fit_mode
  participant _published_holdout
  participant Holdout artifacts
  Trainer.fit->>published_fit_mode: determine the completed fit mode
  published_fit_mode-->>Trainer.fit: return the published mode
  Trainer.fit->>_published_holdout: request holdout metadata
  _published_holdout->>Holdout artifacts: locate artifacts for the published mode
  Holdout artifacts-->>_published_holdout: return artifact metadata or no artifact
  _published_holdout-->>Trainer.fit: return holdout metadata

Loading

Merge Risk: Minimal · previous review ea711

Holdout selection matches the published model in the inspected paths. No issue identified here prevents merging after normal checks.

Pre-merge checks: 2 · 1

Failed checks: 1 inconclusive

Check name	Status	Explanation	Resolution
Change Map Design Review		The required PR conversation comment beginning with <!-- kumo-pr-change-map --> is not available. Repository search found only the review documentation reference, and the supplied PR description is …	Provide the complete PR conversation comment beginning with <!-- kumo-pr-change-map -->, including its linked problem, inspected behavior, proposed changes, verification, and stopping condition.

Passed checks: 2 passed

Check name	Status	Explanation
Pr Size		The pull request changes 5 files. This count does not exceed 40.
Public Resource Access		The pull-request diff adds or modifies no external resource reference. The changed lines contain code, documentation, imports, test paths, and local artifact names only; they add no URL, repository lo…

Full details: Public Resource Access

Explanation

The pull-request diff adds or modifies no external resource reference. The changed lines contain code, documentation, imports, test paths, and local artifact names only; they add no URL, repository location, or package source location that requires public-access verification.

Full details: Change Map Design Review

Explanation

The required PR conversation comment beginning with <!-- kumo-pr-change-map --> is not available. Repository search found only the review documentation reference, and the supplied PR description is explicitly truncated. The change-map design cannot be assessed for necessity, abstraction layer, stopping condition, and focused verification without that comment.

Resolution

Provide the complete PR conversation comment beginning with <!-- kumo-pr-change-map -->, including its linked problem, inspected behavior, proposed changes, verification, and stopping condition.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[coderabbitai]

🧹 Nitpick comments: 1

packages/kumo-engine/test/test_holdout_df.py: 1

112-125: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Cover explanation processing with refit_trainval.

The production path runs best_trainer.explain(...) before _published_holdout(...). The matrix does not combine explain=True with refit_trainval=True, so a regression in mode restoration or holdout selection for this path can pass. Add both reachable refit_trainval explanation cases at this matrix site.

Suggested fix

 @pytest.mark.parametrize('refit_trainval,refit_full,explain,model', [
     (False, False, False, 'fit'),
     (True, True, False, 'refit_trainval'),
+    (True, False, True, 'refit_trainval'),
+    (True, True, True, 'refit_trainval'),
     (False, True, True, 'fit'),
 ])

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA/kumo-engine/.coderabbit.yaml
• Review profile: QUIET
• Plan: Enterprise
• Run ID: b9ce2f06-0cec-4ff9-9845-51a2fb62bc8d

📥 Commits

Reviewing files that changed from the base of the PR and between 3f8cf56 and ab2e389.

📒 Files selected for processing: 1

• packages/kumo-engine/kumo/engine/trainer/trainer.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 5 remain after this review.

[Victsm]

Re-reviewed 86fa3c6. The bot’s requested train/validation-refit + explanations regression is now covered. Holdout selection uses the same published-model priority as prediction, without changing the trainer’s active mode; a full-data refit correctly reports no holdout. No remaining correctness blocker found.

The extra full-refit + explanations matrix case suggested by the bot remains optional coverage; it does not block this change.

GitHub merge gates remain separate: the PR change-table validation is red, and the config/connector test failure is from two missing BigQuery permission-test identities.

Local validation of the exact Python head: all 11 holdout tests passed, including real fit/refit and explanation runs, alongside 36 passing semantic-carry-forward/explanation tests. Used an isolated compatible DiskGraph binary; initial native-version/PATH setup failures are retained in the local logs.