Skip to content

tabicl_baseline: thread offload_mode/batch_size/disk_offload_dir through - #213

Open
amrit110 wants to merge 2 commits into
mainfrom
tabicl-offload-memory-fix
Open

tabicl_baseline: thread offload_mode/batch_size/disk_offload_dir through#213
amrit110 wants to merge 2 commits into
mainfrom
tabicl-offload-memory-fix

Conversation

@amrit110

Copy link
Copy Markdown
Member

Summary

  • TabICL comparator runs have only ever scored the basic 17-feature panel, never the strong 609-feature panel the GBM baseline uses, because a naive fit at the standard row/estimator settings costs an estimated ~70GB per predict_proba call and reliably OOM-killed the host (three confirmed kills, 2026-08-23).
  • TabICLClassifier already exposes offload_mode/batch_size/disk_offload_dir to manage exactly this memory bottleneck, but none of it was wired through this project's tabicl_baseline.py. This threads those three params through fit_tabicl_baselines -> _fit_one_tabicl -> the classifier, and narrows check_inference_cost's RAM-budget gate to skip only for offload_mode="disk" with a real disk_offload_dir, since that combination changes the actual constraint from resident memory to disk I/O (which the existing GB estimate doesn't model, and shouldn't be trusted to block).
  • Adds docs/tabicl_strong_feature_comparison.md: the first TabICL run on the strong feature panel this unblocks, with bootstrap CIs against the tuned GBM on all 12 MIMIC alert cells, plus an honest account of two failed configurations before a one-model-at-a-time loop completed cleanly.

Test plan

  • uv run ruff check / uv run ruff format --check clean on both changed files
  • uv run mypy odyssey/inference/tabicl_baseline.py clean
  • uv run pytest tests/odyssey/inference/test_tabicl_baseline.py — 20/20 passed locally
  • Empirically validated on the GPU host (odyssey-cbm-a100): all 12 (event, horizon) cells fit and scored without OOM using offload_mode="cpu", n_estimators=1, reduced row cap; see docs/tabicl_strong_feature_comparison.md for full results and caveats

🤖 Generated with Claude Code

Every prior TabICL comparator run in this project scored the basic
17-feature panel, never the strong 609-feature one the GBM baseline uses,
because a naive fit at the standard row/estimator settings costs an
estimated ~70GB per predict_proba call and reliably OOM-killed the host.

TabICLClassifier already exposes offload_mode/batch_size/disk_offload_dir
to manage exactly this (the column-wise embedding tensor is the documented
memory bottleneck), but none of it was wired through. This threads those
three params from fit_tabicl_baselines through to the classifier, and
narrows check_inference_cost's RAM-budget gate to skip only when
offload_mode="disk" with a real disk_offload_dir set, since that
combination changes the actual constraint from resident memory to disk
I/O, which the existing GB estimate does not model.

docs/tabicl_strong_feature_comparison.md documents the first strong-panel
TabICL run this unblocks, including two failed configurations along the
way (a disk-offload I/O stall, and a resident-memory OOM from fitting all
12 models up front) before a one-model-at-a-time loop completed cleanly.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.03%. Comparing base (fd11be2) to head (7408591).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #213   +/-   ##
=======================================
  Coverage   89.03%   89.03%           
=======================================
  Files          57       57           
  Lines        8491     8493    +2     
=======================================
+ Hits         7560     7562    +2     
  Misses        931      931           
Files with missing lines Coverage Δ
odyssey/inference/tabicl_baseline.py 98.03% <100.00%> (+0.03%) ⬆️

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant