[Manushpm8]

The client could only reach a model over HTTP: a NIM, or a Databricks serving endpoint. Someone who already has structured-data-models loaded in a notebook had to call its tensor API directly, and then use a different API again once the model was hosted. SDMClient.in_process() runs the model in the caller's own Python process behind the same calls.

import sdm
from sdm_client import KumoRelational, SDMClient

with SDMClient.in_process(sdm.models.KumoRelational()) as client:
    result = client.predict(request, model=KumoRelational())

With no model passed, each one is loaded from its published weights on Hugging Face the first time it is asked for, on CUDA when available, else the CPU.

How it works

The client prepares every request the same way, whichever backend answers it: it checks the request against the model, and the model's adapter writes it out as the contract's request, meaning the task, schema, context, rows to score, outputs and inference settings, as Python data. It then hands that to a Backend (sdm_client/core/backend.py):

• NimBackend sends it to a NIM and reads the answer back: encode, HTTP, decode. It wraps the existing HTTP and Databricks transports, and owns what only a NIM needs, such as letting the adapter explain a NIM's refusal, and reading a 404 on the session routes as "no sessions here".
• InProcessBackend runs the model on it directly, with no JSON and no HTTP. It returns the client's own PredictionItems, which go through the same adapter step a NIM's decoded answer does to be put under the caller's keys. No response is built or parsed.

Sessions are backend operations too: create_session, session_predict and delete_session. So predictions, fit() sessions, rankings and forecasts work unchanged, and nothing above the backend knows where the model runs.

In process, the context is fitted as the hosted deployments fit it: seed 42, one estimator, and on a GPU their autocast precision, on a shallow copy of the model so a passed model is never disturbed. A request the model cannot answer is refused with a StructuredDataError whose code says why, from the codes a NIM uses, such as UNSUPPORTED_OUTPUT. It carries no HTTP status, since nothing was sent, and NimRequestError stays the NIM's.

What the in-process backend checks is what the model needs to read a request as it is stated: a column type the model has no reader for, a context with nothing to learn from, a relational task whose entity, anchor, target or joins differ from what the rows say. It does not re-check the request contract the way a NIM's HTTP layer does, field by field. A difference that matters to the model shows up in the shared tests below, and is fixed where it belongs: in the client's preparation, so neither backend is sent it, or in what the model needs to read a request.

Parity is the same tests on both backends

test_every_backend_answers_alike.py runs the same calls against both backends. In process it always runs, with small random-weight models. Against NIMs it runs under -m live_nim with SDM_TABULAR_URL and SDM_RELATIONAL_URL set. The values differ, since the weights do. Both are held to everything else a caller reads:

• which rows come back, under which keys, including an int64 key past 2**53 and a key of several columns;
• the columns, for tabular classification and regression with quantiles, and relational regression, binary and multiclass;
• that a session answers what one call answers;
• how an inference option is read: use_prediction_time spelled 'no', or given as a NumPy boolean, asks for nothing, and a value that cannot be read as its type is refused;
• the refusal code for an output the task does not produce, a bare LIST_DISTINCT, an inference option the model does not act on, a text column, a table named instance_table, and rows scored without their key;
• the refusal code for a relational request built by hand through Request.materialized that states another question than its rows: another entity, a task option the model ignores, an undeclared anchor, a target it cannot read, a table missing or undeclared, or a table that does not carry exactly the columns its schema declares.

All 33 pass on each backend, the NIM being the current SDM NIM code served locally on the CPU with the published weights. Running them, and the review rounds over this change, found seven real differences, each confirmed against the NIM and fixed here:

• A predictive query longer than 1,024 characters was refused by every NIM. The engine puts the query text in metadata.source_query for logging, and the contract caps that field at 1,024 characters, so a NIM answered 422 VALIDATION_FAILED. The relational adapter now cuts the logged copy to fit, so both backends answer it. This is a fix for NIM users too, and is in the changelog.
• A table named instance_table collides with the wire's own name for the rows scored, leaving a relationship that joins on a column the table does not declare. A NIM refuses it with INVALID_SCHEMA. In process it was answered, so the backend now refuses a join on an undeclared column, with the same code.
• Inference options of the wrong type were read differently. The tabular adapter forwarded a caller's options as given, and a NIM types them: run_mode: 7 and inference_config: None were refused by a NIM and answered in process, and use_prediction_time: 'no' the reverse. Both adapters now write the options in the contract's types before sending, with the coercion a NIM applies, and refuse a value that cannot be read with INVALID_REQUEST. For a NIM user, such a value is now refused before it is sent, rather than with 422 VALIDATION_FAILED.
• A key of several columns came back with no id in process. A NIM answers it under the JSON of the key's values, which the client hands back as the row's id, so in process now gives the same id.
• Rows scored without their key failed in process with INTERNAL_ERROR, after the model had run. A NIM refuses them with INVALID_SCHEMA, and in process now does too, before the model runs.
• A cell that cannot be read as its type failed in process with INTERNAL_ERROR when it sat in a key column or held an integer past the int64 range. A NIM refuses it with VALIDATION_FAILED, and in process now does too.
• A row with a cell too many or too few was VALIDATION_FAILED in process and is INVALID_SCHEMA from a NIM. In process now gives the NIM's code.
• A table that did not carry exactly its declared columns was read anyway in process: a declared column left out was fitted without, an undeclared one read as nothing, a repeated one read once. A NIM refuses each, and in process now does too, with the NIM's code.
• A session kept the request's own task and schema, both in the in-process backend and in the client's handle, so changing the request after fit() changed the session under it. Both now keep a copy, as a NIM does.

Values match the NIM exactly on the same device

Served the same way, with the same published weights and structured-data-models 0.2.1, the two give identical predictions, probabilities and quantiles: the largest difference is 0.0 across 41 comparisons, with no missing value on either side. Those comparisons cover:

	Cases
kumo-tabular	binary (int labels), multiclass (string labels), regression with quantiles
kumo-relational	count and sum regressions with quantiles, binary, multiclass, a bool column, a forecast
through the client	RANK TOP 3, a session scored twice
real data	Adult, Wine (multiclass and regression), Titanic, Bank Marketing, Students (kumo-tabular); Northwind counts, sums, binary and integer regression (kumo-relational), one-shot and in a session

qa/probe_in_process.py repeats the real-data comparison against any NIM. A value missing on one side only fails it. On the same device it needs no tolerance. With this process on a CPU and the NIM on a GPU, set SDM_IN_PROCESS_TOLERANCE.

On an NVIDIA L4, the NIM with its bfloat16 GPU profile and in-process on the same GPU give identical answers: 11 of 11 comparisons, one-shot and in a session, with a largest difference of 0.0.

Tests

test_a_model_runs_in_this_process.py covers what only a model in this process has. It uses small random-weight models, so it needs no network and no GPU:

• A tabular answer equals what the model returns when called directly through its own tensor API, for int labels, string labels and regression. This is exact equality, not a tolerance.
• Rankings and forecasts are answered in process.
• A refusal is a StructuredDataError with a code, never a NimRequestError, and has no status: for what a holdout leaves behind, a table the model cannot read, and a task the model was not loaded for.
• A session keeps its own copy of what it pinned.
• A context the caller fitted on their own model survives.
• A model is loaded once, on first use, on the default device. A load that fails is MODEL_NOT_READY, with the original error as its cause.
• Configuration mistakes are refused when the client is opened, and a model left in training mode is named.
• Without the extra, in_process() raises MissingExtraError naming in-process. This is checked in a fresh interpreter with structured-data-models, PyTorch and PyArrow all absent.

Apart from the long-query fix and the typing of inference options, the NIM path behaves as before. Its tests are unchanged too, except that each one handing the client a fake transport now hands it to NimBackend. CI runs the in-process tests in the engine job, which installs CPU PyTorch and the in-process extra. The other jobs skip them.

Locally: the client suite (2007 passed) and the root suite (255 passed) pass. ruff, mypy and hygiene are clean. Against local NIMs, the existing live suites (31 passed) and both new test files on both backends (82 passed) pass.

Worth a look

• CI installs the wheels it built. structured-data-models-client 1.0.0 reached PyPI while this was in review. After that, --find-links dist with ==1.0.0 resolved the published release instead of the wheel CI had just built. installed-wheel, sbom and the build dry-run now name each wheel file, and installed-wheel asserts that all four distributions came from dist/. This affects every pull request against the client from now on.
• The hygiene rule changes. It refused the bare name structured-data-models as an internal build a reader cannot look up. That package is now public, and it is what the new extra installs, so the rule now covers only the internal build name it was written for.
• Text, multicategorical and sequence columns are refused, as the current NIM refuses them. The client's warning at materialize() still says those columns are dropped, which is now wrong against both. That is left for a follow-up.
• The extra needs Python 3.11, because structured-data-models does. It carries a python_version >= '3.11' marker, without which the workspace lock cannot resolve, so on 3.10 the extra installs nothing and in_process() says it needs Python 3.11.
• uv.lock grows. It resolves every extra, so it gains structured-data-models, PyTorch, and PyTorch's CUDA wheels for Linux. Nothing already in it moves. Main's own lock is already behind its pyproject since Build the relational engine for macOS, Windows and 3.14 #42 added Python 3.14; this change leaves that to main rather than mixing a full relock in here.
• A deployment's protective limits do not apply in process. These are the limits a NIM sets per deployment through TFM_MAX_*, such as string-cell bytes and distinct values per column. They protect a shared server. The limits this client checks before sending still apply, and docs/reference/limits.md says so.
• Inside a session, a refusal keeps its own code. A NIM files a refusal while pinning or scoring against a session under SESSION_CREATE_VALIDATION_FAILED or SESSION_PREDICT_SCHEMA_MISMATCH. In process, it keeps the code of the reason, such as TASK_SCHEMA_MISMATCH. Reproducing the NIM's session codes would be the route imitation this change removes, and the quickstart says so.
• A request is checked for what the model needs to read it, not against every rule of the contract. A name longer than the contract's 1,024 characters, or a key the contract does not name at the top level of a payload passed to Request.materialized, is answered in process, where a NIM's request model refuses it with VALIDATION_FAILED. Re-checking each contract format in process is the imitation this change removes. The quickstart says so.

Follow-ups, not in this change:

• A table named instance_table should be refused when the request is gathered, with a message naming the table, rather than by whichever backend reads it.
• request.py says rows to score may leave out their key, and test_entity_id_parity.py tests that by normalizing a made-up response. A NIM refuses such rows with 422 INVALID_SCHEMA, and so does in process. That predates this change, so it is not changed here.

CPU against GPU. On a CPU, in-process computes at full precision and draws its random numbers on the CPU. A GPU-hosted NIM does neither, so the two can differ by more than rounding: on the L4, a class probability moved by 0.16 and a regression by 0.41. The quickstart gives that figure. On the same device, the two are identical.

🤖 Generated with Claude Code

[adi-kumo]

packages/structured-data-models-client/src/sdm_client/in_process/relational.py:61

P2: Mismatched entity_table_names is answered instead of refused

entity_table_names is never checked, so an envelope naming one entity but linking another is scored instead of refused as a NIM does. Validate it against the single task link before building tensors, with the NIM's error codes.

[Manushpm8]

Fixed in e38b57a. In-process now makes the NIM's relational checks in the NIM's order: related tables present, the target on the instance table, the inference options, one link, the task options, exactly one entity table that is the linked one and not instance_table, a declared timestamp anchor, readable instance columns and a representable target dtype. Each refusal uses the NIM's code and message. I measured these against the NIM on hand-written variations, one-shot and in a session, and all of them match. The same pass found unknown keys being accepted where the NIM's request models forbid them, so those are refused now too, in the NIM's order.

[adi-kumo]

qa/probe_in_process.py:42

P2: Parity probe reports agreement when values are missing

np.nanmax drops NaN gaps, so a value missing on one side counts as a match, and an all-NaN column reports 0.0. Fail when the NaN masks differ, then compare only the finite values.

[Manushpm8]

Fixed in e38b57a. A value missing on one side now fails the comparison. The largest gap is taken only over values that differ, so two missing values agree and an infinity against a number fails. I re-ran the CPU comparison with the fix: still 0.0 everywhere, with no missing values on either side.

[JGagacheva]

Six comments inline. None is blocking. The output-option check, the lock and the hygiene regex matter most.

[JGagacheva]

health_ready() calls _require_open(), so after close() it raises INVALID_CONFIGURATION instead of returning False. A caller polling health as a bool gets an exception. Please return not self._closed here.

[Manushpm8]

Fixed in 638f15f. health_ready() returns not self._closed, so a caller polling it as a boolean gets one either way. Test: test_a_closed_client_says_it_is_not_ready_rather_than_raising, which raises INVALID_CONFIGURATION without the fix.

[JGagacheva]

prediction_statistic and quantile_levels are checked here in _items, which runs after _fit and _run (lines 129-130, and 183-184 for sessions). A typo like prediction_statistic='mode' costs a full fit and predict, possibly minutes on a GPU, before it's refused. Please check them in _fields, or anywhere before the model runs.

[Manushpm8]

Fixed in 638f15f. Both are now checked in _fields, which runs before _fit and _run on the one-shot and the session path alike.

One thing the fix turned up: quantile_levels could never reach the backend at all. Both adapters rebuild the output section when they prepare a call — _output_for in adapters/relational.py returns {"fields": ..., "prediction_statistic": ...} and the tabular adapter sets {"fields": ...} — so a caller has no way to ask for levels other than the nine defaults, even through Request.materialized. That makes the levels check unreachable in both places. I kept it next to the statistic because _levels is what feeds the answer, but the test covers only prediction_statistic, which is the half that is reachable. The unreachable half looks like a real gap against the contract and I have left it out of this change; worth its own issue.

Test: test_an_output_option_is_refused_before_the_model_runs, which asserts the model was never fitted. Without the fix it returns INTERNAL_ERROR after the run.

[JGagacheva]

_running holds self._lock for the whole fit and predict. Every call on one client is serialized, including delete_session and close(). client.py:14 says a client can be used "from several threads". With in-process, a second thread waits for the first thread's whole model run. Please either hold the lock only around the session and model maps, or document that an in-process client runs one call at a time.

[Manushpm8]

Fixed in 638f15f and f8ec557. There are now two locks: _state (an RLock) over the session and model maps and the closed flag, and _runs (a Lock) over model execution. delete_session, close and health_ready take _state only, so they no longer wait for a model. Order is always _runs then _state, never the reverse.

The split had two consequences I had to fix in f8ec557, both found in review:

• close() could now land mid-call, and the call would then write a session, or a loaded model, back into state that closing had just released. Both writes recheck that the client is open.
• _serving held _state across _loaded(), which fetches published weights, so the first prediction still blocked everything. The load happens outside the lock now; _runs already makes it happen once.

The class docstring states the contract: one model runs at a time, and letting go of what the client holds is not a model run. Tests: test_releasing_a_session_does_not_wait_for_a_model_to_finish and test_closing_beats_a_call_that_was_already_running.

[JGagacheva]

A model the client wasn't opened with is refused as INVALID_SCHEMA, but nothing is wrong with the schema. Code that branches on error.code will treat it as a schema problem. Please use the code a NIM uses for an unserved model, or INVALID_CONFIGURATION.

[Manushpm8]

Fixed in 638f15f. It is INVALID_CONFIGURATION now, which is what the two neighbouring refusals at that boundary use — a closed client, and a model left in training mode. UNKNOWN_MODEL is taken, and means a model id the contract does not carry at all, which is a different thing. Test: test_a_model_this_client_was_not_opened_with_is_a_configuration.

[JGagacheva]

The deleted structured-data-models regex did two things. It refused the bare name, and it refused every structured-data-models-<suffix> that isn't one of the five public distributions, such as structured-data-models-nim. Removing it to allow the bare name also drops the second guard. Please keep a pattern for the unknown suffixes, e.g. \bstructured-data-models-(?!(?:client|api|pql|connectors|relational-engine)\b)\w.

[Manushpm8]

Fixed in 638f15f, and tightened in f8ec557. The pattern is back as \bstructured-data-models-(?!(?:client|api|pql|connectors|relational-engine)(?![\w-]))\w.

I used (?![\w-]) rather than \b after the suffix, because \b matches before a hyphen and so would have exempted structured-data-models-client-nim and structured-data-models-api-internal, whose full suffixes are not published either. The original pattern had the same hole.

Test: test_only_a_published_distribution_may_be_named in tests/test_release_contract.py, parametrized over the bare name, the five published suffixes and two that are not. It assembles the names rather than spelling them, because spelling one is the thing the rule refuses — which the hygiene check caught when I first wrote it literally.

[JGagacheva]

np.bool_ and np.integer aren't bool, int or float, so np.False_ or np.int64(0) for use_prediction_time is refused with INVALID_REQUEST. The tabular path only works because json_ready converts first. A hand-built relational payload doesn't get that conversion. Please accept numbers.Integral and np.bool_, or convert before this check.

[Manushpm8]

Fixed in 638f15f, with a follow-up in f8ec557. _boolean reads a NumPy scalar by the Python value it holds, so np.False_ and np.int64(0) are read rather than refused.

Review caught that the first version recursed: np.longdouble(1).item() returns another np.longdouble, so _boolean(np.longdouble(1)) ran until the stack did. A scalar holding no Python value is now refused like any other non-boolean.

Tests: test_a_numpy_boolean_built_by_hand_is_read_as_the_boolean_it_holds on both backends through a hand-built relational payload, which is the path with no json_ready conversion, and test_a_numpy_scalar_holding_no_python_value_is_refused_not_recursed_on, which raises RecursionError without the second fix.

[am-porter]

adapter.answered() is called without request_id here and in session_predict(), so in-process results always have request_id == "". For batched calls, metadata["batch_request_ids"] becomes a list of empty strings even though each prepared payload has a request ID. Could both paths pass through the payload's request ID and add a result-level test?

[Manushpm8]

Fixed in 638f15f. Both paths pass request_id=_request_id(payload), read from the payload's own metadata.request_id, so a batch lists real ids.

The first test I wrote for it only asserted the id was non-empty, which a constant would have passed, and never scored a session — fixed in f8ec557. It now pins the adapter's id generator and asserts the answer carries exactly that, on the one-shot path and on a session scoring call, where the id is the scoring call's and not the one the context was pinned with.

[am-porter]

This checks the declared column names but not the length of each rows entry. In a hand-built Request.materialized() payload, a short row raises IndexError in values() and becomes INTERNAL_ERROR; an overlong row is silently ignored. Could this reject rows whose cell count differs from len(columns) with a validation error?

[Manushpm8]

Fixed in 638f15f. check_columns refuses a row whose cell count differs from the number of columns, with VALIDATION_FAILED, which is what the repeated-column check beside it uses. It stops at the first such row rather than scanning the whole table.

Tests: two more cases in test_a_request_built_by_hand_is_refused_where_it_states_another_question, run on both backends. Without the fix the short row gives INTERNAL_ERROR and the overlong row is answered silently, which is exactly the pair you described.

[JGagacheva]

Do fit and predict here need torch.inference_mode() (or no_grad())? Only autocast is applied, so if the loaded weights have requires_grad=True (the default for an nn.Module), every call builds an autograd graph and keeps activations for backward. On large relational contexts on a GPU that can run out of memory, or run much slower than a NIM. The .detach() in answers suggests the outputs do carry grad.

[Manushpm8]

No change needed: sdm does this itself. fit runs under inference_mode("no_grad") and predict under torch.inference_mode() (sdm/models/base.py), unless the model is training, which _serving refuses, or a callback needs grad, and none is passed. Measured: the raw output has requires_grad=False and is_inference()=True while the weights have requires_grad=True. The .detach() matches the NIM's output_array.

[JGagacheva]

copy.copy is shallow: the copy shares the module's _parameters, _buffers and _modules dicts with the original and with every other session. If fit stores its context as a buffer or submodule (through nn.Module.__setattr__), fitting session B changes session A and the caller's own model, which breaks the docstring's promise that a session never disturbs another. Does sdm's fit only assign plain attributes?

[Manushpm8]

No change needed: yes. fit assigns only self._cache (sdm/models/base.py:329, cleared at :492), a Cache mapping rather than a module, parameter or buffer, so it lands in the copy's own __dict__. The one shared mutable is _transfer_streams, a per-device CUDA stream cache, which is safe to share because runs are serialized. Measured: fitting session B on targets scaled by −10 left session A bit-identical, and the original's _cache stayed None.

[JGagacheva]

If a hand-built Request.materialized payload lacks instance_id (or a join column), the column lookup raises a bare ValueError after the full fit and run, and _running reports it as INTERNAL_ERROR ("the model failed: ValueError: 'instance_id' is not in list"). Could this be checked up front and refused as INVALID_SCHEMA, as the module docstring promises?

[Manushpm8]

Fixed in 26d0907. Every table must now declare instance_id, refused as INVALID_SCHEMA before the fit, as a NIM's _validate_instance_id_column does. A missing join column was already refused up front; instance_id slipped through because _graph adds it to every join without the schema naming it.

[JGagacheva]

An unknown payload['model'] raises KeyError here, which surfaces as INTERNAL_ERROR ("the model failed: KeyError"). It would be clearer to refuse it as an unsupported model and name the models served.

[Manushpm8]

Fixed in 26d0907. Refused as INVALID_CONFIGURATION, naming the models served, like a model the client was not opened with. Through the client it cannot happen today, since the adapter overwrites payload['model'] from the closed MODELS; it could once a model is added there without an in-process reader, which is what the test simulates.

[JGagacheva]

levels or QUANTILE_LEVELS replaces an explicit empty quantile_levels: [] with the nine defaults, so the caller gets quantiles they didn't ask for. Should an empty list be refused (or return none), as a NIM would?

[Manushpm8]

No change needed: this matches the NIM, which does the same requested.quantile_levels or list(_DEFAULT_QUANTILES) (sdm_nim/container/decoding.py), and the contract sets no minItems. It is also unreachable, since _output_for rebuilds output without quantile_levels. Refusing [] here alone would make the two backends disagree.

[JGagacheva]

For a boolean target with no declared classes, the labels are Python True/False, so str(label) gives 'True'/'False', while a NIM returns 'true'/'false'. to_pandas() then has probability_True in process versus probability_true from a NIM, and code that reads by key breaks when switching backends.

[Manushpm8]

No change needed: a NIM gives 'True'/'False' here too. With no declared classes _wire_labels returns the encoded booleans and decode_classification keys them by str(label) (sdm_nim/container/conversion.py, decoding.py). With declared classes, as the client sends for a PQL binary target, both answer 'true'/'false' and the adapter maps both alike. The live NIM is down (503), so this is from the NIM's main, not a live run.

[JGagacheva]

Five comments inline. None is blocking. Only the use_prediction_time question in relational.py needs an answer before merging.

[JGagacheva]

Key columns skip the error handling that _array gives other columns. _whole calls read_cell outside any try, so an int64 join key holding 'abc' raises a plain ValueError from int(cell), and _running reports it as INTERNAL_ERROR ("the model failed: ValueError…"). The pa.array(..., pa.int64()) call is uncaught too, e.g. for text in a float64 key or a value past the int64 range. Only hand-built Request.materialized payloads reach this. Please refuse these as VALIDATION_FAILED, as a non-key column is.

[Manushpm8]

Fixed in 9efa467. Keys are now read under the same refusal as every other column, so text in an int64 key, or a value past the int64 range, is VALIDATION_FAILED. A non-key int64 past the range had the same gap, since pyarrow raises OverflowError there and nothing caught it, so that is fixed too. A NIM gives VALIDATION_FAILED for all three; I checked them against one, and the parity suite now holds both backends to it.

[JGagacheva]

_running wraps the whole call, including reading the request and adapter.answered(...), so any unexpected client-side exception (a missing payload key, a bug in _named_entities) is reported as "the model failed". Could the message be neutral, or the wrap cover only _fit and _run?

[Manushpm8]

Both, in 9efa467. The wrap now ends before adapter.answered(...), so a fault in the client's own code, _named_entities included, is raised as itself, as it is on the NIM path. What stays wrapped is what a NIM does on its side: reading the request, fitting and running the model, and reading what it returned. The message is now "answering in this process failed: …".

[JGagacheva]

_SESSION_PINNED is ('model', 'task', 'schema', 'context'), so a session's context is always fitted without prediction time, and every session_predict with use_prediction_time=True is refused here. So a relational fit() asked with prediction time can never be scored in process. Does a NIM behave the same way? The parity suite only tests use_prediction_time through tabular predict(), so this case isn't covered on either backend.

[Manushpm8]

Yes, a NIM does the same. SessionCreateRequest has no inference field, so a NIM compiles every session without prediction time, and refuses a prediction that asks for it with INVALID_REQUEST ("use_prediction_time is settled when a session is compiled…"). I checked it against a NIM, and the parity suite now covers it on both backends (9efa467).

[JGagacheva]

In-process sessions never expire. Each keeps a fitted copy with its context cache, possibly in GPU memory, so a FittedModel not used as a context manager holds that until the client closes, where a NIM session times out. One sentence in the quickstart would help notebook users.

[Manushpm8]

Added to the quickstart in 9efa467, beside the note that a deployment's limits don't apply in process: the session holds its fitted context, in GPU memory when the model is on one, until it or the client is closed.

[JGagacheva]

Nit: this repeats the quantile-level check that _check_regression_output now makes before the run, and it can't be reached. Also, people(), graph(), _quiet and the random-weight model fixtures are copied between test_a_model_runs_in_this_process.py and test_every_backend_answers_alike.py; a shared conftest.py would hold them once.

[Manushpm8]

Both done in 9efa467. The check is gone, and the models and tables now live once, in tests/conftest.py and tests/sample_tables.py.

[JGagacheva]

@Manushpm8 I added some comments but I think they're minor cleanup details/checks. I'll approve and let you decide how to handle those

[JGagacheva]

On a GPU, _fit and _run run the model under torch.autocast with bfloat16 (or float16). NumPy has no bfloat16, so if output.numerical comes back in that dtype, .numpy() raises TypeError: Got unsupported ScalarType BFloat16. _running would then report every prediction as INTERNAL_ERROR. CUDA is the default device whenever it's available, so this would hit the default GPU setup.

In structured-data-models 0.2.1, predict() calls transform_output with autocast disabled, but that doesn't guarantee the result is float32. I couldn't confirm the dtype from the source. The PR description says the L4 comparison hasn't been re-run since the rework.

Suggest casting before converting, which is harmless if the output is already float32:

array = np.asarray(output.numerical.detach().float().cpu().numpy())

and re-running the L4 comparison (qa/probe_in_process.py) before merge.

[Manushpm8]

Re-ran it on an NVIDIA L4 at f6dd0da: both NIMs with their shipped bfloat16 GPU profile, and in process on the same GPU (structured-data-models 0.2.1, torch 2.14.1+cu130).

• output.numerical arrives as torch.float32 under bfloat16 autocast, checked on every answer.
• qa/probe_in_process.py: 11 of 11 comparisons identical, largest difference 0 (Adult, Wine, Titanic, Bank Marketing, Students; three Northwind queries, one-shot and in a session).
• This PR's tests on the same machine: 118 passed, and 46 passed under -m live_nim against the GPU NIMs.

A NIM converts its output with the same .detach().cpu().numpy(), so a bfloat16 output would fail there as well. I've left the conversion as it is so the two stay identical, and updated the description.