Conversation
deeppcs
had a problem deploying
to
manual-approval
September 15, 2026 18:39 — with
GitHub Actions
Error
deeppcs
had a problem deploying
to
manual-approval
September 15, 2026 18:39 — with
GitHub Actions
Error
deeppcs
had a problem deploying
to
manual-approval
September 15, 2026 18:39 — with
GitHub Actions
Error
… with a clear error Recipe-based trainers (SFTTrainer, DPOTrainer, RLVRTrainer) accept a Compute for serverful training, and Compute now carries instance_preferences. Recipes are rendered for one instance type: the device class selects the image and launcher, and the type is validated against the model's allowed-type list. A service-chosen type cannot apply, so instance_preferences is unsupported on this path, as documented. Today the unsupported case surfaces indirectly. With no instance_type set, the allowed-type check reports "Instance type 'None' is not supported" before ModelTrainer.from_recipe can raise its explicit message. _train_serverful_smtj now rejects a preference list up front, before the recipe fetch, with the same wording from_recipe uses. Two existing tests modelled Compute as a bare MagicMock, whose auto-created instance_preferences attribute is truthy; they now pin it to None, as a real Compute without preferences reports.
…t preference fields Compute's validator converts Unassigned() to None on its own fields but not on the nested InstancePreference objects, so every submit with a preference list emitted one PydanticSerializationUnexpectedValue warning per element (for the unset training_plan_arns list) when _to_resource_config() called model_dump(). The request payload was already correct; the noise was the only effect. Both Compute classes now normalise each preference the same way. The serialized ResourceConfig is unchanged. A test asserts _to_resource_config() completes with warnings turned into errors.
deeppcs
force-pushed
the
ip-recipe-guard
branch
from
September 15, 2026 22:17
e6c5885 to
edc851c
Compare
deeppcs
requested a deployment
to
manual-approval
September 15, 2026 22:30 — with
GitHub Actions
Waiting
deeppcs
requested a deployment
to
manual-approval
September 15, 2026 22:30 — with
GitHub Actions
Waiting
deeppcs
requested a deployment
to
manual-approval
September 15, 2026 22:30 — with
GitHub Actions
Waiting
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Two follow-ups to Instance Preferences (#6249), found while verifying the 3.22.0 release end to end against the service.
Recipe trainers reject
instance_preferenceswith a clear error.SFTTrainer,DPOTrainerandRLVRTraineraccept aComputefor serverful training, andComputenow carriesinstance_preferences. Recipes are rendered for one instance type — the device class selects the image and launcher, and the type is validated against the model's allowed-type list — so a service-chosen type cannot apply and the feature is documented as unsupported there. Today that surfaces indirectly: with noinstance_typeset, the allowed-type check reportsInstance type 'None' is not supportedbeforeModelTrainer.from_recipecan raise its explicit message._train_serverful_smtjnow rejects a preference list up front, before the recipe fetch, with the same wordingfrom_recipeuses.Five existing tests model
Computeas a bareMagicMock, whose auto-createdinstance_preferencesattribute is truthy; they now pin it toNone, as a realComputewithout preferences reports (the same treatment #6270 applied fortraining_plan_arn).No pydantic serializer warnings on submit.
Compute's validator convertedUnassigned()toNoneon its own fields but not on the nestedInstancePreferenceobjects, so every submit with a preference list emitted onePydanticSerializationUnexpectedValuewarning per element (for the unsettraining_plan_arnslist) when_to_resource_config()calledmodel_dump(). The request payload was already correct; the noise was the only effect. BothComputeclasses now normalise each preference the same way; the serializedResourceConfigis unchanged.Testing
sagemaker-core(compute configs, processing, service model, shapes extractor), 156sagemaker-train(model_trainer, defaults, base trainer serverful/compute, serverful recipe validation). No new failures across the SFT/DPO/RLVR trainer suites relative tomaster.SFTTraineron the serverful path: the message changes fromInstance type 'None' is not supported. Allowed values: [...]toTraining recipes do not support instance_preferences. ...; the new test also asserts the recipe is never fetched._to_resource_config()with warnings turned into errors and fails without the fix; the serialized payload was compared byte-for-byte before and after.MaxPendingTimeInSeconds=1800) submitted, echoed and completed on a resolved winner against the service in us-west-2; the warning fixed here appeared once per preference on every one of those submits.