When will an AutoGluon 1.6.x training DLC be published?
Summary
The newest AutoGluon Deep Learning Container available for SageMaker training is 1.5.0, while the current AutoGluon release on PyPI is 1.6.1. This version gap makes autogluon.cloud unusable for any project that runs 1.6.x locally, because a cloud-trained predictor can neither be produced with 1.6.x-only settings nor loaded afterwards by a 1.6.x installation.
I would like to know whether a 1.6.x training container is planned, and if so, roughly when it is expected to be published.
Environment
|
|
| Local AutoGluon |
1.6.1 |
autogluon.cloud |
0.5.0 |
| boto3 / botocore |
1.43.66 |
| Python |
3.12 |
| Region |
eu-central-1 |
| Instance type |
ml.g4dn.2xlarge |
What latest resolves to
>>> from autogluon.cloud.utils.dlc_utils import retrieve_available_framework_versions, retrieve_latest_framework_version
>>> retrieve_available_framework_versions("training")
['0.3.1', '0.3.2', '0.4.0', '0.4.2', '0.4.3', '0.5.2', '0.6.1', '0.6.2',
'0.7.0', '0.8.2', '1.0.0', '1.1.0', '1.1.1', '1.2.0', '1.3.0', '1.4.0', '1.5.0']
>>> retrieve_latest_framework_version("training")
('1.5.0', ['py312'])The resulting image for a GPU training job:
763104351884.dkr.ecr.eu-central-1.amazonaws.com/autogluon-training:1.5.0-gpu-py312-cu126-ubuntu22.04
There is no 1.6.x tag to pin instead, so framework_version="latest" is already the best available choice.
Concrete problems this causes
1. Presets introduced in 1.6.x are rejected by the 1.5.0 container
A TimeSeriesCloudPredictor.fit(...) call using the experimental preset fails inside the container:
AlgorithmError: ExecuteUserScriptError:
ExitCode 1
ErrorMessage "raise ValueError(f"Preset '{preset_og}' was not found. Valid presets: {sorted(set(valid_presets))}")
ValueError: Preset 'experimental' was not found. Valid presets: ['best', 'best_quality',
'bolt_base', 'bolt_mini', 'bolt_small', 'bolt_tiny', 'bq', 'chronos2', 'chronos2_ensemble',
'chronos2_small', 'fast_training', 'high', 'high_quality', 'hq', 'medium', 'medium_quality', 'mq']"
Command "/usr/local/bin/python train.py", exit code: 1
Comparing the TimeSeries preset registries confirms the gap is exactly the two 1.6.x additions:
local 1.6.1 only : ['experimental', 'experimental_quality']
container 1.5.0 only : []
common : the remaining 17 presets
The failure only surfaces after the instance has been provisioned and the training image downloaded, so the GPU time is already billed before the job exits.
2. A predictor trained in the container cannot be loaded locally
TabularPredictor.load() and TimeSeriesPredictor.load() both default to require_version_match=True, and the underlying check compares versions for exact string equality:
# autogluon/common/utils/utils.py
if version_saved != version_current:
...
if require_version_match:
raise AssertionError(
f"Predictor was created on version {version_saved} but is being loaded "
f"with version {version_current}. ..."
)Since 1.5.0 != 1.6.1, every cloud-trained predictor raises AssertionError on load.
To be clear: I am not asking for this check to be relaxed. I am aware of #5662, where that was proposed and declined, and of the maintainer position stated there — that AutoGluon makes no backwards-compatibility promise for loaded artifacts across any version difference. I accept that. I mention the check only to explain why a version-matched container is the only workable path for us: with the check working as intended, a 1.5.0 container and a 1.6.1 local install cannot be combined at all, and require_version_match=False is exactly the unsupported territory the maintainers warn about.
3. Locally resolved hyperparameters reference a model family absent from 1.5.0
Our pipeline resolves a preset to its hyperparameters dict locally (in order to cap each model's history look-back before submitting the job), then passes that dict explicitly to fit(). Under 1.6.1, best_quality resolves to:
['AutoETS', 'Chronos2', 'DeepAR', 'DirectTabular', 'DynamicOptimizedTheta',
'RecursiveTabular', 'SeasonalNaive', 'TemporalFusionTransformer', 'Toto2']
Toto2 does not exist in 1.5.0. Comparing the two model registries confirms it:
# autogluon.timeseries 1.5.0 (from the published wheel)
'TotoModel' in __all__ -> True
'Toto2Model' in __all__ -> False
'Chronos2Model' in __all__ -> True
# autogluon.timeseries 1.6.1 (local)
['ChronosModel', 'Chronos2Model', 'TotoModel', 'Toto2Model']
So even after switching to a preset name that both versions accept, a run that forwards locally resolved hyperparameters hits a second wall: Chronos2 resolves fine, Toto2 does not. Resolving presets locally and executing them remotely is only safe when both sides run the same AutoGluon version.
Related issues
Questions
- Is a 1.6.x AutoGluon training (and inference) DLC planned, and is there a rough target date?
- Is there a rule of thumb for how long after a PyPI release the corresponding container usually appears? Knowing the typical lag would let us decide whether to pin our local environment to the container version instead of waiting.
Current workaround
We have paused cloud training and run training locally on 1.6.1 until a matching container is available. Pinning the local installation down to 1.5.0 is the alternative we are weighing, so an indication of the expected timeline would directly inform that decision.
When will an AutoGluon 1.6.x training DLC be published?
Summary
The newest AutoGluon Deep Learning Container available for SageMaker training is 1.5.0, while the current AutoGluon release on PyPI is 1.6.1. This version gap makes
autogluon.cloudunusable for any project that runs 1.6.x locally, because a cloud-trained predictor can neither be produced with 1.6.x-only settings nor loaded afterwards by a 1.6.x installation.I would like to know whether a 1.6.x training container is planned, and if so, roughly when it is expected to be published.
Environment
autogluon.cloudeu-central-1ml.g4dn.2xlargeWhat
latestresolves toThe resulting image for a GPU training job:
There is no 1.6.x tag to pin instead, so
framework_version="latest"is already the best available choice.Concrete problems this causes
1. Presets introduced in 1.6.x are rejected by the 1.5.0 container
A
TimeSeriesCloudPredictor.fit(...)call using theexperimentalpreset fails inside the container:Comparing the TimeSeries preset registries confirms the gap is exactly the two 1.6.x additions:
The failure only surfaces after the instance has been provisioned and the training image downloaded, so the GPU time is already billed before the job exits.
2. A predictor trained in the container cannot be loaded locally
TabularPredictor.load()andTimeSeriesPredictor.load()both default torequire_version_match=True, and the underlying check compares versions for exact string equality:Since
1.5.0 != 1.6.1, every cloud-trained predictor raisesAssertionErroron load.To be clear: I am not asking for this check to be relaxed. I am aware of #5662, where that was proposed and declined, and of the maintainer position stated there — that AutoGluon makes no backwards-compatibility promise for loaded artifacts across any version difference. I accept that. I mention the check only to explain why a version-matched container is the only workable path for us: with the check working as intended, a 1.5.0 container and a 1.6.1 local install cannot be combined at all, and
require_version_match=Falseis exactly the unsupported territory the maintainers warn about.3. Locally resolved hyperparameters reference a model family absent from 1.5.0
Our pipeline resolves a preset to its hyperparameters dict locally (in order to cap each model's history look-back before submitting the job), then passes that dict explicitly to
fit(). Under 1.6.1,best_qualityresolves to:Toto2does not exist in 1.5.0. Comparing the two model registries confirms it:So even after switching to a preset name that both versions accept, a run that forwards locally resolved hyperparameters hits a second wall:
Chronos2resolves fine,Toto2does not. Resolving presets locally and executing them remotely is only safe when both sides run the same AutoGluon version.Related issues
wontfix. Referenced above; this issue does not revisit it. Enhancement: Relax strict version check on model load to allow patch version differences autogluon#5662Questions
Current workaround
We have paused cloud training and run training locally on 1.6.1 until a matching container is available. Pinning the local installation down to 1.5.0 is the alternative we are weighing, so an indication of the expected timeline would directly inform that decision.