Skip to content

fix: forward tolerance flags from get_jumpstart_configs - #6136

Open
evakravi wants to merge 2 commits into
aws:master-v2from
evakravi:fix/jumpstart-configs-tolerance-v2
Open

fix: forward tolerance flags from get_jumpstart_configs#6136
evakravi wants to merge 2 commits into
aws:master-v2from
evakravi:fix/jumpstart-configs-tolerance-v2

Conversation

@evakravi

@evakravi evakravi commented Jul 31, 2026

Copy link
Copy Markdown
Member

Issue #, if available: #6130

Description of changes:

Problem

tolerate_vulnerable_model=True and tolerate_deprecated_model=True do not work on JumpStartModel. Passing them for a flagged model still raises, so the object cannot be constructed and deploy is unreachable. The flags are the documented escape hatch for exactly this case, and there is no supported way to reach a flagged model without them.

JumpStartModel.__init__ honors both flags through every lookup and then discards them on its final statement. That statement calls get_jumpstart_configs, which accepted neither flag. It called verify_model_region_and_return_specs without them, so the callee fell back to its False defaults and re-ran the model gate that the caller had just asked to skip.

The failure is also hard to attribute. The traceback points into utils.py inside the gate rather than at anything the caller passed, and it fires after super().__init__() has already succeeded.

This blocks any release pipeline that deploys JumpStart models for validation. Such a pipeline has to deploy a model that is flagged, precisely because its job is to test that model.

Solution

Add tolerate_vulnerable_model and tolerate_deprecated_model to get_jumpstart_configs and forward both to verify_model_region_and_return_specs. Both default to False, so every existing caller keeps its current behavior and the gate still fires by default.

JumpStartModel.__init__ then passes the two attributes it already set earlier in the same constructor, so no new plumbing is needed. A flagged model now resolves empty configs and construction completes, which is the same outcome the flags already produce elsewhere. The returned configs feed only list_deployment_configs and the benchmark metrics helpers, so no other path degrades.

JumpStartEstimator.list_training_configs had the identical gap and is fixed by the same signature change. Its docstring already documented raising VulnerableJumpStartModelError, and that behavior is unchanged for a caller that has not opted into tolerance.

The same gap exists on master in sagemaker-core, and is fixed by #6137. This PR is the 2.x half.

Testing done:

Five tests in TestConfigs. Three assert tolerance works for a vulnerable inference model, a deprecated model, and a vulnerable training model. Two assert the gate still raises by default, which is the regression guard for existing callers.

One more in test_estimator.py covers JumpStartEstimator.list_training_configs, which had no test at all. That gap left the estimator call site uncovered and showed up as a small project coverage drop on the first push.

The three tolerance tests fail before the change, on the dropped argument:

E  TypeError: get_jumpstart_configs() got an unexpected keyword argument 'tolerate_vulnerable_model'
E  TypeError: get_jumpstart_configs() got an unexpected keyword argument 'tolerate_deprecated_model'
E  TypeError: get_jumpstart_configs() got an unexpected keyword argument 'tolerate_vulnerable_model'
3 failed, 7 passed

All pass after:

$ python -m pytest tests/unit/sagemaker/jumpstart/test_utils.py::TestConfigs -q
10 passed

No regressions across the JumpStart suite. Baseline master-v2 and this branch fail the same 4 tests, all unrelated to configs and failing before the change:

baseline:     4 failed, 366 passed
this branch:  4 failed, 371 passed

The 4 are test_model_display_benchmark_metrics, test_jumpstart_local_metadata_override_specs_not_exist_both_directories, test_jumpstart_model_specs, and test_inference_configs_parsing.

Lint on the touched files, using the versions pinned in requirements/tox:

$ black --check <touched files>     # 24.3.0: 4 files would be left unchanged
$ flake8 <touched files>            # exit 0, no output
$ pydocstyle <touched src files>    # exit 0, no output
$ pylint --rcfile=.pylintrc <touched src files>   # rated 9.98/10, gate is 9.9

Pylint reports two unnecessary-dunder-call findings in set_deployment_config and set_training_config. Both are pre-existing and outside this diff, which is additions only.

No integration test is included. Reproducing this requires a model whose published specs carry a vulnerability or deprecation flag, which is not something a test can provision, and the fix is a pure argument-forwarding change fully covered by unit tests.

Merge Checklist

General

  • I have read the CONTRIBUTING doc
  • I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
  • I used the commit message format described in CONTRIBUTING
  • I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
  • I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

  • I have added tests that prove my fix is effective or that my feature works (if appropriate)
  • I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
  • I have checked that my tests are not configured for a specific region or account (if appropriate)
  • I have used unique_name_from_base to create resource names in integ tests (if appropriate)
  • If adding any dependency in requirements.txt files, I have spell checked and ensured they exist in PyPi

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

JumpStartModel.__init__ ends with a call to get_jumpstart_configs, which
accepted no tolerate_vulnerable_model or tolerate_deprecated_model argument. It
called verify_model_region_and_return_specs without them, so the callee fell
back to its False defaults and re-ran the model gate. Constructing
JumpStartModel(..., tolerate_vulnerable_model=True) for a flagged model raised
VulnerableJumpStartModelError anyway, which made the flag unusable: the object
could not be built, so deploy was unreachable.

Add both parameters to get_jumpstart_configs, default them to False to keep
current behavior for existing callers, and forward them to
verify_model_region_and_return_specs. Pass the attributes already set earlier in
JumpStartModel.__init__, and do the same in
JumpStartEstimator.list_training_configs, which had the same gap.

---
X-AI-Prompt: Why do vulnerable models get skipped during deploy instead of tolerated?
X-AI-Tool: claude-code
JumpStartEstimator.list_training_configs had no test, so the two forwarded
tolerance arguments were uncovered and codecov reported a project coverage drop.

---
X-AI-Prompt: Can you cover the estimator path so codecov stops complaining?
X-AI-Tool: claude-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant