Repository navigation
Release for security updates after certificates incident #4299
Description
Upgrade to rules_python 2.4.5 to pick up the certifi update, or follow the patch/workarounds in #4299 (comment)
-- rules_python maintainers
A security incident affecting eMudhra intermediate certificates has required removing the emSign root CA certificate from CA bundles such as “certifi”. It’s important to immediately upgrade the software packages below to at least the versions specified and rebuild container images to fetch the latest system certificates.
Failure to do so could result in monster-in-the-middle (MITM) attacks against TLS clients, including python.org and pypi.org .
certifi 2026.10.19
pip 26.2.2
uv 0.13.4
sigstore 4.5.1For all existing virtual environments on your system, please upgrade pip and uv to the latest versions.
In a Python project that uses Bazel, I find an older version of certifi (2026.7.22) in the MODULE.bazel.lock file:
"facts": {
"@@rules_python+//python/extensions:pip.bzl%pip": {
"dist_hashes": {
"https://pypi.org/simple": {
"backports-tarfile": {
"https://files.pythonhosted.org/packages/86/72/cd9b395f25e290e633655a100af28cb253e4393396264a98bd5f5951d50f/backports_tarfile-1.2.0.tar.gz": "sha256:d75e02c268746e1b8144c278978b6e98e85de6ad16f8e4b0844a154557eca991",
"https://files.pythonhosted.org/packages/b9/fa/123043af240e49752f1c4bd24da5053b6bd00cad78c2be53c0d1e8b975bc/backports.tarfile-1.2.0-py3-none-any.whl": "sha256:77e284d754527b01fb1e6fa8a1afe577858ebe4e9dad8919e34c862cb399bc34"
},
"certifi": {
"https://files.pythonhosted.org/packages/0b/a7/71ac2cff56fec219ed242bb11b8efb69fcc4bec75db06fb7bfe35de520e6/certifi-2026.7.22-py3-none-any.whl": "sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775",
"https://files.pythonhosted.org/packages/a3/c2/24167ea9858356b47a87a50d39908bfdb72ceeefe0041586e704e5376b3a/certifi-2026.7.22.tar.gz": "sha256:741e2c3b351ddf169a738da9f2c048608ff7f2c5cc02f1ebc6b118bb090d5d55"
},
....
But I am happy to see that the version update has already been applied in rules_python's main branch: #4297
Is there a new release planned to fix this security issue?
Or, if that release should take some more time, is there a way to use the already modified version, or to define a patch that Bazel can apply locally? Currently the dependency definition looks like this: bazel_dep(name = "rules_python", version = "2.4.2")
Thank you very much!
rickeylevRecommended: Upgrade to rules_python 2.4.4
Upgrade to
rules_python 2.4.4
to pick up both certificate fixes:
-
pip26.2.2andpip-tools7.6.2inpython/private/pypi/deps.bzl
(fixed in2.4.4, PR #4310), which updates thecertifiCA bundle vendored
insidepip(certifi2026.10.19). -
certifi2026.10.19intools/publish/requirements_*.txt(fixed in
2.4.3, PR #4297).
In MODULE.bazel:
bazel_dep(name = "rules_python", version = "2.4.4")If your project tracks MODULE.bazel.lock, refresh it so cached facts reflect
the updated certifi hashes:
bazel mod deps --lockfile_mode=updateNote
Prior to 2.4.4, rules_python was affected in two places:
-
Internal
pip(python/private/pypi/deps.bzl,<= 2.4.3): Pinned
…
Activity
rickeylev commentedon Oct 22, 2026
Looking at what that PR fixes, it looks like the only thing of relevance to users is the tools/publish requirements files. That gets used by, I think its the twine binary used by default for the py_wheel
.publishtarget.As a user, you can specify your own twine binary as an arg. Alternatively you can patch that requirements file in tools/publish. I'll have a my agent post how to do both methods shortly.
#4301 for the patch release
build(deps): bump the uv group across 10 directories with 1 update (#…
rickeylev commentedon Oct 22, 2026
Recommended: Upgrade to
rules_python2.4.4Upgrade to
rules_python2.4.4to pick up both certificate fixes:
pip26.2.2andpip-tools7.6.2inpython/private/pypi/deps.bzl(fixed in
2.4.4, PR fix(pypi): upgrade pip to 26.2.2 and pip-tools to 7.6.2 #4310), which updates thecertifiCA bundle vendoredinside
pip(certifi2026.10.19).certifi2026.10.19intools/publish/requirements_*.txt(fixed in2.4.3, PR build(deps): bump the uv group across 10 directories with 1 update #4297).In
MODULE.bazel:If your project tracks
MODULE.bazel.lock, refresh it so cachedfactsreflectthe updated
certifihashes:Note
Prior to
2.4.4,rules_pythonwas affected in two places:pip(python/private/pypi/deps.bzl,<= 2.4.3): Pinnedpypi__piptopip 26.0.1(which vendorscertifi 2026.01.04). Thisinternal
pipis used whenrules_pythoninvokespipto make networkrequests: locking requirements via
compile_pip_requirements, buildingsdists in
whl_library, or falling back topipto download wheels (forexample, when using
simpleapi_skip). Standard wheel downloads handled byBazel's downloader use the host OS trust store and lockfile SHA-256 hashes.
tools/publish/requirements_*.txt,<= 2.4.2):Pinned
certifi==2026.7.22in therules_python_publish_depshub used by@rules_python//tools/publish:twine(the defaulttwine_binaryforpy_wheel's manual<name>.publishtarget). Becauserules_python'sMODULE.bazelregistersrules_python_publish_depswith thepipextension, Bazel records
certifi-2026.7.22inMODULE.bazel.lockunderfactswhenever thepipextension is evaluated, even if@rules_python//tools/publish:twineis never fetched or built.Workaround 1: Patch
rules_pythonviasingle_version_overrideIf you cannot upgrade to
2.4.4yet, you can patchrules_pythonin placeusing
single_version_override(ensurepatches/BUILD.bazelexists, even ifempty, so the patch labels are valid).
1a. Patch
pipinpython/private/pypi/deps.bzl(for<= 2.4.3)Save
patches/rules-python-pip.patchto upgraderules_python's internalpipto
26.2.2,pip-toolsto7.6.2, and associated internal PyPI toolingdependencies (applies cleanly to
rules_python2.0.0through2.4.3):Note
The full commit (
94a42cae6fdc018ee2df49ac43146fecf4454f20) from PR #4310also adds a new
pypi__typing_extensionsrepository toMODULE.bazel,python/private/pypi/deps.bzl, andpython/private/pypi/pip_compile.bzl.Because
single_version_overridereadsMODULE.bazelfrom the Bazel CentralRegistry before applying patches, new repositories cannot be added to
use_repoviasingle_version_override.The patch below omits
pypi__typing_extensions, which is only required bypip-tools 7.6.2when runningcompile_pip_requirementswith Python< 3.11(on Python
>= 3.11, includingrules_python2.4's default Python 3.14, itis unused). If you use
compile_pip_requirementswith Python 3.10, passextra_deps = ["@pypi//typing_extensions"]tocompile_pip_requirementsorupgrade to
rules_python2.4.4.rules_python-2.4.2-certifi-2026-10-19.patch
Details
Minimal diff (only
pypi__pipandpypi__pip_tools)1b. Patch
certifiintools/publish(for<= 2.4.2; already in2.4.3)(rules_python-2.4.2-certifi-2026-10-19.patch)
This patches
tools/publish/requirements_*.txtinsiderules_pythonbefore thepipmodule extension runs. It both upgrades the defaulttwineenvironmentand replaces
certifi 2026.7.22withcertifi 2026.10.19inMODULE.bazel.lock.Note
The full commit (
711b314bb5809d1ef801b86df9e217052c69f2c6) from PR #4297does not apply cleanly to
2.4.2. The diff below contains the relevant partsand applies cleanly to
2.4.2.Save
patches/rules-python-certifi.patch:2. Apply the patches in
MODULE.bazelFor
rules_python2.4.2(apply both patches; on2.4.3, only//patches:rules-python-pip.patchis needed):3. Refresh
MODULE.bazel.lock(if using a Bazel lockfile)If your project tracks
MODULE.bazel.lock, update it so the cachedfactsreflect
certifi2026.10.19:Workaround 2: Provide Your Own
twinetopy_wheelIf you use
py_wheeland its<name>.publishtarget, you can supplytwinefrom your own locked
pip.parsehub (wheretwineandcertifi>=2026.10.19are locked in your own requirements file) instead of using
@rules_python//tools/publish:twine.Warning
In
py_wheel,twine_binarydefaults toLabel("//tools/publish:twine")when Bzlmod isenabled, and
if twine_binary:is checked beforeelif twine:. Therefore,you must either override
twine_binarydirectly, or settwine_binary = Nonewhen passing
twine.Option A: Override
twine_binarywithpy_console_script_binaryOption B: Pass
twine(py_library) and cleartwine_binary(Note: Workaround 2 only replaces
@rules_python//tools/publish:twineduringwheel publishing; upgrading to
2.4.4or applying Workaround 1 is required toupdate
rules_python's internalpipand removecertifi 2026.7.22fromMODULE.bazel.lock.)aignas commentedon Oct 22, 2026
Related to this, should we use
@pypi//twineso that the override is easier to do? This is similar to what we did forpytest.rickeylev commentedon Oct 22, 2026
Our twine target is just a thin wrapper, right? Like just the entry point generation? If so, yeah, that would make sense. Though, we'd need some way to provide a default implementation. If
@pypigoes back to the user's closure, and they don't have twine, then it'll give an error. Maybe thats OK. It basically pushes the problem of providing dependencies for our tools onto the user and we wouldn't have need to provide the various hard-coded urls for it.stephanretool commentedon Oct 22, 2026
Hi @rickeylev !
If I understand correctly, only upgrading certifi leaves some gaps where rules_python still trusts the compromised certificates when it uses pip. Full disclosure: I'm not familiar with rules_python internals, and I used AI to trace through the code and run the tests below. I may be misreading how things fit together, so feel free to ignore this if it actually is nothing.
python/private/pypi/deps.bzlstill seems to pinpypi__pipto pip 26.0.1. The discuss.python.org post quoted above lists pip 26.2.2 as a required upgrade. A CPython core developer explains why in the same thread:They also say that "Updating pip is the only reliable fix."
pip 26.0.1 vendors certifi 2026.01.04, which still contains
emSign ECC Root CA - G3. So whenever rules_python's pinned pip makes a network request, it still trusts the compromised CAs, even when the host's trust store and the user's own certifi (including the one bumped in rules_python 2.4.3) are up to date.If I understand correctly, rules_python's pip reaches the network in a few places:
compile_pip_requirements: pip-tools resolves and downloads through the pinned pip. This seems the most sensitive case, because with--generate-hashesthe lockfile would record hashes of whatever was served.whl_library: in 2.4.3, pip seems to build sdists with build isolation, so it downloads build dependencies (like setuptools) from the index itself. In a separate test with plain pip, those build-dependency downloads weren't hash-checked even in--require-hashesmode.pip.parse: when a distribution has no URL from the index metadata (for example viasimpleapi_skip),whl_libraryseems to fall back to pip for the download itself.I tried this with a small Bazel 9.3.1 workspace on rules_python 2.4.3, in a Debian trixie container whose OS trust store rejects the eMudhra test host (
dv-g3b-emudhra.we-have-a-problem.com):--index-urlpointing at the test host: pip-tools fetches the page over TLS and only skips it because of its Content-Type.extra_pip_args = ["--no-index", "--no-build-isolation"],may work as a mitigation for backends that are bundled.--index-urlpointing at the test host: the TLS handshake succeeds, and pip only fails with "No matching distribution found".With a
single_version_overridepatch bumpingpypi__pipto 26.2.2, cases 1 and 3 fail withCERTIFICATE_VERIFY_FAILED, as expected. Turns out, with pip 26.2.2, the pinned pip-tools 7.5.3 crashes incompile_pip_requirements. Bumpingpypi__pip_toolsto 7.6.2 fixed that, and resolution against PyPI worked again.The Bazel downloader itself rejected the test host (PKIX path building failed), so the default wheel download path seems unaffected. This only seems to matter where rules_python's own pip goes to the network.
Would it makes sense to ship a fix that bumps
pypi__pipto 26.2.2 (together withpypi__pip_tools7.6.2) make sense? Users with hashed lockfiles are protected for the package files themselves, but I believe cases 1 and 2 aren't covered by hashes. For what it's worth, I haven't tested if the bumps would affect anything else. It seems like rules_python ships a Python 3.9 toolchain still and pip 26.2.2 requires Python >= 10, so there's a chance of some compatibility issues. No fixed 3.9 pip exists as far as I can tell, though.Happy to share the test workspace or the patch.
Thanks for maintaining this!
thomas-ferchau commentedon Oct 22, 2026
@stephanretool
Wow! Maybe you should create a new ticket with this text. I guess your contribution would get lost in this closed ticket.
thomas-ferchau commentedon Oct 22, 2026
@rickeylev
I think it should be 2.4.3
rickeylev commentedon Oct 22, 2026
Argh, pip, forgot about that. Thanks @stephanretool .
fix(pypi): upgrade remaining internal pypi tooling dependencies
fix(pypi): upgrade pip to 26.2.2 and pip-tools to 7.6.2 (bazel-contri…
fix(pypi): upgrade pip to 26.2.2 and pip-tools to 7.6.2 (#4310)
rickeylev commentedon Oct 23, 2026
2.4.4 is released that upgrades pip to 26.2.2
fix(uv): upgrade default uv to 0.13.5 (#4329)
fix(uv): upgrade default uv to 0.13.5 (#4329)
rickeylev commentedon Oct 23, 2026
While talking to some buddies over D&D, one mentioned that uv was also affected by this, so #4329 upgrades the default uv we set as part of the uv extension. It's been released in 2.4.5.
I also published an advisory in our repo: GHSA-f8vq-jw3w-q789
stephanretool commentedon Oct 23, 2026
Thank you, appreciate the quick fix!