-
Notifications
You must be signed in to change notification settings - Fork 697
fix: gnome-autoar - remove flagged files #18626
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Andrew Phelps (anphel31)
wants to merge
1
commit into
4.0
Choose a base branch
from
anphel/fix-gnome-autoar-flagged-files
base: 4.0
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| [components.gnome-autoar] | ||
|
|
||
| # Serve a locally-modified tarball (see modify_source.py) with the three | ||
| # encrypted extract-test fixtures removed. An azldev archive overlay can't be | ||
| # used here: the upstream tarball ships an absolute-target symlink fixture | ||
| # (tests/files/extract/test-symlink-parent/reference/arextract -> /tmp) that | ||
| # azldev's overlay extractor rejects, so the repack is done out-of-band and | ||
| # served from the staging lookaside via origin=download. | ||
| [[components.gnome-autoar.source-files]] | ||
| filename = "gnome-autoar-0.4.5.tar.xz" | ||
| hash = "855c016959c216b233c5a7c07c8a96f8beeb74b55154fdc054f67768c4b4935b7dc35d16dcd8aa8feee9b5ec283acaf92dd0ea686ebcd9a2a00cc6a2753da4a1" | ||
| hash-type = "SHA512" | ||
| origin = { type = "download", uri = "https://azltempstaginglookaside.blob.core.windows.net/repo/pkgs_modified/gnome-autoar/gnome-autoar-0.4.5.tar.xz/sha512/855c016959c216b233c5a7c07c8a96f8beeb74b55154fdc054f67768c4b4935b7dc35d16dcd8aa8feee9b5ec283acaf92dd0ea686ebcd9a2a00cc6a2753da4a1/gnome-autoar-0.4.5.tar.xz" } | ||
| replace-upstream = true | ||
| replace-reason = "Removes the three encrypted extract-test fixtures (tests/files/extract/test-encrypted*/, password-protected zips) that fail the package-signing scan, plus the three meson test cases that read them (modify_source.py). Test-only, not shipped in any binary RPM. Uses origin=download rather than an archive overlay because the tarball ships an absolute-target symlink fixture azldev's overlay extractor rejects." | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| #!/usr/bin/env python3 | ||
| """Repack the gnome-autoar upstream tarball with its encrypted test fixtures removed. | ||
|
|
||
| The three ``tests/files/extract/test-encrypted*/`` fixtures ship | ||
| password-protected ``input/arextract.zip`` files that the package-signing scan | ||
| refuses to inspect and blocks on (the flagged bytes live in the ``.src.rpm`` | ||
| itself, so skipping the tests at ``%check`` time would not help -- the files | ||
| have to physically leave the tarball). This script removes those fixture | ||
| directories, drops the three meson test cases that read them so ``%meson_test`` | ||
| still passes, and repacks the tarball deterministically. | ||
|
|
||
| This component uses ``modify_source.py`` + ``origin=download`` rather than an | ||
| azldev archive overlay because the upstream tarball ships an absolute-target | ||
| symlink fixture (``tests/files/extract/test-symlink-parent/reference/arextract | ||
| -> /tmp``) that azldev's overlay extractor refuses to extract. ``tar`` handles | ||
| it, so the repack is done out-of-band and the result is served from the staging | ||
| lookaside. | ||
|
|
||
| The repack pins the umask and fixes the ``tar`` member order, mtime and owner so | ||
| the output is stable for a given ``tar``/``xz`` toolchain (``xz`` output is not | ||
| guaranteed identical across liblzma versions). The artifact published to the | ||
| lookaside -- pinned by SHA-512 in ``specs/g/gnome-autoar/sources`` -- is the | ||
| source of truth the build downloads; this script only regenerates an equivalent | ||
| tarball. Regenerating on a different toolchain may yield a new hash, which then | ||
| has to be re-uploaded and re-pinned. | ||
|
|
||
| Output lands under ``<repo-root>/base/build/work/scratch/<package>/``. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import hashlib | ||
| import os | ||
| import re | ||
| import shutil | ||
| import subprocess | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| PACKAGE_NAME = "gnome-autoar" | ||
| VERSION = "0.4.5" | ||
| ORIGINAL_NAME = f"{PACKAGE_NAME}-{VERSION}.tar.xz" | ||
| TOPDIR = f"{PACKAGE_NAME}-{VERSION}" | ||
|
|
||
| # Pristine upstream Source0 SHA-512 (download.gnome.org), used to verify the | ||
| # download before repacking. This is the ORIGINAL upstream checksum, not the | ||
| # modified/served hash that now lives in specs/g/gnome-autoar/sources. | ||
| UPSTREAM_SHA512 = ( | ||
| "ba38dfc0ad3c00fd8316d02d1a8e38ce3c743e11032f7c4efff74e7c3f8e8e815" | ||
| "a1debe51eae8e2ee653155356d34992f1bc0e35e6cfab82398265fde8648050" | ||
| ) | ||
| UPSTREAM_URL = ( | ||
| f"https://download.gnome.org/sources/{PACKAGE_NAME}/0.4/{ORIGINAL_NAME}" | ||
| ) | ||
|
|
||
| # Encrypted extract-test fixture directories removed to avoid scan failures on | ||
| # the SRPM. Each ships an encrypted input/arextract.zip. Paths are relative to | ||
| # the tarball's top-level directory. Sorted alphabetically. | ||
| FIXTURE_DIRS_TO_REMOVE = ( | ||
| "tests/files/extract/test-encrypted", | ||
| "tests/files/extract/test-encrypted-request-passphrase", | ||
| "tests/files/extract/test-encrypted-wrong-passphrase", | ||
| ) | ||
|
|
||
| # The meson extract-unit test hard-codes three encrypted cases that read the | ||
| # fixtures above; drop their function definitions and g_test_add_func | ||
| # registrations so the remaining suite still runs. | ||
| TEST_FILE = "tests/test-extract-unit.c" | ||
|
|
||
| # Fixed umask so `tar -x` records reproducible file modes regardless of the | ||
| # caller's environment (extracting the same archive under different umasks would | ||
| # otherwise yield different modes and therefore a different repacked SHA-512). | ||
| REPACK_UMASK = 0o022 | ||
|
|
||
|
|
||
| def run(cmd: list[str], **kwargs) -> subprocess.CompletedProcess: | ||
| return subprocess.run(cmd, check=True, **kwargs) | ||
|
Comment on lines
+76
to
+77
|
||
|
|
||
|
|
||
| def sha512_of(path: Path) -> str: | ||
|
|
||
| digest = hashlib.sha512() | ||
| with path.open("rb") as fh: | ||
| for chunk in iter(lambda: fh.read(1024 * 1024), b""): | ||
| digest.update(chunk) | ||
| return digest.hexdigest() | ||
|
|
||
|
|
||
| def edit_test_file(path: Path) -> None: | ||
| """Drop the three encrypted test cases + their registrations in place.""" | ||
| text = path.read_text(encoding="utf-8") | ||
|
|
||
| # 1) Remove the three contiguous encrypted test function definitions | ||
| # (test_encrypted, test_encrypted_request_passphrase, | ||
| # test_encrypted_wrong_passphrase). | ||
| defs = re.compile( | ||
| r"\nstatic void\ntest_encrypted \(void\)\n.*?" | ||
| r"\nstatic void\ntest_encrypted_wrong_passphrase \(void\)\n\{.*?\n\}\n", | ||
| re.DOTALL, | ||
| ) | ||
| text, n_defs = defs.subn("", text, count=1) | ||
| if n_defs != 1: | ||
| sys.exit(f"expected 1 encrypted-def block, removed {n_defs}") | ||
|
|
||
| # 2) Remove the three contiguous g_test_add_func registrations. | ||
| regs = re.compile( | ||
| r'\n\n g_test_add_func \("/autoar-extract/test-encrypted",\n' | ||
| r".*?test_encrypted_wrong_passphrase\);", | ||
| re.DOTALL, | ||
| ) | ||
| text, n_regs = regs.subn("", text, count=1) | ||
| if n_regs != 1: | ||
| sys.exit(f"expected 1 encrypted-registration block, removed {n_regs}") | ||
|
|
||
| if "test_encrypted" in text: | ||
| sys.exit("residual test_encrypted reference remains after edit") | ||
|
|
||
| path.write_text(text, encoding="utf-8") | ||
|
|
||
|
|
||
| def main() -> None: | ||
| os.umask(REPACK_UMASK) | ||
|
|
||
| repo_root = Path( | ||
| run( | ||
| ["git", "rev-parse", "--show-toplevel"], | ||
| capture_output=True, | ||
| text=True, | ||
| ).stdout.strip() | ||
| ) | ||
| workdir = repo_root / "base" / "build" / "work" / "scratch" / PACKAGE_NAME | ||
| workdir.mkdir(parents=True, exist_ok=True) | ||
| os.chdir(workdir) | ||
|
|
||
| original = Path(ORIGINAL_NAME) | ||
| print(f"[1/6] Downloading {ORIGINAL_NAME}") | ||
| if not original.exists(): | ||
| # No atomic-rename dance: the SHA-512 check below rejects any partial or | ||
| # corrupt download, so a re-run simply re-fetches it. | ||
| run(["curl", "-fsSL", "--retry", "3", "-o", ORIGINAL_NAME, UPSTREAM_URL]) | ||
|
Comment on lines
+136
to
+139
|
||
|
|
||
| print("[2/6] Verifying upstream SHA512") | ||
| computed = sha512_of(original) | ||
| if computed != UPSTREAM_SHA512: | ||
| sys.exit( | ||
| "ERROR: upstream SHA512 mismatch\n" | ||
| f" expected: {UPSTREAM_SHA512}\n" | ||
| f" computed: {computed}" | ||
| ) | ||
|
|
||
| print("[3/6] Extracting") | ||
| extract_dir = Path("extracted") | ||
| if extract_dir.exists(): | ||
| shutil.rmtree(extract_dir) | ||
| extract_dir.mkdir() | ||
| run(["tar", "-xf", ORIGINAL_NAME, "-C", str(extract_dir)]) | ||
|
|
||
| print(f"[4/6] Removing {len(FIXTURE_DIRS_TO_REMOVE)} encrypted fixture dirs") | ||
| for rel in FIXTURE_DIRS_TO_REMOVE: | ||
| target = extract_dir / TOPDIR / rel | ||
| if not target.is_dir(): | ||
| sys.exit( | ||
| f"ERROR: expected fixture dir not present in upstream tarball: {rel}" | ||
| ) | ||
| shutil.rmtree(target) | ||
|
|
||
| print(f"[5/6] Dropping encrypted test cases from {TEST_FILE}") | ||
| test_path = extract_dir / TOPDIR / TEST_FILE | ||
| edit_test_file(test_path) | ||
| # Sanity: sibling tests must survive. | ||
| if "test_readonly_directory" not in test_path.read_text(encoding="utf-8"): | ||
| sys.exit("ERROR: sibling test unexpectedly removed") | ||
|
|
||
| print("[6/6] Repacking deterministically") | ||
| modified = Path(f"{ORIGINAL_NAME}.modified") | ||
| modified.unlink(missing_ok=True) | ||
| # Stable byte output: sorted names, fixed mtime, zeroed owner/group, | ||
| # single-threaded xz. tar writes to xz's stdin; xz writes the archive. | ||
| with modified.open("wb") as out: | ||
| tar = subprocess.Popen( | ||
| [ | ||
| "tar", | ||
| "--sort=name", | ||
| "--mtime=2024-01-01 00:00:00 UTC", | ||
| "--owner=0", | ||
| "--group=0", | ||
| "--numeric-owner", | ||
| "-cf", | ||
| "-", | ||
| TOPDIR, | ||
| ], | ||
| cwd=extract_dir, | ||
| stdout=subprocess.PIPE, | ||
| ) | ||
| xz = subprocess.Popen( | ||
| ["xz", "-T1", "-9e"], stdin=tar.stdout, stdout=out | ||
| ) | ||
| tar.stdout.close() # allow tar to receive SIGPIPE if xz exits | ||
|
Comment on lines
+194
to
+197
|
||
| if xz.wait() != 0 or tar.wait() != 0: | ||
| sys.exit("ERROR: repack failed") | ||
|
|
||
| modified_sha512 = sha512_of(modified) | ||
| # Record the checksum against the actual modified file so `sha512sum -c` works. | ||
| Path(f"{ORIGINAL_NAME}.modified.sha512").write_text( | ||
| f"{modified_sha512} {ORIGINAL_NAME}.modified\n", encoding="utf-8" | ||
| ) | ||
|
|
||
| blob_name = ( | ||
| f"pkgs_modified/{PACKAGE_NAME}/{ORIGINAL_NAME}" | ||
| f"/sha512/{modified_sha512}/{ORIGINAL_NAME}" | ||
| ) | ||
| print( | ||
| f"\nmodified tarball: {workdir / modified.name}\n" | ||
| f"SHA512: {modified_sha512}\n\n" | ||
| "Upload to the staging lookaside (requires write access to the\n" | ||
| "azltempstaginglookaside storage account; if you don't have it, hand the\n" | ||
| "file to someone who does or use the lookaside-upload pipeline):\n" | ||
| " az storage blob upload \\\n" | ||
| " --auth-mode login \\\n" | ||
| " --account-name azltempstaginglookaside \\\n" | ||
| " --container-name repo \\\n" | ||
| f' --name "{blob_name}" \\\n' | ||
| f' --file "{workdir / modified.name}"' | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| SHA512 (gnome-autoar-0.4.5.tar.xz) = ba38dfc0ad3c00fd8316d02d1a8e38ce3c743e11032f7c4efff74e7c3f8e8e815a1debe51eae8e2ee653155356d34992f1bc0e35e6cfab82398265fde8648050 | ||
| SHA512 (gnome-autoar-0.4.5.tar.xz) = 855c016959c216b233c5a7c07c8a96f8beeb74b55154fdc054f67768c4b4935b7dc35d16dcd8aa8feee9b5ec283acaf92dd0ea686ebcd9a2a00cc6a2753da4a1 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Verified via the
modify_source.shpost-conditions andazldev comp render --check-only(no drift): the script checks the upstream SHA512 before editing, removes the encrypted fixtures, drops the threetest_encrypted*cases and theirg_test_add_funcregistrations, and asserts siblings (e.g.test_readonly_directory) survive. meson uses only selective-Werror=flags (no-Werror=unused-functionand no blanketwerror), so removing the cases doesn't break compilation. A full RPM/mock build isn't runnable in my environment; CI performs it.