Skip to content

Commit cae8e93

Browse files
dhrimovDmytro Hrimovclaude
authored
Report the output path and sizes, and fail when no package is produced (#16)
* feat: write a JSON report of the output path and sizes The resolved output path and the two size figures currently exist only inside log lines, so a script that calls this tool cannot find out which file was just written without re-deriving the distribution name itself or scraping stdout. Add an opt-in `--report <path>` flag that writes a single JSON object once packaging has succeeded: { "output_file": "/abs/path/my_app.zip", "distribution_name": "my_app", "output_bytes": 3460000, "uncompressed_bytes": 412000000, "compressed_bytes": 3456789, "nested_zip": false } `output_bytes` is the size of the file at `output_file`, so it is correct under both strategies. `compressed_bytes` remains the size of the dependencies zip, which is the figure compared against the Lambda limit; under the nested strategy the outer zip is larger, because it also holds the loader and the stored inner zip. Writing to a file rather than stdout keeps the report immune to anything else the tool prints, and leaves existing output untouched for callers that do not pass the flag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: raise PackageTooLargeError instead of exiting 0 with no package When the uncompressed size exceeds the Lambda limit and the compressed size does too, the nested-zip strategy cannot help. The tool printed a placeholder message, wrote no file, and returned normally, so the process exited 0. Under `set -euo pipefail` that reads as success: the build goes green with no zip on disk, and whatever consumes the artifact next fails somewhere far from the cause. Raise instead, carrying both sizes and the limit so the message says how much too large the package is. The traceback goes to stderr and the process exits non-zero. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: replace TODO placeholder exceptions with real messages Two ordinary user mistakes surfaced as bare tracebacks that named neither the problem nor the fix: - a venv path with no `lib/python*` directory raised `Exception("input_path")` - a pyproject.toml with no `[project].name` or `[tool.poetry].name` raised `Exception("TODO Exception find_value")` Both now say what was looked for and where, and use a fitting builtin exception type. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: pass re.UNICODE as flags rather than count `re.sub`'s fourth positional parameter is `count`, not `flags`, so `re.sub("[^\w\d.]+", "_", self.name, re.UNICODE)` meant "replace at most 32 occurrences". Names with more than 32 runs of characters to replace were normalised only partway. `re.UNICODE` is already the default for str patterns in Python 3, so it can go entirely. The pattern also becomes a raw string: as a plain string it contains the undefined escapes `\w` and `\d`, which is a SyntaxWarning on current Python and is slated to become an error. Passing `count` positionally is deprecated as of 3.13, so this also removes a DeprecationWarning from the test run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix: validate SOURCE_DATE_EPOCH before packaging `date_time()` runs once per file written, so a bad $SOURCE_DATE_EPOCH was only noticed after the venv had been built and while the zip was being produced. Call it once at startup so the failure lands before any work. A non-integer value also gave a bare ValueError traceback while a pre-1980 value gave SourceDateEpochError. Both now raise SourceDateEpochError, and the message quotes the offending value. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat: make --output and --output-dir mutually exclusive Passing both was accepted, with --output silently winning. Put them in an argparse mutually exclusive group so the conflict is reported instead. The group is not required: --output-dir keeps its default of the current working directory, so invocations that pass neither are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: document --report and the output filename contract Document the new --report flag and each field of the JSON it writes. State the output filename rule as a contract: `<output-dir>/<distribution_name>.zip`, where the name comes from `[project].name` or `[tool.poetry].name`, and case is preserved. The README previously said only "with dashes replaced with underscores", which is both incomplete (every run of characters outside `A-Za-z0-9_.` is replaced) and silent on case, so `My-App` yielding `My_App.zip` was left for callers to discover. That also differs from the lowercased wheel filename for the same project, which is a tempting and wrong way to predict it. Also correct "One of the following must be specified" for `--output` / `--output-dir`: neither is required, and they cannot now be combined. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Dmytro Hrimov <dmytrohrimov@Dmytros-MacBook-Pro.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent da69112 commit cae8e93

6 files changed

Lines changed: 366 additions & 17 deletions

File tree

‎README.md‎

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,8 +15,7 @@ poetry bundle venv .build/.venv --without dev
1515
package-python-function .build/.venv --output-dir .build/lambda
1616
```
1717

18-
The output will be a .zip file with the same name as your project from your `pyproject.toml` file (with dashes replaced
19-
with underscores).
18+
The output will be a .zip file named after your project, as described in [Output file name](#output-file-name).
2019

2120
## Installation
2221
Use [pipx](https://github.com/pypa/pipx) to install:
@@ -27,16 +26,57 @@ pipx install package-python-function
2726

2827
## Usage / Arguments
2928
```shell
30-
package-python-function venv_dir [--project PROJECT] [--output-dir OUTPUT_DIR] [--output OUTPUT]
29+
package-python-function venv_dir [--project PROJECT] [--output-dir OUTPUT_DIR] [--output OUTPUT] [--report REPORT]
3130
```
3231

3332
- `venv_dir` [Required]: The path to the virtual environment to package.
3433
- `--project` [Optional]: Path to the `pyproject.toml` file. Omit to use the `pyproject.toml` file in the current working directory.
34+
- `--report` [Optional]: Path to write a JSON [report file](#report-file) to. Omit to write no report.
3535

36-
One of the following must be specified:
36+
`--output` and `--output-dir` cannot be used together. If neither is given, the zip is written to the current working
37+
directory.
3738
- `--output`: The full output path of the final zip file.
38-
- `--output-dir`: The output directory for the final zip file. The name of the zip file will be based on the project's
39-
name in the `pyproject.toml` file (with dashes replaced with underscores).
39+
- `--output-dir`: The output directory for the final zip file. The name of the zip file is described in
40+
[Output file name](#output-file-name).
41+
42+
## Output file name
43+
44+
Unless `--output` gives an exact path, the file written is `<output-dir>/<distribution_name>.zip`.
45+
46+
`distribution_name` is the project's name — `[project].name`, or `[tool.poetry].name` if that is absent — with each run
47+
of characters outside `A-Z a-z 0-9 _ .` replaced by a single underscore, following the
48+
[PyPA escaping rules](https://peps.python.org/pep-0427/#escaping-and-unicode).
49+
50+
**Case is preserved.** A project named `My-App` produces `My_App.zip`, not `my_app.zip`. Note that this differs from the
51+
wheel your build tool produces for the same project, whose filename is lowercased — so a wheel's name is not a safe way
52+
to predict the name of this file.
53+
54+
## Report file
55+
56+
Pass `--report <path>` to have the tool write a JSON description of what it produced, so a calling script does not have
57+
to re-derive the output path or re-measure the package.
58+
59+
```json
60+
{
61+
"output_file": "/abs/path/lambda/my_app.zip",
62+
"distribution_name": "my_app",
63+
"output_bytes": 3460000,
64+
"uncompressed_bytes": 412000000,
65+
"compressed_bytes": 3456789,
66+
"nested_zip": false
67+
}
68+
```
69+
70+
| Field | Meaning |
71+
| --- | --- |
72+
| `output_file` | Absolute path of the zip that was written. |
73+
| `distribution_name` | The normalized project name, as described in [Output file name](#output-file-name). |
74+
| `output_bytes` | Size of the file at `output_file`. This is the artifact you deploy. |
75+
| `uncompressed_bytes` | Total size of the packaged files before compression. This is the figure compared against the AWS Lambda 250 MiB unzipped limit. |
76+
| `compressed_bytes` | Size of the dependencies zip. Equal to `output_bytes` unless the nested-zip strategy was used, in which case the outer zip also holds the loader. |
77+
| `nested_zip` | Whether the nested-zip strategy was used. |
78+
79+
The report is written only when packaging succeeds, so its presence is a reliable signal that the zip is really there.
4080

4181
## Notes on Reproducibility
4282

‎package_python_function/main.py‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -4,24 +4,33 @@
44
import sys
55

66
from .packager import Packager
7+
from .reproducible_zipfile import date_time
78

89

910
def main() -> None:
1011
logging.basicConfig(level=logging.INFO, stream=sys.stdout, format="%(message)s")
1112

1213
args = parse_args()
14+
15+
# Validate $SOURCE_DATE_EPOCH here, so that a bad value fails before any packaging work rather than partway
16+
# through writing the zip.
17+
date_time()
18+
1319
project_path = Path(args.project).resolve()
1420
venv_path = Path(args.venv_dir).resolve()
1521
output_dir_path = Path(args.output_dir).resolve()
1622
output_file_path = Path(args.output).resolve() if args.output else None
17-
packager = Packager(venv_path, project_path, output_dir_path, output_file_path)
23+
report_file_path = Path(args.report).resolve() if args.report else None
24+
packager = Packager(venv_path, project_path, output_dir_path, output_file_path, report_file_path)
1825
packager.package()
1926

2027

2128
def parse_args() -> argparse.Namespace:
2229
arg_parser = argparse.ArgumentParser()
2330
arg_parser.add_argument("venv_dir", type=str, help="The directory path to the virtual environment to package into a zip file")
2431
arg_parser.add_argument("--project", type=str, default='pyproject.toml', help="The path to the project's pyproject.toml file. Omit to use pyproject.toml in the current working directory.")
25-
arg_parser.add_argument("--output-dir", type=str, default='.', help="The directory path to save the output zip file. Default is the current working directory.")
26-
arg_parser.add_argument("--output", type=str, default='', help="The full file path for the output file. Use this instead of --output-dir if you want total control of the output file path.")
32+
output_group = arg_parser.add_mutually_exclusive_group()
33+
output_group.add_argument("--output-dir", type=str, default='.', help="The directory path to save the output zip file. Default is the current working directory.")
34+
output_group.add_argument("--output", type=str, default='', help="The full file path for the output file. Use this instead of --output-dir if you want total control of the output file path.")
35+
arg_parser.add_argument("--report", type=str, default='', help="The file path to write a JSON report of the packaging result to. Omit to write no report.")
2736
return arg_parser.parse_args()

‎package_python_function/packager.py‎

Lines changed: 55 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
from __future__ import annotations
22

3+
import json
34
import logging
45
import shutil
56
from pathlib import Path
@@ -11,26 +12,52 @@
1112

1213
logger = logging.getLogger(__name__)
1314

15+
class PackageTooLargeError(Exception):
16+
"""Raise when the content is too large for AWS Lambda, both uncompressed and compressed"""
17+
18+
def __init__(self, uncompressed_bytes: int, compressed_bytes: int, limit_bytes: int):
19+
self.uncompressed_bytes = uncompressed_bytes
20+
self.compressed_bytes = compressed_bytes
21+
self.limit_bytes = limit_bytes
22+
super().__init__(
23+
f"The uncompressed size ({uncompressed_bytes:,} bytes) is too large for AWS Lambda, and the compressed "
24+
f"size ({compressed_bytes:,} bytes) also exceeds the limit of {limit_bytes:,} bytes, so the nested-zip "
25+
"strategy cannot be used either. No package was written."
26+
)
27+
1428
class Packager:
1529
AWS_LAMBDA_MAX_UNZIP_SIZE = 262_144_000
1630
DIRS_TO_EXCLUDE = ["__pycache__"]
1731
DIST_INFO_FILES_TO_EXCLUDE = ["RECORD", "direct_url.json"]
1832
EXTENSIONS_TO_EXCLUDE = [".pyc", ".pyo"]
1933

20-
def __init__(self, venv_path: Path, project_path: Path, output_dir: Path, output_file: Path | None):
34+
def __init__(
35+
self,
36+
venv_path: Path,
37+
project_path: Path,
38+
output_dir: Path,
39+
output_file: Path | None,
40+
report_file: Path | None = None,
41+
):
2142
self.project = PythonProject(project_path)
2243
self.venv_path = venv_path
2344

2445
self.output_dir = output_file.parent if output_file else output_dir
2546
self.output_file = output_file if output_file else output_dir / f'{self.project.distribution_name}.zip'
47+
self.report_file = report_file
2648

2749
self._uncompressed_bytes = 0
50+
self._compressed_bytes = 0
51+
self._nested_zip = False
2852

2953
@property
3054
def input_path(self) -> Path:
3155
python_paths = list((self.venv_path / 'lib').glob('python*'))
3256
if not python_paths:
33-
raise Exception("input_path")
57+
raise FileNotFoundError(
58+
f"No 'lib/python*' directory was found in '{self.venv_path}'. Check that this path points at a "
59+
"virtual environment with the function's dependencies installed into it."
60+
)
3461
return python_paths[0] / 'site-packages'
3562

3663
def package(self) -> None:
@@ -41,6 +68,27 @@ def package(self) -> None:
4168
with NamedTemporaryFile(suffix=".zip") as dependencies_zip:
4269
self.zip_all_dependencies(Path(dependencies_zip.name))
4370

71+
if self.report_file:
72+
self.write_report()
73+
74+
def write_report(self) -> None:
75+
"""
76+
Write a JSON report describing the package that was just produced, so that a calling script does not have to
77+
re-derive the output path or re-measure the sizes.
78+
"""
79+
report = {
80+
"output_file": str(self.output_file),
81+
"distribution_name": self.project.distribution_name,
82+
"output_bytes": self.output_file.stat().st_size,
83+
"uncompressed_bytes": self._uncompressed_bytes,
84+
"compressed_bytes": self._compressed_bytes,
85+
"nested_zip": self._nested_zip,
86+
}
87+
88+
logger.info(f"Writing report to '{self.report_file}'...")
89+
self.report_file.parent.mkdir(parents=True, exist_ok=True)
90+
self.report_file.write_text(json.dumps(report, indent=2) + "\n")
91+
4492
def zip_all_dependencies(self, target_path: Path) -> None:
4593
logger.info(f"Zipping to {target_path}...")
4694

@@ -64,17 +112,20 @@ def zip_dir(path: Path) -> None:
64112

65113
zip_dir(self.input_path)
66114

67-
compressed_bytes = target_path.stat().st_size
115+
compressed_bytes = self._compressed_bytes = target_path.stat().st_size
68116

69117
logger.info(f"Uncompressed size: {self._uncompressed_bytes:,} bytes. Compressed size: {compressed_bytes:,} bytes.")
70118

71119
if self._uncompressed_bytes > self.AWS_LAMBDA_MAX_UNZIP_SIZE:
72120
logger.info(f"The uncompressed size of the ZIP file is greater than the AWS Lambda limit of {self.AWS_LAMBDA_MAX_UNZIP_SIZE:,} bytes.")
73121
if(compressed_bytes < self.AWS_LAMBDA_MAX_UNZIP_SIZE):
74122
logger.info(f"The compressed size ({compressed_bytes:,}) is less than the AWS limit, so the nested-zip strategy will be used.")
123+
self._nested_zip = True
75124
self.generate_nested_zip(target_path)
76125
else:
77-
print("TODO Error. The unzipped size it too large for AWS Lambda.")
126+
raise PackageTooLargeError(
127+
self._uncompressed_bytes, compressed_bytes, self.AWS_LAMBDA_MAX_UNZIP_SIZE
128+
)
78129
else:
79130
logger.info(f"Copying '{target_path}' to '{self.output_file}'")
80131
shutil.copy(str(target_path), str(self.output_file))

‎package_python_function/python_project.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ def name(self) -> str:
2525
"""
2626
@cached_property
2727
def distribution_name(self) -> str:
28-
return re.sub("[^\w\d.]+", "_", self.name, re.UNICODE)
28+
return re.sub(r"[^\w\d.]+", "_", self.name)
2929

3030
@cached_property
3131
def entrypoint_package_name(self) -> str:
@@ -41,7 +41,8 @@ def find_value(self, paths: tuple[tuple[str]]) -> str:
4141
value = self.get_value(path)
4242
if value is not None:
4343
return value
44-
raise Exception("TODO Exception find_value")
44+
searched = ", ".join(".".join(path) for path in paths)
45+
raise ValueError(f"None of the following were found in '{self.path}': {searched}.")
4546

4647
def get_value(self, path: tuple[str]) -> Optional[str]:
4748
node = self.toml

‎package_python_function/reproducible_zipfile.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,14 @@ def date_time() -> Tuple[int, int, int, int, int, int]:
2525
"""
2626
source_date_epoch = os.environ.get("SOURCE_DATE_EPOCH", None)
2727
if source_date_epoch is not None:
28-
dt = time.gmtime(int(source_date_epoch))[:6]
28+
try:
29+
seconds_since_epoch = int(source_date_epoch)
30+
except ValueError as error:
31+
raise SourceDateEpochError(
32+
f"$SOURCE_DATE_EPOCH must be an integer number of seconds since the Epoch, but was "
33+
f"'{source_date_epoch}'."
34+
) from error
35+
dt = time.gmtime(seconds_since_epoch)[:6]
2936
if dt[0] < 1980:
3037
raise SourceDateEpochError(
3138
"$SOURCE_DATE_EPOCH must be >= 315532800, since ZIP files need MS-DOS date/time format, which can be 1/1/1980, at minimum."

0 commit comments

Comments
 (0)