Document the commands provided by Platforms/WASI/__main__.py (GH-158976)

Along the way, fix some bugs and add a `path` command to make is easier to programmatically figure out where things are.

The README should also be self-contained enough that one could point an LLM at it for working on WASI.

Author: brettcannon

Platforms/WASI/README.md

@@ -9,9 +9,237 @@ use WASM runtimes such as [wasmtime](https://wasmtime.dev/).
 **NOTE**: If you are looking for general information about WebAssembly that is
 not directly related to CPython, please see https://github.com/psf/webassembly.
 
-## Build
 
-See [the devguide on how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi).
+## Working on the WASI build
+
+This directory provides a CLI for building and packaging WASI builds for
+distribution.
+
+Run all commands below from the root of the CPython source checkout.
+
+To see all the available commands, run:
+
+```shell
+python3 Platforms/WASI --help
+```
+
+The `python3` interpreter used to run this CLI must be a Python version that
+is receiving bugfixes. This requirement does not apply to the build Python,
+which the CLI builds from the source checkout.
+
+
+### Prerequisites
+
+There are some tools that must be available to successfully build.
+
+1. C compiler
+2. `make`
+3. WASI SDK
+4. Wasmtime (or some other WASI runtime configured via `--host-runner`)
+
+The default runner requires Wasmtime be on `PATH`.
+
+The WASI SDK must be the same version as specified in config.toml. The search
+for the WASI SDK is done via:
+
+1. `--wasi-sdk` CLI option
+2. `WASI_SDK_PATH` environment variable
+3. `/opt` where the WASI SDK has been unpacked from its tarball
+
+Note that all prerequisites are included and configured appropriately in the
+[WASI dev container image](https://github.com/python/cpython-devcontainers/pkgs/container/wasicontainer).
+You can download it via:
+
+```shell
+podman pull ghcr.io/python/wasicontainer:latest
+```
+
+The `latest` image contains the WASI SDK versions required by all supported
+CPython branches.
+
+### Development loop
+
+The common way to get started is to first do a full build:
+
+```shell
+python3 Platforms/WASI build --quiet --logdir cross-build/logs -- --with-pydebug --config-cache
+```
+
+In the end, you will end up with a "build Python" which is a local build of
+Python used for cross-builds. You will also have the WASI build.
+
+Once you have the build you can run the test you want.
+
+Bash:
+```bash
+"$(python3 Platforms/WASI path)/python.sh" -m test test_os
+```
+
+Fish:
+```fish
+set -l wasi_dir (python3 Platforms/WASI path); "$wasi_dir/python.sh" -m test test_os
+```
+
+If you are working on C code and need a rebuild:
+
+```shell
+python3 Platforms/WASI make-host
+```
+
+
+### Building
+
+In general,
+[the devguide covers how to build and run for WASI](https://devguide.python.org/getting-started/setup-building/#wasi),
+but we will cover some of the details here.
+
+The simplest way to get a pydebug WASI build is:
+
+```shell
+python3 Platforms/WASI build -- --with-pydebug
+```
+
+This builds the build Python and the WASI build with `--with-pydebug` passed to
+`configure` (as is anything that comes after `--`). The builds are placed in
+the `cross-build/` directory of the source checkout, each in a subdirectory
+matching the compiler triple for the build.
+
+You can do the two builds separately if you want:
+
+```shell
+python3 Platforms/WASI build-python -- --with-pydebug
+python3 Platforms/WASI build-host
+```
+
+This can be broken down even more to the separate `configure` and `make` steps:
+
+```shell
+python3 Platforms/WASI configure-build-python -- --with-pydebug
+python3 Platforms/WASI make-build-python
+python3 Platforms/WASI configure-host
+python3 Platforms/WASI make-host
+```
+
+Note that `configure-host` figures out to do a pydebug build by looking at the
+build Python.
+
+There is a `--quiet` flag to redirect output from the underlying commands to a
+directory. The `--logdir` flag controls where the log files go (which defaults
+to `/tmp`).
+
+```shell
+python3 Platforms/WASI build --quiet --logdir cross-build/logs
+```
+
+
+### Packaging
+
+The `package` command is used to gather all the files necessary to make a
+release and place them in an archive:
+
+```shell
+python3 Platforms/WASI package
+```
+
+The files are gathered into a versioned directory inside `dist/` in the source
+checkout. An archive containing that directory is placed alongside it in
+`dist/`. The gathered directory includes a `bin/python3.wasmtime` file to ease
+launching the interpreter.
+
+If you just need to gather the files for a release, you can use the `gather`
+command:
+
+```shell
+python3 Platforms/WASI gather
+```
+
+Both `package` and `gather` require a completed build and delete the entire
+existing `dist/` directory before gathering files.
+
+There is no command just to archive the gathered files.
+
+
+### Paths
+
+The `path` command prints the location of the build Python, WASI build, or
+gathered distribution files:
+
+```shell
+python3 Platforms/WASI path build-python
+python3 Platforms/WASI path wasi
+python3 Platforms/WASI path dist
+```
+
+With no location argument, `path` defaults to `wasi`. The `dist` location
+returns the versioned distribution directory inside `dist/`, not the top-level
+`dist/` directory. The `build-python` and `wasi` locations can be queried before
+building; the `dist` location requires WASI build metadata to determine the
+versioned directory name.
+
+
+### Cleanup
+
+The `clean` command deletes the entire `cross-build/` and `dist/` directories,
+including all build files, gathered distribution files, and archives:
+
+```shell
+python3 Platforms/WASI clean
+```
+
+
+### Testing
+
+To find out where the WASI build directory is, you can run:
+
+```shell
+python3 Platforms/WASI path
+```
+
+From there, you can run the test suite.
+
+For Bash-like shells:
+
+```bash
+make buildbottest -C "$(python3 Platforms/WASI path)"
+```
+
+For fish:
+
+```fish
+make buildbottest -C (python3 Platforms/WASI path)
+```
+
+There is a `python.sh` file in the WASI build directory, so you can also run
+tests that way.
+
+Bash:
+
+```bash
+"$(python3 Platforms/WASI path)/python.sh" -m test
+```
+
+Fish:
+
+```fish
+set -l wasi_dir (python3 Platforms/WASI path); "$wasi_dir/python.sh" -m test
+```
+
+If you want to test the files meant for distribution, the directory containing
+the files can be found via `python3 Platforms/WASI path dist` and there is a
+`bin/python3.wasmtime` shell script.
+
+Bash:
+
+```bash
+"$(python3 Platforms/WASI path dist)/bin/python3.wasmtime" -m test
+```
+
+Fish:
+
+```fish
+set -l dist_dir (python3 Platforms/WASI path dist); "$dist_dir/bin/python3.wasmtime" -m test
+```
+
 
 ## Detecting WASI builds
 
@@ -48,6 +276,7 @@ posix.uname_result(
 'wasi'
 ```
 
+
 ### C code
 
 WASI SDK defines several built-in macros. You can dump a full list of built-ins

Platforms/WASI/__main__.py

@@ -68,6 +68,19 @@ def main():
     package = subcommands.add_parser(
         "package", help="Package the host/WASI Python into an archive"
     )
+    gather = subcommands.add_parser(
+        "gather", help="Gather all the files for distribution"
+    )
+    path = subcommands.add_parser(
+        "path", help="Print the path to a build or distribution directory"
+    )
+    path.add_argument(
+        "location",
+        nargs="?",
+        choices=("build-python", "wasi", "dist"),
+        default="wasi",
+        help="Directory whose path to print",
+    )
     subcommands.add_parser(
         "clean", help="Delete files and directories created by this script"
     )
@@ -144,7 +157,9 @@ def main():
         make_host,
         build_host,
         pythoninfo_host,
+        gather,
         package,
+        path,
     ):
         subcommand.add_argument(
             "--host-triple",
@@ -191,9 +206,18 @@ def main():
                 _build.pythoninfo_wasi_python(context)
         case "clean":
             _build.clean_contents(context)
+        case "gather":
+            _package.gather(context)
         case "package":
             _package.gather(context)
             _package.archive(context)
+        case "path":
+            paths = {
+                "build-python": "build_python_path",
+                "wasi": "wasi_build_path",
+                "dist": "archive_dir",
+            }
+            print(getattr(context, paths[context.location]))
         case None:
             parser.print_help()
         case _:

Platforms/WASI/_build.py

@@ -113,6 +113,7 @@ def call(command, *, context=None, quiet=False, **kwargs):
     else:
         if (log_path := getattr(context, "log_path", None)) is None:
             log_path = pathlib.Path(tempfile.gettempdir())
+        log_path.mkdir(parents=True, exist_ok=True)
         stdout = tempfile.NamedTemporaryFile(
             "w",
             encoding="utf-8",
@@ -279,9 +280,10 @@ def make_wasi_python(context, working_dir):
 def clean_contents(context):
     """Delete all files created by this script."""
     context.clean = True
-    if context.cross_build_path.exists():
-        _shared.log("🧹", f"Deleting {context.cross_build_path} ...")
-        shutil.rmtree(context.cross_build_path)
+    for path in [context.cross_build_path, context.dist_path]:
+        if path.exists():
+            _shared.log("🧹", f"Deleting {path} ...")
+            shutil.rmtree(path)
 
 
 @subdir("build_python_path")

Platforms/WASI/_package.py

@@ -264,18 +264,6 @@ def config_symlink(config_path, context):
     return [(symlink, config_path) for symlink in symlinks]
 
 
-def filename_stem(context):
-    """Calculate the stem of the archive file name."""
-    version_info = context.wasi_build_details["language"]["version_info"]
-    version = f"python-{version_info['major']}.{version_info['minor']}.{version_info['micro']}"
-    if version_info["releaselevel"] != "final":
-        version += version_info["releaselevel"][0] + str(
-            version_info["serial"]
-        )
-
-    return f"{version}-{context.host_triple}"
-
-
 def copy_files(files, base):
     for dest, src in files:
         target = base / dest
@@ -296,13 +284,13 @@ def gather(context):
     py_version = python_version(context)
     py_d_version = python_version(context, debug_ok=True)
 
-    dist = context.checkout / "dist"
+    dist = context.dist_path
     if dist.exists():
         _shared.log("🧹", f"Deleting {dist} ...")
         shutil.rmtree(dist)
 
     indent = "  "
-    base = dist / filename_stem(context)
+    base = context.archive_dir
     _shared.log("📝", f"Copying files to {base} ...")
 
     _shared.log("📁", "bin/", spacing=indent * 2)
@@ -365,12 +353,12 @@ def gather(context):
 
 
 def archive(context):
-    file_name = f"{filename_stem(context)}.tar.xz"
-    file_path = context.checkout / "dist" / file_name
+    file_name = f"{context.archive_stem}.tar.xz"
+    file_path = context.dist_path / file_name
     if file_path.exists():
         _shared.log("🧹", f"Deleting {file_path} ...")
         file_path.unlink()
-    to_compress = context.checkout / "dist" / filename_stem(context)
+    to_compress = context.archive_dir
     _shared.log("🗜️", f"Archiving to {file_path} ...")
     mtime_format = "%Y-%m-%dT%H:%M:%SZ"
     if source_date_epoch := os.environ.get("SOURCE_DATE_EPOCH"):