Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Fixed
- **`run`/`list` now work for every output language**: `discover_scripts()` previously only found `.py` files, so `reverse-api-engineer run <run_id>` failed with "No Python scripts found" for JS/TS/Go/Java/C#/PHP/Ruby/C clients. Discovery now covers all supported extensions (excluding build dirs and the vendored cJSON sources), and the run command dispatches to the right toolchain per language (compile+execute for C), with a clear error when the required tool isn't on PATH.
- **Windows-safe run-command quoting**: the Java/C#/PHP/Ruby/C run commands quoted paths with POSIX-only `shlex.quote`, which cmd.exe/PowerShell parse incorrectly for paths containing spaces. Paths are now quoted per-platform (`subprocess.list2cmdline` on Windows).

### Added
- **Go output language**: `output_language: "go"` is now supported alongside python/javascript/typescript, generating a standard-library-first (`net/http`, `encoding/json`) Go program, with the same auth-hardcoding/refresh and bot-detection-fallback guidance as the other languages.
- **Java output language**: `output_language: "java"` is now supported alongside python/javascript/typescript, generating a small Maven project using `java.net.http.HttpClient` (JDK 11+, no HTTP library dependency) and Gson for JSON, with the same auth-hardcoding/refresh guidance as the other languages.
Expand Down
55 changes: 36 additions & 19 deletions src/reverse_api/base_engineer.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
import asyncio
import os
import shlex
import subprocess
import sys
from abc import ABC, abstractmethod
from pathlib import Path
from typing import Any
Expand All @@ -13,7 +15,13 @@
from .session import SessionManager
from .sync import FileSyncWatcher, get_available_directory
from .tui import THEME_PRIMARY, THEME_SECONDARY, ClaudeUI
from .utils import generate_folder_name, get_docs_dir, get_history_path, get_scripts_dir
from .utils import (
OUTPUT_LANGUAGE_EXTENSIONS,
generate_folder_name,
get_docs_dir,
get_history_path,
get_scripts_dir,
)

DEBUG = os.environ.get("DEBUG", "0") == "1"

Expand All @@ -30,17 +38,9 @@
class BaseEngineer(ABC):
"""Abstract base class for API reverse engineering implementations."""

_OUTPUT_LANGUAGE_EXTENSIONS = {
"python": ".py",
"javascript": ".js",
"typescript": ".ts",
"go": ".go",
"java": ".java",
"csharp": ".cs",
"php": ".php",
"ruby": ".rb",
"c": ".c",
}
# Single source of truth lives in utils.OUTPUT_LANGUAGE_EXTENSIONS so
# script discovery and the run command dispatch stay in sync with codegen.
_OUTPUT_LANGUAGE_EXTENSIONS = OUTPUT_LANGUAGE_EXTENSIONS

def __init__(
self,
Expand Down Expand Up @@ -474,6 +474,23 @@ def _get_client_filename(self) -> str:
return "openapi.json"
return f"api_client{self._get_output_extension()}"

@staticmethod
def _quote_path(path) -> str:
"""Shell-quote a path for the platform's default shell.

shlex.quote is POSIX-only: cmd.exe/PowerShell pass its single quotes
through literally, so a spaced Windows path would break apart.
list2cmdline applies the double-quoting rules cmd.exe/CreateProcess
parse. Known boundary: PowerShell still expands `$` and backtick
inside double quotes, and no quoting satisfies cmd.exe and PowerShell
simultaneously for such paths — cmd-safe is the chosen baseline, and
paths whose components contain `$`/backtick are not supported on
Windows.
"""
if sys.platform == "win32":
return subprocess.list2cmdline([str(path)])
Comment on lines +490 to +491

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 PowerShell Paths Remain Unsafe

list2cmdline() applies Windows C-runtime/cmd.exe quoting, but the returned text is interpolated into a shell command that may run in PowerShell. A valid path containing PowerShell metacharacters such as $ or a backtick can therefore be expanded or escaped, causing the generated run command to use the wrong path or fail.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/reverse_api/base_engineer.py
Line: 485-486

Comment:
**PowerShell Paths Remain Unsafe**

`list2cmdline()` applies Windows C-runtime/cmd.exe quoting, but the returned text is interpolated into a shell command that may run in PowerShell. A valid path containing PowerShell metacharacters such as `$` or a backtick can therefore be expanded or escaped, causing the generated run command to use the wrong path or fail.

How can I resolve this? If you propose a fix, please make it concise.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — this is a real boundary, but not one that quoting can close: PowerShell expands $/backtick inside double quotes, while PowerShell-safe alternatives (single quotes, backtick escapes) are parsed literally/incorrectly by cmd.exe, and the executing shell isn't detectable from here. list2cmdline (cmd.exe/CreateProcess-safe) is the deliberate baseline; 21fc967 documents in the docstring that Windows paths containing $/backtick are unsupported. Happy to revisit if we ever pin the agent's shell on Windows.

return shlex.quote(str(path))

def _get_run_command(self) -> str:
"""Return the command to run the generated client."""
if self.output_language == "java":
Expand All @@ -495,7 +512,7 @@ def _get_run_command(self) -> str:
# exec:java invokes main() reflectively in-process, which fails
# on the package-private ApiClient class the Java partial
# requires ("symbolic reference class is not accessible").
pom = shlex.quote(str(self.scripts_dir.resolve() / "pom.xml"))
pom = self._quote_path(str(self.scripts_dir.resolve() / "pom.xml"))
return f"mvn -q -f {pom} compile exec:exec"
if self.output_language == "csharp":
# Unlike python/node/npx (which happily take a plain relative
Expand All @@ -512,7 +529,7 @@ def _get_run_command(self) -> str:
# re-interpreted against the agent's cwd (scripts_dir.parent.
# parent) instead of the original cwd it was relative to,
# pointing --project at the wrong, doubly-nested location.
csproj = shlex.quote(str(self.scripts_dir.resolve() / "ApiClient.csproj"))
csproj = self._quote_path(str(self.scripts_dir.resolve() / "ApiClient.csproj"))
return f"dotnet run --project {csproj}"
if self.output_language == "php":
# Full path, not a bare relative "php api_client.php": the
Expand All @@ -532,7 +549,7 @@ def _get_run_command(self) -> str:
# re-interpreted against the agent's cwd (scripts_dir.parent.
# parent) instead of the original cwd it was relative to,
# pointing this command at the wrong, doubly-nested location.
path = shlex.quote(str(self.scripts_dir.resolve() / self._get_client_filename()))
path = self._quote_path(str(self.scripts_dir.resolve() / self._get_client_filename()))
return f"php {path}"
if self.output_language == "ruby":
# Full path, not a bare relative "ruby api_client.rb": the
Expand All @@ -546,7 +563,7 @@ def _get_run_command(self) -> str:
# re-interpreted against the agent's cwd (scripts_dir.parent.
# parent) instead of the original cwd it was relative to,
# pointing this command at the wrong, doubly-nested location.
path = shlex.quote(str(self.scripts_dir.resolve() / self._get_client_filename()))
path = self._quote_path(str(self.scripts_dir.resolve() / self._get_client_filename()))
return f"ruby {path}"
if self.output_language == "c":
# Unlike every other language here, C needs an explicit compile
Expand All @@ -562,9 +579,9 @@ def _get_run_command(self) -> str:
# dir.parent.parent) instead of the original cwd it was relative
# to, pointing all three at the wrong, doubly-nested location.
resolved = self.scripts_dir.resolve()
source = shlex.quote(str(resolved / self._get_client_filename()))
cjson = shlex.quote(str(resolved / "cJSON.c"))
binary = shlex.quote(str(resolved / "api_client"))
source = self._quote_path(str(resolved / self._get_client_filename()))
cjson = self._quote_path(str(resolved / "cJSON.c"))
binary = self._quote_path(str(resolved / "api_client"))
return f"cc {source} {cjson} -lcurl -o {binary} && {binary}"
return {
"python": "python api_client.py",
Expand Down
73 changes: 71 additions & 2 deletions src/reverse_api/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -2663,6 +2663,27 @@ def _extract_missing_module(stderr: str) -> str | None:
return missing


def _run_non_python_script(script, script_args) -> None:
"""Run a non-Python generated client with its language's toolchain and exit."""
import shutil
import subprocess

from .utils import build_script_commands

try:
steps, tool = build_script_commands(script, script_args)
except ValueError as e:
raise click.ClickException(str(e)) from e
if shutil.which(tool) is None:
raise click.ClickException(f"cannot run {script.name}: '{tool}' is missing from PATH — install it and retry")
returncode = 0
for cmd in steps:
returncode = subprocess.run(cmd, cwd=str(script.parent)).returncode
if returncode != 0:
break
raise SystemExit(returncode)


def _run_script_machine_payload(
*,
identifier: str,
Expand Down Expand Up @@ -2691,7 +2712,7 @@ def _run_script_machine_payload(
identifier=identifier,
run_id=run_id,
scripts=scripts,
error=f"No Python scripts found for run {run_id}",
error=f"No runnable scripts found for run {run_id}",
error_kind_hint="engine_failure",
)

Expand Down Expand Up @@ -2729,6 +2750,50 @@ def _run_script_machine_payload(

emit_event("script_selected", run_id=run_id, script_path=str(script))

if script.suffix != ".py":
import shutil

from .utils import build_script_commands

try:
steps, tool = build_script_commands(script, script_args)
except ValueError as e:
return _build_run_payload(
identifier=identifier,
run_id=run_id,
script_path=str(script),
script_args=script_args,
scripts=scripts,
error=str(e),
error_kind_hint="misuse",
)
if shutil.which(tool) is None:
return _build_run_payload(
identifier=identifier,
run_id=run_id,
script_path=str(script),
script_args=script_args,
scripts=scripts,
error=f"cannot run {script.name}: '{tool}' is missing from PATH — install it and retry",
error_kind_hint="config_invalid",
)
result = None
for cmd in steps:
emit_event("process_started", run_id=run_id, script_path=str(script))
result = subprocess.run(cmd, cwd=str(script.parent), capture_output=True, text=True)
if result.returncode != 0:
break
return _build_run_payload(
identifier=identifier,
run_id=run_id,
script_path=str(script),
script_args=script_args,
returncode=result.returncode,
stdout=result.stdout or "",
stderr=result.stderr or "",
scripts=scripts,
)

from .utils import get_base_output_dir as _get_base

venv_dir = _get_base(output_dir) / ".venv"
Expand Down Expand Up @@ -2930,7 +2995,7 @@ def emit_event(event: str, **fields) -> None:
console.print(f"{scripts[0].parent}", style="dim")

if not scripts:
console.print(f"[red]No Python scripts found for run {run_id}[/red]")
console.print(f"[red]No runnable scripts found for run {run_id}[/red]")
raise SystemExit(1)

# --ls: just list and exit
Expand Down Expand Up @@ -2974,6 +3039,10 @@ def emit_event(event: str, **fields) -> None:
if script is None:
raise click.Abort()

# Non-Python clients: dispatch on extension, no venv involved
if script.suffix != ".py":
_run_non_python_script(script, script_args)

# Shared venv at ~/.reverse-api/runs/.venv (with requests pre-installed)
from .utils import get_base_output_dir as _get_base

Expand Down
1 change: 1 addition & 0 deletions src/reverse_api/prompts/partials/_language_c.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
- Reuse one `CURL` handle across requests rather than creating a new one per call
- Create a separate function for each distinct API endpoint, with a small struct for its response shape
- Check every `libcurl`/allocation return value; don't ignore errors
- Keep the code warning-clean under `-Wall -Wextra` — no unused parameters or variables
- Include a `main` function with example usage

**Authentication & credentials:**
Expand Down
102 changes: 93 additions & 9 deletions src/reverse_api/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,23 @@

from . import __version__

# One entry per supported output language. BaseEngineer's extension map is
# built from this, and discover_scripts()/build_script_commands() key off the
# extensions, so adding a language here is the single required registration.
OUTPUT_LANGUAGE_EXTENSIONS = {
"python": ".py",
"javascript": ".js",
"typescript": ".ts",
"go": ".go",
"java": ".java",
"csharp": ".cs",
"php": ".php",
"ruby": ".rb",
"c": ".c",
}

SCRIPT_EXTENSIONS = frozenset(OUTPUT_LANGUAGE_EXTENSIONS.values())


def check_for_updates() -> str | None:
"""Check PyPI for newer version.
Expand Down Expand Up @@ -631,7 +648,7 @@ def resolve_run(identifier: str, session_manager, *, interactive: bool = True) -


def discover_scripts(run_id: str, output_dir: str | None = None, run_metadata: dict | None = None) -> list[Path]:
"""Find all executable Python scripts in a run's script directory.
"""Find all runnable generated scripts in a run's script directory.

Tries the stored script path from run metadata first, then falls back
to the current output_dir config.
Expand All @@ -642,7 +659,9 @@ def discover_scripts(run_id: str, output_dir: str | None = None, run_metadata: d
run_metadata: Optional run dict with paths.script_path to resolve from

Returns:
Sorted list of .py file Paths (excludes __pycache__, .venv, __init__.py)
Sorted list of script Paths in any supported output language
(excludes build/venv directories and support files like __init__.py
and the vendored cJSON sources)

Raises:
ValueError: If run_id contains invalid characters
Expand Down Expand Up @@ -672,18 +691,83 @@ def discover_scripts(run_id: str, output_dir: str | None = None, run_metadata: d
if not scripts_dir.exists():
return []

exclude_dirs = {"__pycache__", ".venv"}
exclude_files = {"__init__.py"}
# Support files that share a script extension but are never the client
# itself: package markers and the vendored cJSON library (C output).
# Build/venv directories (__pycache__, .venv, node_modules, bin, obj,
# target) need no explicit filter: iterdir() doesn't recurse, and
# matching names in *parent* components (e.g. a custom output_dir under
# /tmp/target) must not exclude anything.
exclude_files = {"__init__.py", "cJSON.c", "cJSON.h"}

scripts = []
for f in scripts_dir.iterdir():
if f.is_file() and f.suffix == ".py" and f.name not in exclude_files:
scripts.append(f)
scripts = [s for s in scripts if not any(part in exclude_dirs for part in s.parts)]
scripts = [
f
for f in scripts_dir.iterdir()
if f.is_file() and f.suffix in SCRIPT_EXTENSIONS and f.name not in exclude_files
]

return sorted(scripts, key=lambda p: p.name)


def build_script_commands(script: Path, script_args: tuple[str, ...] = ()) -> tuple[list[list[str]], str]:
"""Build the subprocess command(s) that run a non-Python generated client.

Python clients are executed by the caller's shared-venv flow and are not
handled here. Commands use absolute paths so they work from any cwd; run
them with cwd=script.parent so relative artifacts (cookie jars, build
output) land next to the script.

Args:
script: Path to the client script (any supported non-.py extension)
script_args: Extra arguments to pass through to the client

Returns:
(steps, tool): ordered argv lists to run (C compiles then executes,
everything else is a single step) and the executable that must be on
PATH for them to work.

Raises:
ValueError: For unsupported extensions, or script_args with a Java
client (mvn exec's program arguments are fixed in the pom).
"""
# Resolve before building argv: callers run these with cwd=script.parent,
# and a relative script path (from a relative output_dir) would otherwise
# be re-resolved by the child against that new cwd and fail to start.
script = script.resolve()
d = script.parent

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Resolve script paths before changing cwd

If the configured output_dir is relative, discover_scripts() returns a relative script, but both non-Python runners execute the argv with cwd=script.parent; e.g. out/scripts/<run>/api_client.js is then resolved by node from out/scripts/<run> as out/scripts/<run>/out/scripts/<run>/api_client.js, so every non-Python client fails to start. Build commands from script.resolve() or pass only the filename once the cwd is changed.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 21fc967build_script_commands() now resolves the script path before building argv, so commands stay absolute regardless of a relative output_dir, and cwd=script.parent can no longer re-anchor them. Verified live with a relative path + node subprocess, plus a regression test.

suffix = script.suffix
if suffix == ".js":
return [["node", str(script), *script_args]], "node"
if suffix == ".ts":
return [["npx", "tsx", str(script), *script_args]], "npx"
if suffix == ".go":
return [["go", "run", str(script), *script_args]], "go"
if suffix == ".java":
if script_args:
raise ValueError(
"script arguments are not supported for Java clients: "
"mvn exec:exec's program arguments are fixed in the pom.xml"
)
return [["mvn", "-q", "-f", str(d / "pom.xml"), "compile", "exec:exec"]], "mvn"
if suffix == ".cs":
cmd = ["dotnet", "run", "--project", str(d / "ApiClient.csproj")]
if script_args:
cmd += ["--", *script_args]
return [cmd], "dotnet"
if suffix == ".php":
return [["php", str(script), *script_args]], "php"
if suffix == ".rb":
return [["ruby", str(script), *script_args]], "ruby"
if suffix == ".c":
binary = d / script.stem
compile_cmd = ["cc", str(script)]
cjson = d / "cJSON.c"
if cjson.exists():
compile_cmd.append(str(cjson))
compile_cmd += ["-lcurl", "-o", str(binary)]
return [compile_cmd, [str(binary), *script_args]], "cc"
raise ValueError(f"unsupported script type: {script.name}")


def extract_domain_from_har(har_path: Path) -> str | None:
"""Extract the primary domain from a HAR file.

Expand Down
Loading