diff --git a/.claude/architecture/monorepo.md b/.claude/architecture/monorepo.md new file mode 100644 index 0000000..4b05b22 --- /dev/null +++ b/.claude/architecture/monorepo.md @@ -0,0 +1,92 @@ +# Monorepo Mechanics + +## Workspaces + +```json +"workspaces": ["packages/*", "!packages/native-app-wrapper", "grab-help-docs"] +``` + +Note the **negation**: `packages/native-app-wrapper` is deliberately excluded. +It is a Tauri scaffold with a Rust toolchain requirement — it is not installed +by a root `npm install` and nothing at the root builds or tests it. + +`pnpm-workspace.yaml` lists the same globs *without* the negation, for anyone +using pnpm. + +## Package manager + +**npm.** `packageManager` pins `npm@11.19.1`, the committed lockfile is +`package-lock.json`, and CI runs `npm install` (tests) and `npm ci` (Pages). +`.npmrc` sets `package-manager-strict=false`. + +A `pnpm-workspace.yaml` exists but there is **no pnpm lockfile**, so pnpm is +tolerated rather than supported. Do not switch, and do not commit a second +lockfile — the lockfile is what CI installs from. + +This is the one repo in this family that is not on Bun. `bun x standard-version` +appears in the `ship` script, and `npm-publish.yml` sets up Bun, but installs +and tests are npm. + +## Two kinds of package + +| Kind | Directories | What "publishing" means | +| --- | --- | --- | +| **Internal** | `grab-api`, `grab-url-cli`, `log-json` (`@grab-url/*`, all `"private": true`) | Never published. Compiled into `grab-url`'s `dist/` by the root build. | +| **Published** | `api2client`, `archiver-web`, `loading-animations`, `quantum-sphere-loading-animation` | Published on their own **and** bundled into a `grab-url` entry | +| **Excluded** | `native-app-wrapper` | Outside the workspace; its own thing | + +Consequence: editing `packages/grab-api/src` changes the `grab-url` package. +There is no separate `@grab-url/grab-api` for a consumer to install, so its +"public API" is really `grab-url`'s. + +## Turbo + +`turbo.json` defines exactly one task: + +```json +"build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**"] } +``` + +Turbo is used for one thing here — building `grab-help-docs` +(`npm run make:docs` → `turbo run build --filter=grab-help-docs`). Everything +else runs through root npm scripts and the single Vite build. Don't reach for +turbo filters expecting the other repos' task graph; it isn't there. + +## Tests + +All tests live in the **root `test/` folder** and run under Vitest through the +same `vite.config.ts` as the build — so they see the same aliases and externals +the shipped bundle does. + +``` +test/grab.test.ts test/downloader.test.ts test/ytdlp.test.ts +test/api2client.test.ts test/archiver.test.ts test/aria2.test.ts +test/command.test.ts test/icon.test.ts test/log.test.ts +test/page-archive.test.ts +``` + +There are no per-package test folders. A new test for +`packages/whatever/src/thing.ts` goes in `test/thing.test.ts`. + +Coverage (v8) includes `packages/**/src/**` and excludes `dist`, `.d.ts`, +Svelte sources, `svg/` and `demo/` directories. + +```bash +npm run test +npm run test:coverage # what CI runs +npm run test:ui +npm run test:cli # a real download against a live Ubuntu ISO URL +``` + +`test:cli` hits the network and downloads a large file. It is a smoke test, not +part of the suite — don't wire it into CI. + +## `postinstall` downloads yt-dlp + +`scripts/install-yt-dlp.mjs --postinstall` runs on every install. If an install +appears to hang or fails behind a proxy, that is where to look: + +```bash +npm run ytdlp # force a re-download +npm run ytdlp:sidecar # fetch the sidecar binary for a packaged app +``` diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..69d4c8a --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,21 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "permissions": { + "allow": [ + "Bash(npm install)", + "Bash(npm ci)", + "Bash(npm run build:*)", + "Bash(npm run test:*)", + "Bash(npm run make:*)", + "Bash(npx turbo run:*)", + "Bash(git status:*)", + "Bash(git diff:*)", + "Bash(git log:*)" + ], + "deny": [ + "Read(./**/.env)", + "Read(./**/.env.*)", + "Read(./**/.dev.vars)" + ] + } +} diff --git a/.gitignore b/.gitignore index c00e1f7..3628d8d 100644 --- a/.gitignore +++ b/.gitignore @@ -50,3 +50,10 @@ build.log pnpm-lock.yaml .turbo grab-help-docs/.next_old + +.claude/* +# Agent docs for Claude Code — checked in on purpose (see CLAUDE.md) +!.claude/architecture +!.claude/settings.json +.claude/settings.local.json +CLAUDE.local.md diff --git a/grab-help-docs/CLAUDE.md b/grab-help-docs/CLAUDE.md new file mode 100644 index 0000000..680dfe0 --- /dev/null +++ b/grab-help-docs/CLAUDE.md @@ -0,0 +1,32 @@ +# CLAUDE.md — `grab-help-docs` + +The documentation site for `grab-url`: a Fumadocs site, and a workspace (it is +listed explicitly in the root `workspaces` array). + +## It deploys as a static export, not `next build` + +`.github/workflows/pages.yml` runs `npm ci` and then: + +```bash +node grab-help-docs/scripts/build-static-pages.mjs +``` + +and publishes the result to GitHub Pages. **That script is what ships** — if the +deployed site and a local dev run disagree, trust the static export. Build +through it before claiming a docs change works. + +```bash +npm run make:docs # turbo run build --filter=grab-help-docs +``` + +## One page is generated — don't edit it + +`content/docs/claude-skill.mdx` is generated from +`skills/use-grab-request/SKILL.md` by `scripts/sync-skill-docs.mjs`, and says so +in a comment at the top. Edit the skill, then `npm run make:skill`. +`node scripts/sync-skill-docs.mjs --check` fails when it is stale. + +## Not to be confused with + +The root `docs/` directory — a small static landing page for grab.js.org +(`index.html`, `_config.yml`). Documentation does not go there. diff --git a/packages/api2client/CLAUDE.md b/packages/api2client/CLAUDE.md new file mode 100644 index 0000000..79fc9b3 --- /dev/null +++ b/packages/api2client/CLAUDE.md @@ -0,0 +1,34 @@ +# CLAUDE.md — `api2client` + +**Published on its own** — and also reachable through the `grab-url` build. + +A [Hey API](https://heyapi.dev) client that sends generated OpenAPI SDK requests +through **`grab-url`** instead of fetch or axios, so every generated operation +inherits caching, retries, rate limiting and request dedupe. + +## The alias that makes this work + +`vite.config.ts` aliases the bare specifier `"grab-url"` to +`packages/grab-api/src/index.ts`, because the generated Hey API client imports +the **published package name**. Inside the monorepo that has to resolve to the +same source — otherwise a build ends up with two copies of the client and the +cache/dedupe layers stop being shared. + +If you change how this package imports `grab-url`, check that alias. + +## Rules + +- **Generated SDK code is output.** This package is the *adapter*, not the + generated client — fix behaviour here, never by editing someone's generated + SDK. +- Preserve the Hey API client contract, including its error shape. Consumers + generated against Hey API expect it. +- The point of the package is that operations get `grab-url`'s guarantees + automatically. Don't add a path that falls back to bare `fetch`. + +## Layout + +`src/index.ts` · `src/client.ts` · `src/generate.ts` · `src/cli.ts` · +`src/core/` · `src/types.ts` · `src/utils.ts` + +Tests: `test/api2client.test.ts`. Example: `examples/api2client-petstore`. diff --git a/packages/archiver-web/CLAUDE.md b/packages/archiver-web/CLAUDE.md new file mode 100644 index 0000000..bc18ac3 --- /dev/null +++ b/packages/archiver-web/CLAUDE.md @@ -0,0 +1,33 @@ +# CLAUDE.md — `archiver-web` + +**Published on its own**, and built into three `grab-url` entries: +`archiver-web`, `bin-extract`, `bin-compress`. + +A universal archive extractor and creator on **JSZip** — frontend-capable, so it +runs in a browser as well as in Node. + +## Safety is the main design constraint + +- **Zip slip.** An archive entry named `../../etc/thing` must never be written + outside the destination directory. Normalize and verify every entry path + against the resolved destination before writing — not after. +- **Zip bombs.** A small archive can expand to gigabytes. Respect and keep any + size/entry-count limits; don't remove one to make a large legitimate file + work. +- Symlink entries and absolute paths in archives are both traversal vectors. +- These rules apply doubly because two of the three entries are **executable + bins** (`bin-extract`, `bin-compress`) that users point at untrusted files. + +## Build notes + +- `jszip` is **externalized** — the caller provides it. +- `archiver-web` is externalized from the **slim** `grab-url` entry, so the slim + build doesn't carry it. Keep the import surface externalizable. +- The `bin-*` chunks get the `#!/usr/bin/env node` banner from the root build, + not from source. + +## Layout + +`src/index.ts` · `src/bin-extract.ts` · `src/bin-compress.ts` · `src/types.ts` + +Tests: `test/archiver.test.ts`. diff --git a/packages/grab-api/CLAUDE.md b/packages/grab-api/CLAUDE.md new file mode 100644 index 0000000..cdfdcb4 --- /dev/null +++ b/packages/grab-api/CLAUDE.md @@ -0,0 +1,41 @@ +# CLAUDE.md — `@grab-url/grab-api` + +**Private — never published.** This package *is* `grab-url`: the root Vite build +compiles `src/index.ts` into `dist/grab-api.*` and the root `package.json` +exposes it as the package's main entry. + +So its public API is `grab-url`'s public API. There is no separate +`@grab-url/grab-api` for anyone to install, and a breaking change here is a +breaking change to the published package. + +## Two entries, and the difference is the point + +| Source | Ships as | Difference | +| --- | --- | --- | +| `src/index.ts` | `grab-url` | Everything bundled | +| `src/index.slim.ts` | `grab-url/slim` | `archiver-web` and `linkedom` externalized | + +The slim build exists so a browser consumer doesn't pay for the heavy +dependencies. **Anything you add to `index.ts` that the slim entry also imports +must stay externalizable**, or `slim` quietly stops being slim. Check +`slimExternalPkgs` in `vite.config.ts` when adding a dependency. + +## What the client guarantees + +Caching, retries, rate limiting and request dedupe on every call — that is the +reason to use it over `fetch`. Consumers (including `debate-api-client` in the +sibling debate repo, and `api2client` here) rely on those behaviours being +automatic. Don't add a path that bypasses them. + +## Layout + +`src/core/` · `src/common/` · `src/response/` · `src/devtools/` · +`src/index.ts` · `src/index.slim.ts` + +## Rules + +- **It must run in a browser.** No Node builtins on the library path — the CLI + is a different entry, with a different externals list. +- Tests live in the **root `test/`** folder (`test/grab.test.ts`), not here. +- After changing it: `npm run build`, then confirm both `dist/grab-api.*` and + `dist/grab-api-slim.*` are produced. diff --git a/packages/grab-url-cli/CLAUDE.md b/packages/grab-url-cli/CLAUDE.md new file mode 100644 index 0000000..efe0876 --- /dev/null +++ b/packages/grab-url-cli/CLAUDE.md @@ -0,0 +1,46 @@ +# CLAUDE.md — `@grab-url/cli` + +**Private — never published.** Built into `dist/grab-url-cli.*` and installed as +the `grab-url`, `grab` and `g` bins of the `grab-url` package. + +## Things that bite + +- **The shebang comes from the build, not the source.** `vite.config.ts` adds + `#!/usr/bin/env node` via `rollupOptions.output.banner` for this chunk and the + `bin-*` ones. Remove it and the bins stop being executable. +- **`extract-webpage` must stay a runtime `import()`.** It backs `--page`, it is + an *optional* peer dependency, and it pulls in jsdom/linkedom. It is + externalized on purpose — a static import would drag a DOM implementation into + every CLI install. `inlineDynamicImports: false` exists to keep that dynamic. +- **yt-dlp is an external binary**, fetched by `scripts/install-yt-dlp.mjs` at + postinstall. Media-site URLs route through it. It may be missing, outdated, or + blocked — fail with a message that says how to fix it (`npm run ytdlp`), not a + stack trace. + +## Safety — this writes to the user's filesystem from a URL they typed + +- **Never write outside the target directory.** Path traversal from a + server-supplied filename is the classic download-tool vulnerability; sanitize + the name, don't trust `Content-Disposition`. +- Don't follow a redirect into a local/private address on a user-supplied URL. +- Don't overwrite an existing file without saying so. + +## Layout + +`src/index.ts` · `src/cli-args.ts` · `src/file-downloader.ts` · +`src/download-spinners.ts` · `src/keyboard-controls.ts` · `src/cancel-state.ts` +· `src/background.ts` · `src/display/` · `src/page/` · `src/transfer/` + +Cancellation is real state (`cancel-state.ts`, `keyboard-controls.ts`) — a +partially written file must be cleaned up or resumable, not left as a plausible +looking truncated download. + +## Testing + +Tests are in the root `test/` folder (`downloader.test.ts`, `command.test.ts`, +`ytdlp.test.ts`, `aria2.test.ts`, `page-archive.test.ts`). + +```bash +npm run test +npm run test:cli # a real end-to-end download — run by hand, not in CI +``` diff --git a/packages/loading-animations/CLAUDE.md b/packages/loading-animations/CLAUDE.md new file mode 100644 index 0000000..1aaede5 --- /dev/null +++ b/packages/loading-animations/CLAUDE.md @@ -0,0 +1,33 @@ +# CLAUDE.md — `loading-animations` + +**Published on its own**, and built into `dist/animations.*` (exposed as +`grab-url/animations`). + +Two halves, one package: **tree-shakable SVG spinners** (`src/svg/`) for the web +and **terminal spinners** (`src/cli/`) for the CLI. + +## Tree-shakability is the feature + +`src/svg/index.ts` is a **generated barrel**. It is produced by: + +```bash +npm run make:icons +# npx export-svg-typescript -i packages/loading-animations/src/svg -o packages/loading-animations/src/svg/index.ts +``` + +So: **add or edit the `.svg` files, then regenerate the barrel.** Hand-editing +`index.ts` is overwritten on the next `npm run make`. + +One export per animation, no module-scope side effects — a consumer importing +one spinner must not pull in all of them. + +## Build notes + +The `svg/` directory is excluded from coverage (it is generated assets, not +logic). The CLI spinners are the tested half. + +## Layout + +`src/svg/` (the SVGs + the generated barrel) · `src/cli/` (terminal spinners) + +Tests: `test/icon.test.ts`. diff --git a/packages/log-json/CLAUDE.md b/packages/log-json/CLAUDE.md new file mode 100644 index 0000000..b7f576e --- /dev/null +++ b/packages/log-json/CLAUDE.md @@ -0,0 +1,24 @@ +# CLAUDE.md — `@grab-url/log` + +**Private — never published.** Built into `dist/log.*` and exposed as +`grab-url/log`. + +A JSON logger with structured output and terminal colors. + +## Rules + +- **It is used by the CLI and the library both.** Colors must degrade: no ANSI + when the output is not a TTY, when `NO_COLOR` is set, or when the consumer is + capturing JSON. A logger that emits escape codes into a piped JSON stream + breaks every caller that parses it. +- **Never log credentials, tokens, or full URLs with query strings** — the + client this ships with sends authenticated requests, and a logged URL is a + logged API key. +- Structured output is the contract (`structure.ts`). Adding or renaming a field + changes what consumers parse. + +## Layout + +`src/log-json.ts` (entry) · `src/structure.ts` · `src/colors.ts` + +Tests: `test/log.test.ts` in the root test folder. diff --git a/packages/native-app-wrapper/CLAUDE.md b/packages/native-app-wrapper/CLAUDE.md new file mode 100644 index 0000000..d42a165 --- /dev/null +++ b/packages/native-app-wrapper/CLAUDE.md @@ -0,0 +1,29 @@ +# CLAUDE.md — `native-app-wrapper` + +**Excluded from the workspace on purpose.** The root `workspaces` array carries +an explicit negation: + +```json +"workspaces": ["packages/*", "!packages/native-app-wrapper", "grab-help-docs"] +``` + +So a root `npm install` does not install it, the root Vite build does not build +it, and the root test run never reaches it. Nothing at the root will tell you +you broke it. + +A Tauri scaffold that packages a website, or a bundled CLI, as a native desktop +app. + +## Things that bite + +- **Rust toolchain required.** A machine that builds the rest of this repo + cannot necessarily build this. +- **yt-dlp ships as a Tauri sidecar** here. Sidecar binaries are referenced with + platform-suffixed names — get one wrong and it fails on exactly one OS, at + runtime, after packaging. `npm run ytdlp:sidecar` fetches it. +- Native behaviour (window, tray, updater, deep links) lives on the Rust side, + not in the web layer. +- A green build on one platform says nothing about the others. + +Install and build from inside this directory, and say in the PR how you verified +it. diff --git a/packages/quantum-sphere-loading-animation/CLAUDE.md b/packages/quantum-sphere-loading-animation/CLAUDE.md new file mode 100644 index 0000000..ae2d5c8 --- /dev/null +++ b/packages/quantum-sphere-loading-animation/CLAUDE.md @@ -0,0 +1,34 @@ +# CLAUDE.md — `quantum-sphere-loading-icon` + +**npm name:** `quantum-sphere-loading-icon` (directory: +`quantum-sphere-loading-animation`). Published on its own, and built into +`dist/quantum-sphere.*` (exposed as `grab-url/icons/quantum-sphere`). + +A parabolic spherical orbital loading component for **React and Svelte**. + +## The two build rules that must not be broken + +Both are enforced in the root `vite.config.ts`, and both exist because of a real +failure: + +1. **React is externalized.** `react`, `react-dom`, `react/jsx-runtime` and + `react/jsx-dev-runtime` never enter the bundle. A second React copy makes + every hook in `QuantumOrbital` throw *"Invalid hook call"*. No other entry + imports React, so this costs nothing elsewhere. +2. **`"use client"` is re-applied in `generateBundle`.** Rollup drops the + source's module-level directive, and a `banner` does not survive terser, + which re-parses the chunk and discards a directive it reads as dead code. + Writing it after minification is the only point where it sticks. Without it, + a React Server Component importing the sphere fails on the first hook. + +If you are debugging "Invalid hook call" or an RSC boundary error from this +component, those two are where to look — not in the component. + +## Layout + +`src/icons.ts` (the built entry) · `src/index.ts` · `src/react/` · `src/svelte/` +· `src/shared/` · `src/types/` + +The Svelte sources and `demo/` are excluded from the type build and from +coverage. Keep genuinely shared logic in `src/shared/` so the two framework +wrappers stay thin.