From d9186684b51bfba6b79a0b65edfbdc28218be1e0 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 12:04:08 +0000 Subject: [PATCH 1/2] =?UTF-8?q?=F0=9F=93=9D=20docs(claude):=20add=20.claud?= =?UTF-8?q?e/=20agent=20docs=20and=20per-package=20CLAUDE.md=20files?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopt the qwksearch-research-agent layout: a root CLAUDE.md operating manual backed by .claude/architecture/ deep notes, plus one CLAUDE.md per package. - .claude/architecture/{overview,build,monorepo,documentation,conventions}.md - CLAUDE.md in all 8 packages/ directories and in grab-help-docs - .claude/settings.json pre-approving the repo's own build/test commands and denying reads of .env / .dev.vars The notes record what is specific to this repo rather than restating the README: this is a monorepo in layout but a single published package in practice, npm is the package manager (not bun, unlike its siblings), most packages/* are private internals compiled into grab-url's dist, all tests live in the root test/ folder, and CI never runs the build so a broken vite.config.ts passes. build.md documents the packaging decisions that look removable and are not — externalizing extract-webpage and React, restoring "use client" in generateBundle after terser, and the shebang banner the bins depend on. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EoGK1G8bnF6riGQekWLtYr --- .claude/architecture/build.md | 81 ++++++++++++++++ .claude/architecture/conventions.md | 85 +++++++++++++++++ .claude/architecture/documentation.md | 43 +++++++++ .claude/architecture/monorepo.md | 92 +++++++++++++++++++ .claude/architecture/overview.md | 61 ++++++++++++ .claude/settings.json | 21 +++++ .gitignore | 8 +- CLAUDE.md | 82 +++++++++++++++++ grab-help-docs/CLAUDE.md | 32 +++++++ packages/api2client/CLAUDE.md | 34 +++++++ packages/archiver-web/CLAUDE.md | 33 +++++++ packages/grab-api/CLAUDE.md | 41 +++++++++ packages/grab-url-cli/CLAUDE.md | 46 ++++++++++ packages/loading-animations/CLAUDE.md | 33 +++++++ packages/log-json/CLAUDE.md | 24 +++++ packages/native-app-wrapper/CLAUDE.md | 29 ++++++ .../CLAUDE.md | 34 +++++++ 17 files changed, 778 insertions(+), 1 deletion(-) create mode 100644 .claude/architecture/build.md create mode 100644 .claude/architecture/conventions.md create mode 100644 .claude/architecture/documentation.md create mode 100644 .claude/architecture/monorepo.md create mode 100644 .claude/architecture/overview.md create mode 100644 .claude/settings.json create mode 100644 CLAUDE.md create mode 100644 grab-help-docs/CLAUDE.md create mode 100644 packages/api2client/CLAUDE.md create mode 100644 packages/archiver-web/CLAUDE.md create mode 100644 packages/grab-api/CLAUDE.md create mode 100644 packages/grab-url-cli/CLAUDE.md create mode 100644 packages/loading-animations/CLAUDE.md create mode 100644 packages/log-json/CLAUDE.md create mode 100644 packages/native-app-wrapper/CLAUDE.md create mode 100644 packages/quantum-sphere-loading-animation/CLAUDE.md diff --git a/.claude/architecture/build.md b/.claude/architecture/build.md new file mode 100644 index 0000000..8b1b4fd --- /dev/null +++ b/.claude/architecture/build.md @@ -0,0 +1,81 @@ +# The Build + +One `vite build` at the repo root produces every published entry. `vite.config.ts` +is therefore the most consequential file in the repo: it is the packaging, the +module boundaries and the runtime contract all at once. + +Its comments record real production failures. Read them before editing, and do +not remove any of the following. + +## Aliases: how `packages/*` resolve + +```ts +"@grab-url/log" → packages/log-json/src/log-json.ts +"@grab-url/grab-api" → packages/grab-api/src/index.ts +"grab-url" → packages/grab-api/src/index.ts +``` + +That last one matters: the Hey API client imports the **published package +name**, and inside the monorepo it has to resolve to the same source. Without +it you get two copies of the client in one bundle. + +## Externals — three separate reasons + +| Externalized | Why | +| --- | --- | +| Node builtins (`node:*` and a long explicit list) | The library runs in browsers too | +| `chalk`, `cli-table3`, `cli-progress`, `cli-spinners` | CLI-only deps, kept out of library bundles | +| **`extract-webpage`** | The optional peer behind `grab-url --page`. It is loaded via runtime `import()` and pulls in jsdom/linkedom. **Bundling it would drag a DOM implementation into the CLI.** | +| **`react`, `react-dom`, `react/jsx-runtime`, `react/jsx-dev-runtime`** | A second React copy makes every hook in `QuantumOrbital` throw *"Invalid hook call"*. No other entry imports React, so this costs nothing elsewhere. | +| `jszip` | Heavy; the archive tooling's caller provides it | +| `archiver-web`, `linkedom` — **slim entry only** | The point of `grab-url/slim`: the same client without the heavy deps. Externalized only when the importer is `index.slim`. | + +## The two output hacks + +Both exist because the obvious approach silently fails. + +**1. `"use client"` on the quantum-sphere bundles.** Rollup drops the source +file's module-level directive when bundling, and a `banner` does not survive +either — terser re-parses the chunk afterwards and discards a directive it reads +as dead code in an ES module. Writing it in `generateBundle`, which runs *after* +minification, is the one point where it sticks. Without it, a React Server +Component importing the sphere fails on the first hook. + +**2. The shebang banner.** `rollupOptions.output.banner` adds +`#!/usr/bin/env node` to `grab-url-cli` and any `bin-*` chunk. The bins in +`package.json` point at these files directly; without the shebang they are not +executable. + +Also note `inlineDynamicImports: false` — the runtime `import()` of +`extract-webpage` depends on dynamic imports staying dynamic. + +## Output shape + +- Formats: **ES and CJS**, named `..js`. +- Target `es2022`, minified with **terser**, sourcemaps on. +- Types via `vite-plugin-dts` over `packages/**/*.ts(x)`, excluding the + quantum-sphere Svelte, demo and dist directories. + +## After changing the build + +```bash +npm run build +ls dist/ # the entry you touched must be there, in both formats +``` + +A missing or renamed `dist` file is a broken `exports` map, which consumers hit +at import time and no test here will catch. + +## `npm run make` + +The full pipeline, in order: + +``` +make:icons → export-svg-typescript over packages/loading-animations/src/svg +make:skill → scripts/sync-skill-docs.mjs (skill → docs page) +make:docs → turbo build --filter=grab-help-docs +build → vite build +``` + +Use it when you changed icons, the skill or the docs — `npm run build` alone +skips all three. diff --git a/.claude/architecture/conventions.md b/.claude/architecture/conventions.md new file mode 100644 index 0000000..cf93996 --- /dev/null +++ b/.claude/architecture/conventions.md @@ -0,0 +1,85 @@ +# Conventions and General Rules + +## Language and style + +- TypeScript, ESM (`"type": "module"`) throughout. A few `.mjs` scripts under + `scripts/`. +- Match the surrounding file's style — naming, import order, comment density. + There is no repo-wide formatter enforcing it in CI. +- **Comments explain why, not what.** The best comments here record a failure + and its cause: why the `"use client"` directive has to be written in + `generateBundle`, why `extract-webpage` must stay external, why React is + externalized for one entry. `vite.config.ts` and + `scripts/sync-skill-docs.mjs` are the house style — copy that register. +- The library runs in **browsers, Node and CLIs**. Before reaching for a Node + API in `packages/grab-api`, check which entry it ends up in. + +## Commits + +Gitmoji + conventional commits, lowercase subject, imperative mood: + +``` +✨ feat(cli): archive a page into a folder with --page +✨ feat(cli): route media-site URLs through yt-dlp +📝 docs(skill): cover api2ai, the transfer CLI, and the rest of the surface +💄 feat(homepage): show the full README badge set under the hero tagline +``` + +Scope is the package or surface name. + +## Pull requests + +- Target `master`. One concern per PR; no drive-by refactors. +- Say what changed, why, and **which `dist` entries it affects** — a change in + `packages/grab-api` ships to every `grab-url` consumer. +- Say whether the published API changed and whether a version bump is needed. +- Include test results. If you changed the CLI, say whether you ran + `npm run test:cli`. +- If you changed the skill, say you ran `npm run make:skill`. + +## Tests + +- Add or update tests for every behaviour change and bug fix. +- **All tests live in the root `test/` folder** — there are no per-package test + directories. See [monorepo.md](monorepo.md#tests). +- `npm run test:coverage` is exactly what CI runs. +- Tests run through the same `vite.config.ts` as the build, so they see the + real aliases and externals. That is deliberate: a test that passes against + bundled behaviour is worth more here than one against raw source. +- `npm run test:cli` performs a real network download. Run it by hand for CLI + changes; do not add it to CI. + +## CI + +| Workflow | Trigger | What it guards | +| --- | --- | --- | +| `tests.yml` | push to `master`, PR | Node 20, `npm install`, `npm run test:coverage`, upload to Codecov | +| `npm-publish.yml` | push to `master` | Iterates `packages/*` and publishes each **non-private** one, then commits version bumps | +| `pages.yml` | push to `master` | Node 22, `npm ci`, `node grab-help-docs/scripts/build-static-pages.mjs`, deploy to GitHub Pages | + +Note what CI does **not** do: it never runs `npm run build`. A change that +breaks the Vite build passes CI. Build locally before you push. + +## Publishing + +- The root package `grab-url` is the main artifact; `npm run ship` is the manual + release path (`standard-version` patch, then `npm publish`), and + `prepublishOnly` forces a build first. +- `npm-publish.yml` publishes the **non-private** `packages/*` — `api2client`, + `archiver-web`, `loading-animations`, `quantum-sphere-loading-icon`. +- `@grab-url/*` packages are `"private": true` and must stay that way: they are + bundled into `grab-url`, and publishing them would create two copies of the + same code in the wild. + +## Security + +- Never commit secrets or tokens. +- **The CLI downloads and writes files from user-supplied URLs.** Path traversal + out of the target directory, following a redirect to a local address, and + archive extraction that escapes its destination ("zip slip") are the three + failure modes to keep tested. `archiver-web` and the downloader both touch + this. +- yt-dlp is fetched at install time by `scripts/install-yt-dlp.mjs`. Keep the + download pinned and verified; never execute an arbitrary URL's payload. +- `extract-webpage` parses hostile HTML. It stays an optional runtime import for + that reason as well as for bundle size. diff --git a/.claude/architecture/documentation.md b/.claude/architecture/documentation.md new file mode 100644 index 0000000..ec0aa3c --- /dev/null +++ b/.claude/architecture/documentation.md @@ -0,0 +1,43 @@ +# Documentation + +| If it is… | It goes in… | +| --- | --- | +| A guide for users of `grab-url` | `grab-help-docs/content/docs` | +| How an agent should *use* `grab-url` | `skills/use-grab-request/SKILL.md` | +| How an agent should *work on* a package | That package's `CLAUDE.md` | +| Repo-wide agent orientation | root `CLAUDE.md` + `.claude/architecture/` | + +## The skill is the source of truth for its docs page + +`grab-help-docs/content/docs/claude-skill.mdx` is **generated** from +`skills/use-grab-request/SKILL.md` by `scripts/sync-skill-docs.mjs`. The +generated file carries a comment saying so. Edit the skill, then: + +```bash +npm run make:skill # write +node scripts/sync-skill-docs.mjs --check # fail if stale +``` + +`npm run make` runs `make:skill` before the docs build, so the full pipeline +keeps them in sync — but a bare `npm run build` does not. + +The skill is what agents load to use this library. When the client's public +behaviour or the CLI's flags change, updating it is part of the change, not a +follow-up. + +## The docs site — `grab-help-docs` + +A Fumadocs site, and a workspace. It is **statically exported and deployed to +GitHub Pages** by `.github/workflows/pages.yml`, which runs +`node grab-help-docs/scripts/build-static-pages.mjs` after `npm ci` — not a +plain `next build`. If the deployed site and a local dev run disagree, the +static export path is the one that ships. + +```bash +npm run make:docs # turbo run build --filter=grab-help-docs +``` + +## `docs/` is not the docs site + +The root `docs/` directory is a small static landing page (`index.html`, +`_config.yml`, `README.md`) for grab.js.org. Do not put documentation there. diff --git a/.claude/architecture/monorepo.md b/.claude/architecture/monorepo.md new file mode 100644 index 0000000..4b05b22 --- /dev/null +++ b/.claude/architecture/monorepo.md @@ -0,0 +1,92 @@ +# Monorepo Mechanics + +## Workspaces + +```json +"workspaces": ["packages/*", "!packages/native-app-wrapper", "grab-help-docs"] +``` + +Note the **negation**: `packages/native-app-wrapper` is deliberately excluded. +It is a Tauri scaffold with a Rust toolchain requirement — it is not installed +by a root `npm install` and nothing at the root builds or tests it. + +`pnpm-workspace.yaml` lists the same globs *without* the negation, for anyone +using pnpm. + +## Package manager + +**npm.** `packageManager` pins `npm@11.19.1`, the committed lockfile is +`package-lock.json`, and CI runs `npm install` (tests) and `npm ci` (Pages). +`.npmrc` sets `package-manager-strict=false`. + +A `pnpm-workspace.yaml` exists but there is **no pnpm lockfile**, so pnpm is +tolerated rather than supported. Do not switch, and do not commit a second +lockfile — the lockfile is what CI installs from. + +This is the one repo in this family that is not on Bun. `bun x standard-version` +appears in the `ship` script, and `npm-publish.yml` sets up Bun, but installs +and tests are npm. + +## Two kinds of package + +| Kind | Directories | What "publishing" means | +| --- | --- | --- | +| **Internal** | `grab-api`, `grab-url-cli`, `log-json` (`@grab-url/*`, all `"private": true`) | Never published. Compiled into `grab-url`'s `dist/` by the root build. | +| **Published** | `api2client`, `archiver-web`, `loading-animations`, `quantum-sphere-loading-animation` | Published on their own **and** bundled into a `grab-url` entry | +| **Excluded** | `native-app-wrapper` | Outside the workspace; its own thing | + +Consequence: editing `packages/grab-api/src` changes the `grab-url` package. +There is no separate `@grab-url/grab-api` for a consumer to install, so its +"public API" is really `grab-url`'s. + +## Turbo + +`turbo.json` defines exactly one task: + +```json +"build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**"] } +``` + +Turbo is used for one thing here — building `grab-help-docs` +(`npm run make:docs` → `turbo run build --filter=grab-help-docs`). Everything +else runs through root npm scripts and the single Vite build. Don't reach for +turbo filters expecting the other repos' task graph; it isn't there. + +## Tests + +All tests live in the **root `test/` folder** and run under Vitest through the +same `vite.config.ts` as the build — so they see the same aliases and externals +the shipped bundle does. + +``` +test/grab.test.ts test/downloader.test.ts test/ytdlp.test.ts +test/api2client.test.ts test/archiver.test.ts test/aria2.test.ts +test/command.test.ts test/icon.test.ts test/log.test.ts +test/page-archive.test.ts +``` + +There are no per-package test folders. A new test for +`packages/whatever/src/thing.ts` goes in `test/thing.test.ts`. + +Coverage (v8) includes `packages/**/src/**` and excludes `dist`, `.d.ts`, +Svelte sources, `svg/` and `demo/` directories. + +```bash +npm run test +npm run test:coverage # what CI runs +npm run test:ui +npm run test:cli # a real download against a live Ubuntu ISO URL +``` + +`test:cli` hits the network and downloads a large file. It is a smoke test, not +part of the suite — don't wire it into CI. + +## `postinstall` downloads yt-dlp + +`scripts/install-yt-dlp.mjs --postinstall` runs on every install. If an install +appears to hang or fails behind a proxy, that is where to look: + +```bash +npm run ytdlp # force a re-download +npm run ytdlp:sidecar # fetch the sidecar binary for a packaged app +``` diff --git a/.claude/architecture/overview.md b/.claude/architecture/overview.md new file mode 100644 index 0000000..e1ce9a6 --- /dev/null +++ b/.claude/architecture/overview.md @@ -0,0 +1,61 @@ +# Architecture Overview + +A monorepo in layout, a **single published package** in practice. `packages/*` +are source modules; one Vite build at the repo root compiles them into the +`grab-url` package's `dist/`, and the root `package.json` exposes them as +subpath exports. + +## What ships + +`grab-url`, version-pinned at the repo root, with these public entries: + +| Import | Built from | What it is | +| --- | --- | --- | +| `grab-url` | `packages/grab-api/src/index.ts` | The HTTP client: caching, retries, rate limiting, request dedupe, mock support | +| `grab-url/slim` | `packages/grab-api/src/index.slim.ts` | The same client with the heavy dependencies externalized | +| `grab-url/animations` | `packages/loading-animations/src/svg/index.ts` | Tree-shakable SVG spinners | +| `grab-url/icons/quantum-sphere` | `packages/quantum-sphere-loading-animation/src/icons.ts` | The React orbital loader | +| `grab-url/log` | `packages/log-json/src/log-json.ts` | JSON logger | +| `grab-url/cli` | `packages/grab-url-cli/src/index.ts` | The CLI, also installed as the `grab-url`, `grab` and `g` bins | + +Plus three more build entries that back the archive tooling: `archiver-web`, +`bin-extract`, `bin-compress`. + +## Source packages + +| Package | npm name | Published? | +| --- | --- | --- | +| `grab-api` | `@grab-url/grab-api` | **No** — bundled into `grab-url` | +| `grab-url-cli` | `@grab-url/cli` | **No** — bundled | +| `log-json` | `@grab-url/log` | **No** — bundled | +| `api2client` | `api2client` | Yes | +| `archiver-web` | `archiver-web` | Yes | +| `loading-animations` | `loading-animations` | Yes | +| `quantum-sphere-loading-animation` | `quantum-sphere-loading-icon` | Yes | +| `native-app-wrapper` | `native-app-wrapper` | No — **and excluded from the workspace globs** | + +So there are two kinds of directory under `packages/`: internals that only exist +to be bundled, and packages that publish in their own right *and* get bundled +into a `grab-url` entry. Know which one you are in before you reason about how a +consumer receives your change. + +## Other top-level directories + +| Directory | What it is | +| --- | --- | +| `test/` | **All the tests.** Vitest, run through the root `vite.config.ts`. | +| `grab-help-docs/` | The Fumadocs documentation site, deployed to GitHub Pages. A workspace. | +| `skills/use-grab-request/` | The agent skill — the source of truth for the docs page about it | +| `examples/` | Runnable examples: basic request, cookbook, reactive React/Svelte/Vue, api2client | +| `scripts/` | `install-yt-dlp.mjs` (postinstall), `sync-skill-docs.mjs` | +| `docs/` | A small static landing page (`index.html`, `_config.yml`) — not the docs site | +| `dist/` | Build output. Generated; never hand-edited, never a source of truth. | + +## Optional peers, loaded at runtime + +`grab-url` declares `react`, `react-dom` and **`extract-webpage`** as optional +peer dependencies. `extract-webpage` is the qwksearch content extractor behind +`grab-url --page`; it is loaded through a runtime `import()` and drags in +jsdom/linkedom, so it is externalized and **must never enter the CLI bundle**. +The same applies to React for the quantum-sphere entry — see +[build.md](build.md). diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..69d4c8a --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,21 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "permissions": { + "allow": [ + "Bash(npm install)", + "Bash(npm ci)", + "Bash(npm run build:*)", + "Bash(npm run test:*)", + "Bash(npm run make:*)", + "Bash(npx turbo run:*)", + "Bash(git status:*)", + "Bash(git diff:*)", + "Bash(git log:*)" + ], + "deny": [ + "Read(./**/.env)", + "Read(./**/.env.*)", + "Read(./**/.dev.vars)" + ] + } +} diff --git a/.gitignore b/.gitignore index b8da84b..4bb4bc9 100644 --- a/.gitignore +++ b/.gitignore @@ -43,8 +43,14 @@ test/test-downloads .vercel .grab-downloads/ grab-help-docs/dist/ -.claude build.log pnpm-lock.yaml .turbo grab-help-docs/.next_old + +.claude/* +# Agent docs for Claude Code — checked in on purpose (see CLAUDE.md) +!.claude/architecture +!.claude/settings.json +.claude/settings.local.json +CLAUDE.local.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..87c0194 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,82 @@ +# CLAUDE.md — GRAB-URL + +Orientation for Claude agents working in this repository. Read this first; the +detailed notes live in [`.claude/architecture/`](.claude/architecture/) and are +linked from each section below. + +`grab-url` ("Generate Request to API from Browser") is one npm package with +several subpath entries — an HTTP client with caching, retries, rate limiting +and request dedupe; a download/archive CLI; loading animations; and a JSON +logger. The repo is a monorepo in layout only: **`packages/*` are source +modules, and one Vite build at the root bundles them all into a single +published `dist/`.** + +## Ground rules + +1. **npm, not bun or pnpm.** `packageManager` pins `npm@11.19.1` and CI runs + `npm install` / `npm ci` against `package-lock.json`. A `pnpm-workspace.yaml` + exists for pnpm compatibility, but there is no pnpm lockfile — do not switch + package managers or commit a second lockfile. +2. **There is no root `src/`.** All source lives under `packages/*/src`, and the + published entry points are built from there. See + [`architecture/build.md`](.claude/architecture/build.md). +3. **Most `packages/*` are private internals**, not separately published + packages. `@grab-url/grab-api`, `@grab-url/cli` and `@grab-url/log` are + compiled *into* `grab-url`. See + [`architecture/monorepo.md`](.claude/architecture/monorepo.md). +4. **Changing `vite.config.ts` is changing the product.** Externals, aliases, + the `"use client"` restoration and the CLI shebang banner each exist because + something broke without them. Read the comments before touching it. +5. **Tests live in the root `test/` folder**, run by Vitest through the same + `vite.config.ts`. Coverage counts `packages/**/src/**`. +6. **`postinstall` downloads yt-dlp** (`scripts/install-yt-dlp.mjs`). An install + that "hangs" is usually that. +7. **Never commit secrets** or build output. `dist/` is generated. + +## Where things live + +| You want to change… | Go to | +| --- | --- | +| The HTTP client, caching, retries, dedupe | `packages/grab-api` | +| The `grab` / `grab-url` CLI | `packages/grab-url-cli` | +| Archive extract/create | `packages/archiver-web` | +| SVG + terminal loading spinners | `packages/loading-animations` | +| The React/Svelte orbital loader | `packages/quantum-sphere-loading-animation` | +| JSON logging | `packages/log-json` | +| The Hey API client adapter | `packages/api2client` | +| Documentation | `grab-help-docs` | +| The agent skill | `skills/use-grab-request` | +| Build entries, externals, bundling | `vite.config.ts` | + +Full map: [`architecture/overview.md`](.claude/architecture/overview.md). + +## Commands + +```bash +npm install # not bun, not pnpm +npm run build # vite build — produces every dist entry +npm run make # icons → skill docs → help docs → build +npm run test # vitest +npm run test:coverage +npm run test:cli # a real end-to-end CLI download +``` + +## Before you open a PR + +- Run `npm run test:coverage` — that is what CI runs. +- Run `npm run build` if you touched anything under `packages/*/src` or + `vite.config.ts`, and check the entry you changed actually appears in `dist/`. +- If you changed the CLI's public behaviour, update + `skills/use-grab-request/SKILL.md` and run `npm run make:skill` — the docs + page is generated from the skill. +- Target `master`. Keep the PR focused; no drive-by refactors. + +## Detailed notes + +| Note | Covers | +| --- | --- | +| [overview.md](.claude/architecture/overview.md) | What ships, every entry point, every source package | +| [build.md](.claude/architecture/build.md) | The one Vite build: entries, externals, aliases, and the hacks that must not be removed | +| [monorepo.md](.claude/architecture/monorepo.md) | Workspaces, npm vs pnpm, which packages actually publish, tests | +| [documentation.md](.claude/architecture/documentation.md) | `grab-help-docs`, the generated skill page, the Pages deploy | +| [conventions.md](.claude/architecture/conventions.md) | Code style, commits, PRs, CI, publishing, security | diff --git a/grab-help-docs/CLAUDE.md b/grab-help-docs/CLAUDE.md new file mode 100644 index 0000000..680dfe0 --- /dev/null +++ b/grab-help-docs/CLAUDE.md @@ -0,0 +1,32 @@ +# CLAUDE.md — `grab-help-docs` + +The documentation site for `grab-url`: a Fumadocs site, and a workspace (it is +listed explicitly in the root `workspaces` array). + +## It deploys as a static export, not `next build` + +`.github/workflows/pages.yml` runs `npm ci` and then: + +```bash +node grab-help-docs/scripts/build-static-pages.mjs +``` + +and publishes the result to GitHub Pages. **That script is what ships** — if the +deployed site and a local dev run disagree, trust the static export. Build +through it before claiming a docs change works. + +```bash +npm run make:docs # turbo run build --filter=grab-help-docs +``` + +## One page is generated — don't edit it + +`content/docs/claude-skill.mdx` is generated from +`skills/use-grab-request/SKILL.md` by `scripts/sync-skill-docs.mjs`, and says so +in a comment at the top. Edit the skill, then `npm run make:skill`. +`node scripts/sync-skill-docs.mjs --check` fails when it is stale. + +## Not to be confused with + +The root `docs/` directory — a small static landing page for grab.js.org +(`index.html`, `_config.yml`). Documentation does not go there. diff --git a/packages/api2client/CLAUDE.md b/packages/api2client/CLAUDE.md new file mode 100644 index 0000000..79fc9b3 --- /dev/null +++ b/packages/api2client/CLAUDE.md @@ -0,0 +1,34 @@ +# CLAUDE.md — `api2client` + +**Published on its own** — and also reachable through the `grab-url` build. + +A [Hey API](https://heyapi.dev) client that sends generated OpenAPI SDK requests +through **`grab-url`** instead of fetch or axios, so every generated operation +inherits caching, retries, rate limiting and request dedupe. + +## The alias that makes this work + +`vite.config.ts` aliases the bare specifier `"grab-url"` to +`packages/grab-api/src/index.ts`, because the generated Hey API client imports +the **published package name**. Inside the monorepo that has to resolve to the +same source — otherwise a build ends up with two copies of the client and the +cache/dedupe layers stop being shared. + +If you change how this package imports `grab-url`, check that alias. + +## Rules + +- **Generated SDK code is output.** This package is the *adapter*, not the + generated client — fix behaviour here, never by editing someone's generated + SDK. +- Preserve the Hey API client contract, including its error shape. Consumers + generated against Hey API expect it. +- The point of the package is that operations get `grab-url`'s guarantees + automatically. Don't add a path that falls back to bare `fetch`. + +## Layout + +`src/index.ts` · `src/client.ts` · `src/generate.ts` · `src/cli.ts` · +`src/core/` · `src/types.ts` · `src/utils.ts` + +Tests: `test/api2client.test.ts`. Example: `examples/api2client-petstore`. diff --git a/packages/archiver-web/CLAUDE.md b/packages/archiver-web/CLAUDE.md new file mode 100644 index 0000000..bc18ac3 --- /dev/null +++ b/packages/archiver-web/CLAUDE.md @@ -0,0 +1,33 @@ +# CLAUDE.md — `archiver-web` + +**Published on its own**, and built into three `grab-url` entries: +`archiver-web`, `bin-extract`, `bin-compress`. + +A universal archive extractor and creator on **JSZip** — frontend-capable, so it +runs in a browser as well as in Node. + +## Safety is the main design constraint + +- **Zip slip.** An archive entry named `../../etc/thing` must never be written + outside the destination directory. Normalize and verify every entry path + against the resolved destination before writing — not after. +- **Zip bombs.** A small archive can expand to gigabytes. Respect and keep any + size/entry-count limits; don't remove one to make a large legitimate file + work. +- Symlink entries and absolute paths in archives are both traversal vectors. +- These rules apply doubly because two of the three entries are **executable + bins** (`bin-extract`, `bin-compress`) that users point at untrusted files. + +## Build notes + +- `jszip` is **externalized** — the caller provides it. +- `archiver-web` is externalized from the **slim** `grab-url` entry, so the slim + build doesn't carry it. Keep the import surface externalizable. +- The `bin-*` chunks get the `#!/usr/bin/env node` banner from the root build, + not from source. + +## Layout + +`src/index.ts` · `src/bin-extract.ts` · `src/bin-compress.ts` · `src/types.ts` + +Tests: `test/archiver.test.ts`. diff --git a/packages/grab-api/CLAUDE.md b/packages/grab-api/CLAUDE.md new file mode 100644 index 0000000..cdfdcb4 --- /dev/null +++ b/packages/grab-api/CLAUDE.md @@ -0,0 +1,41 @@ +# CLAUDE.md — `@grab-url/grab-api` + +**Private — never published.** This package *is* `grab-url`: the root Vite build +compiles `src/index.ts` into `dist/grab-api.*` and the root `package.json` +exposes it as the package's main entry. + +So its public API is `grab-url`'s public API. There is no separate +`@grab-url/grab-api` for anyone to install, and a breaking change here is a +breaking change to the published package. + +## Two entries, and the difference is the point + +| Source | Ships as | Difference | +| --- | --- | --- | +| `src/index.ts` | `grab-url` | Everything bundled | +| `src/index.slim.ts` | `grab-url/slim` | `archiver-web` and `linkedom` externalized | + +The slim build exists so a browser consumer doesn't pay for the heavy +dependencies. **Anything you add to `index.ts` that the slim entry also imports +must stay externalizable**, or `slim` quietly stops being slim. Check +`slimExternalPkgs` in `vite.config.ts` when adding a dependency. + +## What the client guarantees + +Caching, retries, rate limiting and request dedupe on every call — that is the +reason to use it over `fetch`. Consumers (including `debate-api-client` in the +sibling debate repo, and `api2client` here) rely on those behaviours being +automatic. Don't add a path that bypasses them. + +## Layout + +`src/core/` · `src/common/` · `src/response/` · `src/devtools/` · +`src/index.ts` · `src/index.slim.ts` + +## Rules + +- **It must run in a browser.** No Node builtins on the library path — the CLI + is a different entry, with a different externals list. +- Tests live in the **root `test/`** folder (`test/grab.test.ts`), not here. +- After changing it: `npm run build`, then confirm both `dist/grab-api.*` and + `dist/grab-api-slim.*` are produced. diff --git a/packages/grab-url-cli/CLAUDE.md b/packages/grab-url-cli/CLAUDE.md new file mode 100644 index 0000000..efe0876 --- /dev/null +++ b/packages/grab-url-cli/CLAUDE.md @@ -0,0 +1,46 @@ +# CLAUDE.md — `@grab-url/cli` + +**Private — never published.** Built into `dist/grab-url-cli.*` and installed as +the `grab-url`, `grab` and `g` bins of the `grab-url` package. + +## Things that bite + +- **The shebang comes from the build, not the source.** `vite.config.ts` adds + `#!/usr/bin/env node` via `rollupOptions.output.banner` for this chunk and the + `bin-*` ones. Remove it and the bins stop being executable. +- **`extract-webpage` must stay a runtime `import()`.** It backs `--page`, it is + an *optional* peer dependency, and it pulls in jsdom/linkedom. It is + externalized on purpose — a static import would drag a DOM implementation into + every CLI install. `inlineDynamicImports: false` exists to keep that dynamic. +- **yt-dlp is an external binary**, fetched by `scripts/install-yt-dlp.mjs` at + postinstall. Media-site URLs route through it. It may be missing, outdated, or + blocked — fail with a message that says how to fix it (`npm run ytdlp`), not a + stack trace. + +## Safety — this writes to the user's filesystem from a URL they typed + +- **Never write outside the target directory.** Path traversal from a + server-supplied filename is the classic download-tool vulnerability; sanitize + the name, don't trust `Content-Disposition`. +- Don't follow a redirect into a local/private address on a user-supplied URL. +- Don't overwrite an existing file without saying so. + +## Layout + +`src/index.ts` · `src/cli-args.ts` · `src/file-downloader.ts` · +`src/download-spinners.ts` · `src/keyboard-controls.ts` · `src/cancel-state.ts` +· `src/background.ts` · `src/display/` · `src/page/` · `src/transfer/` + +Cancellation is real state (`cancel-state.ts`, `keyboard-controls.ts`) — a +partially written file must be cleaned up or resumable, not left as a plausible +looking truncated download. + +## Testing + +Tests are in the root `test/` folder (`downloader.test.ts`, `command.test.ts`, +`ytdlp.test.ts`, `aria2.test.ts`, `page-archive.test.ts`). + +```bash +npm run test +npm run test:cli # a real end-to-end download — run by hand, not in CI +``` diff --git a/packages/loading-animations/CLAUDE.md b/packages/loading-animations/CLAUDE.md new file mode 100644 index 0000000..1aaede5 --- /dev/null +++ b/packages/loading-animations/CLAUDE.md @@ -0,0 +1,33 @@ +# CLAUDE.md — `loading-animations` + +**Published on its own**, and built into `dist/animations.*` (exposed as +`grab-url/animations`). + +Two halves, one package: **tree-shakable SVG spinners** (`src/svg/`) for the web +and **terminal spinners** (`src/cli/`) for the CLI. + +## Tree-shakability is the feature + +`src/svg/index.ts` is a **generated barrel**. It is produced by: + +```bash +npm run make:icons +# npx export-svg-typescript -i packages/loading-animations/src/svg -o packages/loading-animations/src/svg/index.ts +``` + +So: **add or edit the `.svg` files, then regenerate the barrel.** Hand-editing +`index.ts` is overwritten on the next `npm run make`. + +One export per animation, no module-scope side effects — a consumer importing +one spinner must not pull in all of them. + +## Build notes + +The `svg/` directory is excluded from coverage (it is generated assets, not +logic). The CLI spinners are the tested half. + +## Layout + +`src/svg/` (the SVGs + the generated barrel) · `src/cli/` (terminal spinners) + +Tests: `test/icon.test.ts`. diff --git a/packages/log-json/CLAUDE.md b/packages/log-json/CLAUDE.md new file mode 100644 index 0000000..b7f576e --- /dev/null +++ b/packages/log-json/CLAUDE.md @@ -0,0 +1,24 @@ +# CLAUDE.md — `@grab-url/log` + +**Private — never published.** Built into `dist/log.*` and exposed as +`grab-url/log`. + +A JSON logger with structured output and terminal colors. + +## Rules + +- **It is used by the CLI and the library both.** Colors must degrade: no ANSI + when the output is not a TTY, when `NO_COLOR` is set, or when the consumer is + capturing JSON. A logger that emits escape codes into a piped JSON stream + breaks every caller that parses it. +- **Never log credentials, tokens, or full URLs with query strings** — the + client this ships with sends authenticated requests, and a logged URL is a + logged API key. +- Structured output is the contract (`structure.ts`). Adding or renaming a field + changes what consumers parse. + +## Layout + +`src/log-json.ts` (entry) · `src/structure.ts` · `src/colors.ts` + +Tests: `test/log.test.ts` in the root test folder. diff --git a/packages/native-app-wrapper/CLAUDE.md b/packages/native-app-wrapper/CLAUDE.md new file mode 100644 index 0000000..d42a165 --- /dev/null +++ b/packages/native-app-wrapper/CLAUDE.md @@ -0,0 +1,29 @@ +# CLAUDE.md — `native-app-wrapper` + +**Excluded from the workspace on purpose.** The root `workspaces` array carries +an explicit negation: + +```json +"workspaces": ["packages/*", "!packages/native-app-wrapper", "grab-help-docs"] +``` + +So a root `npm install` does not install it, the root Vite build does not build +it, and the root test run never reaches it. Nothing at the root will tell you +you broke it. + +A Tauri scaffold that packages a website, or a bundled CLI, as a native desktop +app. + +## Things that bite + +- **Rust toolchain required.** A machine that builds the rest of this repo + cannot necessarily build this. +- **yt-dlp ships as a Tauri sidecar** here. Sidecar binaries are referenced with + platform-suffixed names — get one wrong and it fails on exactly one OS, at + runtime, after packaging. `npm run ytdlp:sidecar` fetches it. +- Native behaviour (window, tray, updater, deep links) lives on the Rust side, + not in the web layer. +- A green build on one platform says nothing about the others. + +Install and build from inside this directory, and say in the PR how you verified +it. diff --git a/packages/quantum-sphere-loading-animation/CLAUDE.md b/packages/quantum-sphere-loading-animation/CLAUDE.md new file mode 100644 index 0000000..ae2d5c8 --- /dev/null +++ b/packages/quantum-sphere-loading-animation/CLAUDE.md @@ -0,0 +1,34 @@ +# CLAUDE.md — `quantum-sphere-loading-icon` + +**npm name:** `quantum-sphere-loading-icon` (directory: +`quantum-sphere-loading-animation`). Published on its own, and built into +`dist/quantum-sphere.*` (exposed as `grab-url/icons/quantum-sphere`). + +A parabolic spherical orbital loading component for **React and Svelte**. + +## The two build rules that must not be broken + +Both are enforced in the root `vite.config.ts`, and both exist because of a real +failure: + +1. **React is externalized.** `react`, `react-dom`, `react/jsx-runtime` and + `react/jsx-dev-runtime` never enter the bundle. A second React copy makes + every hook in `QuantumOrbital` throw *"Invalid hook call"*. No other entry + imports React, so this costs nothing elsewhere. +2. **`"use client"` is re-applied in `generateBundle`.** Rollup drops the + source's module-level directive, and a `banner` does not survive terser, + which re-parses the chunk and discards a directive it reads as dead code. + Writing it after minification is the only point where it sticks. Without it, + a React Server Component importing the sphere fails on the first hook. + +If you are debugging "Invalid hook call" or an RSC boundary error from this +component, those two are where to look — not in the component. + +## Layout + +`src/icons.ts` (the built entry) · `src/index.ts` · `src/react/` · `src/svelte/` +· `src/shared/` · `src/types/` + +The Svelte sources and `demo/` are excluded from the type build and from +coverage. Keep genuinely shared logic in `src/shared/` so the two framework +wrappers stay thin. From c847ef0271956a09788f98363351f2f2452017e5 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 11 Sep 2026 12:30:41 +0000 Subject: [PATCH 2/2] =?UTF-8?q?=F0=9F=93=9D=20docs(claude):=20record=20why?= =?UTF-8?q?=20the=20Vercel=20deploy=20fails,=20and=20what=20docs/=20actual?= =?UTF-8?q?ly=20is?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first pass called the root `docs/` folder "a small static landing page for grab.js.org". It is a GitHub Pages Jekyll stub — a `_config.yml` and an `index.html` redirect, kept because `/docs` is the only folder name Pages accepts besides the repository root — and its own README says so. The omission that mattered: that folder is what the Vercel project's Root Directory still points at, and it is why every deploy fails. Turbo resolves no package from `docs/` (it matches neither workspace glob), so no `.next` is produced and Vercel reports the missing `docs/.next/routes-manifest.json`. The fix is a dashboard setting — Root Directory `grab-help-docs`, with the Install/Build Command overrides cleared so `grab-help-docs/vercel.json` supplies them — so no commit here can fix it, and an agent should not go hunting for it in turbo.json or vercel.json. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01EoGK1G8bnF6riGQekWLtYr --- .claude/architecture/documentation.md | 26 +++++++++++++++++++++++--- .claude/architecture/overview.md | 2 +- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/.claude/architecture/documentation.md b/.claude/architecture/documentation.md index ec0aa3c..f04054f 100644 --- a/.claude/architecture/documentation.md +++ b/.claude/architecture/documentation.md @@ -37,7 +37,27 @@ static export path is the one that ships. npm run make:docs # turbo run build --filter=grab-help-docs ``` -## `docs/` is not the docs site +## `docs/` is not the docs site — and it is what breaks Vercel -The root `docs/` directory is a small static landing page (`index.html`, -`_config.yml`, `README.md`) for grab.js.org. Do not put documentation there. +The root `docs/` directory holds no documentation. It is a **GitHub Pages +Jekyll stub**: a `_config.yml` and an `index.html` that redirects to +grab.js.org, kept because `/docs` is the only folder name Pages accepts besides +the repository root. Its own `README.md` says so. Do not put documentation +there. + +It matters for one operational reason. **The Vercel project's Root Directory +must be `grab-help-docs`.** Left pointing at `docs/`, every deploy fails the +same way: turbo resolves no package from `docs/` (it matches neither +`packages/*` nor `grab-help-docs` in the workspace globs), so no `.next` is +produced and Vercel reports + +``` +The file "/vercel/path0/docs/.next/routes-manifest.json" couldn't be found. +``` + +With the Root Directory set correctly, `grab-help-docs/vercel.json` supplies +the framework, install command and build command — so the project's own +Install/Build Command overrides should be left unset. + +**This is a dashboard setting, not a diff.** No commit in this repository can +fix it, so do not go looking for the bug in `turbo.json` or `vercel.json`. diff --git a/.claude/architecture/overview.md b/.claude/architecture/overview.md index e1ce9a6..5929104 100644 --- a/.claude/architecture/overview.md +++ b/.claude/architecture/overview.md @@ -48,7 +48,7 @@ consumer receives your change. | `skills/use-grab-request/` | The agent skill — the source of truth for the docs page about it | | `examples/` | Runnable examples: basic request, cookbook, reactive React/Svelte/Vue, api2client | | `scripts/` | `install-yt-dlp.mjs` (postinstall), `sync-skill-docs.mjs` | -| `docs/` | A small static landing page (`index.html`, `_config.yml`) — not the docs site | +| `docs/` | A GitHub Pages Jekyll stub — `_config.yml` plus an `index.html` redirect. No documentation; see [documentation.md](documentation.md), which also explains why it is what breaks the Vercel deploy. | | `dist/` | Build output. Generated; never hand-edited, never a source of truth. | ## Optional peers, loaded at runtime