Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 92 additions & 0 deletions .claude/architecture/monorepo.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Monorepo Mechanics

## Workspaces

```json
"workspaces": ["packages/*", "!packages/native-app-wrapper", "grab-help-docs"]
```

Note the **negation**: `packages/native-app-wrapper` is deliberately excluded.
It is a Tauri scaffold with a Rust toolchain requirement β€” it is not installed
by a root `npm install` and nothing at the root builds or tests it.

`pnpm-workspace.yaml` lists the same globs *without* the negation, for anyone
using pnpm.

## Package manager

**npm.** `packageManager` pins `npm@11.19.1`, the committed lockfile is
`package-lock.json`, and CI runs `npm install` (tests) and `npm ci` (Pages).
`.npmrc` sets `package-manager-strict=false`.

A `pnpm-workspace.yaml` exists but there is **no pnpm lockfile**, so pnpm is
tolerated rather than supported. Do not switch, and do not commit a second
lockfile β€” the lockfile is what CI installs from.

This is the one repo in this family that is not on Bun. `bun x standard-version`
appears in the `ship` script, and `npm-publish.yml` sets up Bun, but installs
and tests are npm.

## Two kinds of package

| Kind | Directories | What "publishing" means |
| --- | --- | --- |
| **Internal** | `grab-api`, `grab-url-cli`, `log-json` (`@grab-url/*`, all `"private": true`) | Never published. Compiled into `grab-url`'s `dist/` by the root build. |
| **Published** | `api2client`, `archiver-web`, `loading-animations`, `quantum-sphere-loading-animation` | Published on their own **and** bundled into a `grab-url` entry |
| **Excluded** | `native-app-wrapper` | Outside the workspace; its own thing |

Consequence: editing `packages/grab-api/src` changes the `grab-url` package.
There is no separate `@grab-url/grab-api` for a consumer to install, so its
"public API" is really `grab-url`'s.

## Turbo

`turbo.json` defines exactly one task:

```json
"build": { "dependsOn": ["^build"], "outputs": ["dist/**", ".next/**"] }
```

Turbo is used for one thing here β€” building `grab-help-docs`
(`npm run make:docs` β†’ `turbo run build --filter=grab-help-docs`). Everything
else runs through root npm scripts and the single Vite build. Don't reach for
turbo filters expecting the other repos' task graph; it isn't there.

## Tests

All tests live in the **root `test/` folder** and run under Vitest through the
same `vite.config.ts` as the build β€” so they see the same aliases and externals
the shipped bundle does.

```
test/grab.test.ts test/downloader.test.ts test/ytdlp.test.ts
test/api2client.test.ts test/archiver.test.ts test/aria2.test.ts
test/command.test.ts test/icon.test.ts test/log.test.ts
test/page-archive.test.ts
```

There are no per-package test folders. A new test for
`packages/whatever/src/thing.ts` goes in `test/thing.test.ts`.

Coverage (v8) includes `packages/**/src/**` and excludes `dist`, `.d.ts`,
Svelte sources, `svg/` and `demo/` directories.

```bash
npm run test
npm run test:coverage # what CI runs
npm run test:ui
npm run test:cli # a real download against a live Ubuntu ISO URL
```

`test:cli` hits the network and downloads a large file. It is a smoke test, not
part of the suite β€” don't wire it into CI.

## `postinstall` downloads yt-dlp

`scripts/install-yt-dlp.mjs --postinstall` runs on every install. If an install
appears to hang or fails behind a proxy, that is where to look:

```bash
npm run ytdlp # force a re-download
npm run ytdlp:sidecar # fetch the sidecar binary for a packaged app
```
21 changes: 21 additions & 0 deletions .claude/settings.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"$schema": "https://json.schemastore.org/claude-code-settings.json",
"permissions": {
"allow": [
"Bash(npm install)",
"Bash(npm ci)",
"Bash(npm run build:*)",
"Bash(npm run test:*)",
"Bash(npm run make:*)",
"Bash(npx turbo run:*)",
"Bash(git status:*)",
"Bash(git diff:*)",
"Bash(git log:*)"
],
"deny": [
"Read(./**/.env)",
"Read(./**/.env.*)",
"Read(./**/.dev.vars)"
]
}
}
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -50,3 +50,10 @@ build.log
pnpm-lock.yaml
.turbo
grab-help-docs/.next_old

.claude/*
# Agent docs for Claude Code β€” checked in on purpose (see CLAUDE.md)
!.claude/architecture
!.claude/settings.json
.claude/settings.local.json
CLAUDE.local.md
32 changes: 32 additions & 0 deletions grab-help-docs/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# CLAUDE.md β€” `grab-help-docs`

The documentation site for `grab-url`: a Fumadocs site, and a workspace (it is
listed explicitly in the root `workspaces` array).

## It deploys as a static export, not `next build`

`.github/workflows/pages.yml` runs `npm ci` and then:

```bash
node grab-help-docs/scripts/build-static-pages.mjs
```

and publishes the result to GitHub Pages. **That script is what ships** β€” if the
deployed site and a local dev run disagree, trust the static export. Build
through it before claiming a docs change works.

```bash
npm run make:docs # turbo run build --filter=grab-help-docs
```

## One page is generated β€” don't edit it

`content/docs/claude-skill.mdx` is generated from
`skills/use-grab-request/SKILL.md` by `scripts/sync-skill-docs.mjs`, and says so
in a comment at the top. Edit the skill, then `npm run make:skill`.
`node scripts/sync-skill-docs.mjs --check` fails when it is stale.

## Not to be confused with

The root `docs/` directory β€” a small static landing page for grab.js.org
(`index.html`, `_config.yml`). Documentation does not go there.
34 changes: 34 additions & 0 deletions packages/api2client/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# CLAUDE.md β€” `api2client`

**Published on its own** β€” and also reachable through the `grab-url` build.

A [Hey API](https://heyapi.dev) client that sends generated OpenAPI SDK requests
through **`grab-url`** instead of fetch or axios, so every generated operation
inherits caching, retries, rate limiting and request dedupe.

## The alias that makes this work

`vite.config.ts` aliases the bare specifier `"grab-url"` to
`packages/grab-api/src/index.ts`, because the generated Hey API client imports
the **published package name**. Inside the monorepo that has to resolve to the
same source β€” otherwise a build ends up with two copies of the client and the
cache/dedupe layers stop being shared.

If you change how this package imports `grab-url`, check that alias.

## Rules

- **Generated SDK code is output.** This package is the *adapter*, not the
generated client β€” fix behaviour here, never by editing someone's generated
SDK.
- Preserve the Hey API client contract, including its error shape. Consumers
generated against Hey API expect it.
- The point of the package is that operations get `grab-url`'s guarantees
automatically. Don't add a path that falls back to bare `fetch`.

## Layout

`src/index.ts` Β· `src/client.ts` Β· `src/generate.ts` Β· `src/cli.ts` Β·
`src/core/` Β· `src/types.ts` Β· `src/utils.ts`

Tests: `test/api2client.test.ts`. Example: `examples/api2client-petstore`.
33 changes: 33 additions & 0 deletions packages/archiver-web/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# CLAUDE.md β€” `archiver-web`

**Published on its own**, and built into three `grab-url` entries:
`archiver-web`, `bin-extract`, `bin-compress`.

A universal archive extractor and creator on **JSZip** β€” frontend-capable, so it
runs in a browser as well as in Node.

## Safety is the main design constraint

- **Zip slip.** An archive entry named `../../etc/thing` must never be written
outside the destination directory. Normalize and verify every entry path
against the resolved destination before writing β€” not after.
- **Zip bombs.** A small archive can expand to gigabytes. Respect and keep any
size/entry-count limits; don't remove one to make a large legitimate file
work.
- Symlink entries and absolute paths in archives are both traversal vectors.
- These rules apply doubly because two of the three entries are **executable
bins** (`bin-extract`, `bin-compress`) that users point at untrusted files.

## Build notes

- `jszip` is **externalized** β€” the caller provides it.
- `archiver-web` is externalized from the **slim** `grab-url` entry, so the slim
build doesn't carry it. Keep the import surface externalizable.
- The `bin-*` chunks get the `#!/usr/bin/env node` banner from the root build,
not from source.

## Layout

`src/index.ts` Β· `src/bin-extract.ts` Β· `src/bin-compress.ts` Β· `src/types.ts`

Tests: `test/archiver.test.ts`.
41 changes: 41 additions & 0 deletions packages/grab-api/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# CLAUDE.md β€” `@grab-url/grab-api`

**Private β€” never published.** This package *is* `grab-url`: the root Vite build
compiles `src/index.ts` into `dist/grab-api.*` and the root `package.json`
exposes it as the package's main entry.

So its public API is `grab-url`'s public API. There is no separate
`@grab-url/grab-api` for anyone to install, and a breaking change here is a
breaking change to the published package.

## Two entries, and the difference is the point

| Source | Ships as | Difference |
| --- | --- | --- |
| `src/index.ts` | `grab-url` | Everything bundled |
| `src/index.slim.ts` | `grab-url/slim` | `archiver-web` and `linkedom` externalized |

The slim build exists so a browser consumer doesn't pay for the heavy
dependencies. **Anything you add to `index.ts` that the slim entry also imports
must stay externalizable**, or `slim` quietly stops being slim. Check
`slimExternalPkgs` in `vite.config.ts` when adding a dependency.

## What the client guarantees

Caching, retries, rate limiting and request dedupe on every call β€” that is the
reason to use it over `fetch`. Consumers (including `debate-api-client` in the
sibling debate repo, and `api2client` here) rely on those behaviours being
automatic. Don't add a path that bypasses them.

## Layout

`src/core/` Β· `src/common/` Β· `src/response/` Β· `src/devtools/` Β·
`src/index.ts` Β· `src/index.slim.ts`

## Rules

- **It must run in a browser.** No Node builtins on the library path β€” the CLI
is a different entry, with a different externals list.
- Tests live in the **root `test/`** folder (`test/grab.test.ts`), not here.
- After changing it: `npm run build`, then confirm both `dist/grab-api.*` and
`dist/grab-api-slim.*` are produced.
46 changes: 46 additions & 0 deletions packages/grab-url-cli/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# CLAUDE.md β€” `@grab-url/cli`

**Private β€” never published.** Built into `dist/grab-url-cli.*` and installed as
the `grab-url`, `grab` and `g` bins of the `grab-url` package.

## Things that bite

- **The shebang comes from the build, not the source.** `vite.config.ts` adds
`#!/usr/bin/env node` via `rollupOptions.output.banner` for this chunk and the
`bin-*` ones. Remove it and the bins stop being executable.
- **`extract-webpage` must stay a runtime `import()`.** It backs `--page`, it is
an *optional* peer dependency, and it pulls in jsdom/linkedom. It is
externalized on purpose β€” a static import would drag a DOM implementation into
every CLI install. `inlineDynamicImports: false` exists to keep that dynamic.
- **yt-dlp is an external binary**, fetched by `scripts/install-yt-dlp.mjs` at
postinstall. Media-site URLs route through it. It may be missing, outdated, or
blocked β€” fail with a message that says how to fix it (`npm run ytdlp`), not a
stack trace.

## Safety β€” this writes to the user's filesystem from a URL they typed

- **Never write outside the target directory.** Path traversal from a
server-supplied filename is the classic download-tool vulnerability; sanitize
the name, don't trust `Content-Disposition`.
- Don't follow a redirect into a local/private address on a user-supplied URL.
- Don't overwrite an existing file without saying so.

## Layout

`src/index.ts` Β· `src/cli-args.ts` Β· `src/file-downloader.ts` Β·
`src/download-spinners.ts` Β· `src/keyboard-controls.ts` Β· `src/cancel-state.ts`
Β· `src/background.ts` Β· `src/display/` Β· `src/page/` Β· `src/transfer/`

Cancellation is real state (`cancel-state.ts`, `keyboard-controls.ts`) β€” a
partially written file must be cleaned up or resumable, not left as a plausible
looking truncated download.

## Testing

Tests are in the root `test/` folder (`downloader.test.ts`, `command.test.ts`,
`ytdlp.test.ts`, `aria2.test.ts`, `page-archive.test.ts`).

```bash
npm run test
npm run test:cli # a real end-to-end download β€” run by hand, not in CI
```
33 changes: 33 additions & 0 deletions packages/loading-animations/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# CLAUDE.md β€” `loading-animations`

**Published on its own**, and built into `dist/animations.*` (exposed as
`grab-url/animations`).

Two halves, one package: **tree-shakable SVG spinners** (`src/svg/`) for the web
and **terminal spinners** (`src/cli/`) for the CLI.

## Tree-shakability is the feature

`src/svg/index.ts` is a **generated barrel**. It is produced by:

```bash
npm run make:icons
# npx export-svg-typescript -i packages/loading-animations/src/svg -o packages/loading-animations/src/svg/index.ts
```

So: **add or edit the `.svg` files, then regenerate the barrel.** Hand-editing
`index.ts` is overwritten on the next `npm run make`.

One export per animation, no module-scope side effects β€” a consumer importing
one spinner must not pull in all of them.

## Build notes

The `svg/` directory is excluded from coverage (it is generated assets, not
logic). The CLI spinners are the tested half.

## Layout

`src/svg/` (the SVGs + the generated barrel) Β· `src/cli/` (terminal spinners)

Tests: `test/icon.test.ts`.
24 changes: 24 additions & 0 deletions packages/log-json/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# CLAUDE.md β€” `@grab-url/log`

**Private β€” never published.** Built into `dist/log.*` and exposed as
`grab-url/log`.

A JSON logger with structured output and terminal colors.

## Rules

- **It is used by the CLI and the library both.** Colors must degrade: no ANSI
when the output is not a TTY, when `NO_COLOR` is set, or when the consumer is
capturing JSON. A logger that emits escape codes into a piped JSON stream
breaks every caller that parses it.
- **Never log credentials, tokens, or full URLs with query strings** β€” the
client this ships with sends authenticated requests, and a logged URL is a
logged API key.
- Structured output is the contract (`structure.ts`). Adding or renaming a field
changes what consumers parse.

## Layout

`src/log-json.ts` (entry) Β· `src/structure.ts` Β· `src/colors.ts`

Tests: `test/log.test.ts` in the root test folder.
Loading
Loading