[Extract] Add extract, map and crawl commands for the Extract API by strzibny · Pull Request #21 · serpapi/serpapi-cli

strzibny · 2026-10-22T13:21:44Z

This PR adds Extract API support. I wanted to add specific extract subcommand but though it's actually useful for CLI to also add map and crawl aliases for developer experience.

serpapi extract https://example.com output=md
serpapi map https://books.toscrape.com --jq '.urls[].url'   # = extract URL mode=map
serpapi crawl https://books.toscrape.com max_pages=5        # = extract URL mode=crawl

Done with Opus 5.5 High and Max. Some futher notes:

The Extract API is engine=extract on /search.json, so these commands reuse search's request and output code: --fields, --jq, output=md, --debug, --timeout and archive work as in search.
Unlike search, they need an API key up front; the API returns 401 without one.
Bad input fails before any request is sent, so it costs nothing: no URL or several URLs, a URL that isn't absolute http(s), an engine= other than extract, a mode= that contradicts map/crawl, a parameter missing its =, and search-only flags such as --all-pages. An argument counts as the URL when it has :, /, . or ? before any =.
crawl waits up to 300s instead of 60s unless --timeout or SERPAPI_TIMEOUT is set, since SerpApi still finishes a crawl the CLI gave up on and charges one search per page.
A timed-out Extract request no longer gets search's "slow engines such as google_ai_mode" advice, and a timed-out crawl suggests at least --timeout 300. This changes shared code: networkError in pkg/api/client.go now picks the advice from the request's engine and mode.
crawl costs one search per extracted page (default max_pages=10). The help and README say so.
One change to search: an unknown flag with dashes now suggests the parameter with underscores, so --no-cache suggests no_cache=VALUE instead of no-cache=VALUE.

Note Extract API can already be used by passing engine=extract, but this makes things nicer.

`serpapi extract URL [key=value...]` sends an engine=extract search, so --fields, --jq, output=md, --debug and --timeout work as in search. `serpapi map URL` and `serpapi crawl URL` are shortcuts for `serpapi extract URL mode=map` and `mode=crawl`: same request, parameters and flags. The URL may be given bare or as url=, and must be one absolute http or https URL; anything else is a usage error before any request, as are engine= other than extract, a mode= that contradicts map or crawl, a word without "=", which search would take as q=, and search's --all-pages and --image. Unlike search, the commands need an API key up front, since the Extract API answers nothing without one. A timed-out extraction no longer blames slow engines such as google_ai_mode: extract and map say only how to retry, and crawl suggests --timeout 300 and that each page counts. The README documents the three commands, says map and crawl are shortcuts for mode=map and mode=crawl, and lists crawl among the requests that can outlast the default --timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

trusche

Thanks @strzibny I don't know Go, so this is a fully automated AI review with my usual coding agents. Let me know if that's of any use or you'd rather not get reviews like this.

Everything below and inline is verbatim from Opus 5.5. High

Claude Opus 5.5 AI-assisted review: read the code, ran the tests and checked the CLI by hand.

Approve with comments. Nothing blocks the merge. Please add the release:minor label before you merge: this PR adds three new commands, and without a label the merge releases only a patch version (see AGENTS.md, "Releases").

QA: go test ./pkg/..., go vet and the offline e2e tests pass. I ran extract, map and crawl against the live API (3 searches), and 14 usage-error and timeout cases offline. The help text matches the Extract API docs. Details are in the inline comments; all of them are non-blocking.

trusche · 2026-10-23T10:46:22Z

+	if raw {
+		body, err := withSpinner(label, func() ([]byte, error) { return client.SearchRaw(ctx, p) })
+		if err != nil {
+			return err
+		}
+		return outputError(output.PrintRaw(cmd.OutOrStdout(), body))
+	}
+	result, err := withSpinner(label, func() (json.RawMessage, error) { return client.Search(ctx, p) })
+	if err != nil {
+		return err
+	}
+	if restricted && isEmptyObject(result) {
+		warnNothingMatched(cmd)
+	}
+	return handleOutput(cmd, result, searchHint(p))


These lines are a copy of sendSearch (search.go:180-201). Only the spinner text is different. The extract commands do not have --all-pages, so allPagesFlag is always false here. Add a label argument to sendSearch and call it from here. That removes about 15 lines.

The output= and --jq check at lines 117-125 is now in 3 places (also search.go:130-138 and archive.go:47-53). Maybe move it into one helper.

Done in 3a99407: sendSearch now takes the spinner text and extract calls it. The output=/--jq check is one helper, checkOutput, shared by search, extract and archive.

trusche · 2026-10-23T10:46:22Z

+		case strings.EqualFold(query.Get("engine"), "extract") && strings.EqualFold(query.Get("mode"), "crawl"):
+			message = timedOut + ". A crawl extracts every page it visits, so it takes longer than one page: " +
+				retry(max(suggestedTimeout(c.http.Timeout), 300)) +
+				". The crawl may still finish on SerpApi's side, count one search per page and be available in your search archive"
+		case strings.EqualFold(query.Get("engine"), "extract"):
+			message = timedOut + ": " + retry(suggestedTimeout(c.http.Timeout)) + ". The extraction" + onSerpApi
+		default:


The HTTP client now reads engine=extract and mode=crawl from the request URL, and 300 is a fixed number here. The /image and google_ai_mode cases already do something similar, so this is OK. But this is the second engine-specific branch in the client. Maybe give 300 a name.

A question: when a crawl times out on the CLI side, it can still finish on the server and charge one search per page. This advice arrives only after that cost. Should crawl default to a 300s timeout when the user does not set --timeout or SERPAPI_TIMEOUT? Then this branch may not be needed.

Agreed. In 3a99407 a crawl waits up to 300s unless --timeout or SERPAPI_TIMEOUT is set. The limit is api.DefaultCrawlTimeout, and the advice uses it too.

I kept the crawl branch, though: someone with SERPAPI_TIMEOUT=60 still needs to hear that a crawl wants --timeout 300, and after the 300s default it suggests 600.

trusche · 2026-10-23T10:46:22Z

+  mode=crawl         every page of the website it visits; same as 'serpapi crawl'
+  include_tags=, exclude_tags=   CSS selectors to keep or drop
+  include_markdown=true          add the main content as Markdown to JSON
+  output=md                      only the main content, as Markdown


"only the main content" is not quite correct: the Markdown starts with a YAML block with the search metadata (live: serpapi extract https://example.com/ output=md). Maybe: "the main content as Markdown, after the search metadata".

Right, fixed in 3a99407 with your wording.

trusche · 2026-10-23T10:46:22Z

+			continue
+		}
+		// search takes a bare word as q=; here it is a parameter missing its "=".
+		if !strings.Contains(a, "=") {


Good. But the root help at root.go:60-61 still says "words without "=" are joined into q" for all commands. That is not true for extract, map and crawl now. Maybe add "in search" there.

Fixed in 3a99407: the root help now says "search joins words without "=" into q".

trusche · 2026-10-23T10:46:22Z

+// as the URL, so it is reported as an invalid one.
+func isURLArg(arg string) bool {
+	key, _, _ := strings.Cut(arg, "=")
+	return arg == "" || strings.ContainsAny(key, ":/.?")


An empty argument counts as the URL. When it comes with a real URL, the error has an empty first item:

$ serpapi extract "" https://example.com extract takes one URL, got 2: , https://example.com; ...

Maybe report the empty URL first, before the CLI counts the URLs.

Fixed in 3a99407: an empty URL is reported first, as Invalid URL "", before the URLs are counted.

trusche · 2026-10-23T10:46:22Z

+	}
+	msg := fmt.Sprintf("Invalid URL %q: the Extract API takes an absolute http or https URL", raw)
+	if !strings.Contains(raw, "://") {
+		if u, err := url.Parse("https://" + raw); err == nil && strings.Contains(u.Hostname(), ".") {


Small: serpapi extract mailto:me@x.com suggests e.g. https://mailto:me@x.com. url.Parse reads mailto:me as a user and password, so the hostname is x.com. Maybe suggest only when u.User == nil.

Fixed in 3a99407: it suggests https:// only when the parsed URL has no user part, so mailto:me@x.com gets no suggestion.

trusche · 2026-10-23T10:46:22Z

+		if isExtractCommand(cmd) {
+			// search's --all-pages and --image have no Extract API parameter;
+			// its --max-pages does, as crawl's max_pages.
+			if name != "max-pages" && searchCmd.Flags().Lookup(name) != nil {
+				return msg + "; --" + name + " works only with search; " + params
+			}
+			// Extract API parameters use underscores, as in max_pages.
+			name = strings.ReplaceAll(name, "-", "_")
+		}


The change from - to _ applies only to the extract commands. search --no-cache still suggests no-cache=VALUE. That bug is not from this PR, but the same fix applies there. Maybe apply it to all commands, or leave it for a separate PR.

Also, serpapi extract URL --max-pages 3 suggests max_pages=VALUE, but only crawl uses that parameter.

Both fixed in 3a99407. It's the same line, so I fixed search here too: search --no-cache now suggests no_cache=VALUE. --max-pages suggests max_pages=VALUE only on crawl; extract and map say it works only with search.

trusche · 2026-10-23T10:46:22Z

+serpapi extract https://example.com --jq '.inline_links[].link'
+```
+
+It counts one search. The Extract API runs no JavaScript and follows each site's `robots.txt`. Search pages that have their own engine, such as Google or Bing, are redirected to it and return that engine's JSON. Unlike `search`, `extract`, `map` and `crawl` need an API key.


"It counts one search" comes right after the table row for mode=crawl, which costs one search per page. The help at extract.go:37-38 has the correct sentence: "scrape and map count one search each; crawl counts one per page it extracts".

Fixed in 3a99407: scraping a page and mapping a website count one search each, a crawl one per page it extracts.

trusche · 2026-10-23T10:46:22Z

+
+`serpapi crawl URL` is a shortcut for `serpapi extract URL mode=crawl`: it sends the same request and takes the same parameters and flags.
+
+Starts at the URL, follows links on the same website and extracts every page it visits, one entry each in `pages`. `max_pages` caps the pages, the first included (default 10, at most 100), and `max_depth` how many links deep it goes (default 2, at most 3; `0` is only the first page).


The help (extract.go:81) and the API docs list include_subdomains=true for crawl, but this section does not mention it.

Added in 3a99407.

trusche · 2026-10-23T10:46:22Z

+func TestExtractCommandsTimeOut(t *testing.T) {
+	const extraction = "Request to /search.json timed out after 50ms: retry with a higher --timeout, such as --timeout 120"
+	const crawl = "Request to /search.json timed out after 50ms. A crawl extracts every page it visits, so it takes longer than one page: " +
+		"retry with a higher --timeout, such as --timeout 300"
+	for _, tc := range []struct {
+		name, env, want string
+		args            []string
+	}{
+		{"extract --timeout", "", extraction, []string{"--timeout", "0.05", "extract"}},
+		{"map --timeout", "", extraction, []string{"--timeout", "50ms", "map"}},
+		{"extract mode=crawl", "", crawl, []string{"--timeout", "50ms", "extract", "mode=crawl"}},
+		{"crawl --timeout", "", crawl, []string{"--timeout", "0.05", "crawl"}},
+		{"crawl SERPAPI_TIMEOUT", "0.05", crawl, []string{"crawl"}},
+	} {
+		t.Run(tc.name, func(t *testing.T) {
+			fakeAPI(t, hang)
+			t.Setenv("SERPAPI_TIMEOUT", tc.env)
+
+			start := time.Now()
+			res := execute(t, "", append(append([]string{"--api-key", extractKey}, tc.args...), "https://example.com/")...)
+
+			wantError(t, res, 1, "network_error", tc.want)
+			if strings.Contains(res.stdout+res.stderr, "Slow engines") {
+				t.Errorf("advice blames slow engines:\n%s%s", res.stdout, res.stderr)
+			}
+			if elapsed := time.Since(start); elapsed > 2*time.Second {
+				t.Errorf("took %v, want the 50ms timeout to stop it", elapsed)
+			}
+		})
+	}
+}


Some tests check the same thing more than once:

The timeout messages are checked here (5 cases) and also in client_test.go TestExtractTimeoutAdvice.

The debug trace runs 3 times (lines 209-245).

"Needs an API key" is in lines 118-124 and 197-204, and in 2 rows of tests/e2e_test.go.

One case for each is enough. Not blocking.

Trimmed in 3a99407: one unit test and one e2e row for the API key, one debug trace test, and the timeout advice wording only in pkg/api, plus one crawl timeout end to end. A new table checks which timeout each command runs with, including the 300s crawl default.

A crawl extracts up to max_pages pages in one request, and SerpApi still finishes one the CLI gave up on and charges it one search per page. Without --timeout or SERPAPI_TIMEOUT, a crawl now waits up to 300s instead of 60s. api.DefaultCrawlTimeout names the limit, and the timeout advice uses it too: a crawl that times out is told to retry with --timeout 300, or twice the limit when that is more. The help and README give the new default. extract, map and crawl now send through search's sendSearch, which takes the spinner text, and search, extract and archive share checkOutput for the output= and --jq check. An empty URL, as from an unset variable, is reported before the URLs are counted, and mailto:me@x.com is no longer told to try https://mailto:me@x.com. search and locations now also suggest a dashed flag's parameter with underscores, so search --no-cache suggests no_cache=VALUE, not no-cache=VALUE. --max-pages suggests max_pages only on crawl; extract and map say it works only with search. The help says output=md puts the search metadata before the Markdown, and that only search joins words without "=" into q. The README says a crawl counts one search per page and mentions include_subdomains=true. Each behavior is tested once, and one table checks the timeout each command runs with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

strzibny requested review from ilyazub, jvmvik and trusche October 22, 2026 13:24

trusche approved these changes Oct 23, 2026

View reviewed changes

strzibny added the release:minor Merging releases the next minor version label Oct 26, 2026

strzibny merged commit 70681a8 into main Oct 26, 2026
8 checks passed


		`serpapi crawl URL` is a shortcut for `serpapi extract URL mode=crawl`: it sends the same request and takes the same parameters and flags.

		Starts at the URL, follows links on the same website and extracts every page it visits, one entry each in `pages`. `max_pages` caps the pages, the first included (default 10, at most 100), and `max_depth` how many links deep it goes (default 2, at most 3; `0` is only the first page).

Conversation

strzibny commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

trusche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

strzibny commented Oct 22, 2026 •

edited

Loading