Skip to content

Commit 7f2dec3

Browse files
adarshdigievocodex
andauthored
Feat/extract api (#45)
* chore: add Python 3.15 support in CI and pyproject.toml * feat: add `client.extract` dedicated method for Extract API * docs: align Extract examples and tests with current API Document direct Markdown output and current JSON fields, and consolidate Extract coverage into one test file. Correct the documentation validator for successful empty Google pagination responses and refresh examples. Co-authored-by: Codex <codex@openai.com> --------- Co-authored-by: Codex <codex@openai.com>
1 parent 30e7d1e commit 7f2dec3

14 files changed

Lines changed: 429 additions & 15 deletions

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ jobs:
1515
runs-on: ubuntu-latest
1616
strategy:
1717
matrix:
18-
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
18+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
1919

2020
steps:
2121
- name: Checkout repository
@@ -36,7 +36,7 @@ jobs:
3636
API_KEY: ${{ secrets.API_KEY }}
3737

3838
- name: Run example tests
39-
if: matrix.python-version == '3.14'
39+
if: matrix.python-version == '3.15'
4040
run: pytest tests/example_search_*_test.py
4141
env:
4242
API_KEY: ${{ secrets.API_KEY }}

‎.github/workflows/release.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ jobs:
2020
runs-on: ubuntu-latest
2121
strategy:
2222
matrix:
23-
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
23+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "3.15"]
2424
steps:
2525
- uses: actions/checkout@v6
2626

@@ -43,7 +43,7 @@ jobs:
4343
API_KEY: ${{ secrets.API_KEY }}
4444

4545
- name: Run engine and documentation examples
46-
if: matrix.python-version == '3.14'
46+
if: matrix.python-version == '3.15'
4747
continue-on-error: ${{ inputs.allow_example_test_failures == true }}
4848
run: pytest --ignore=tests/test_docs_publishing.py -k "example" --require-docs-key
4949
env:

‎CONTRIBUTING.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Bug reports and pull requests are welcome on GitHub. Run commands from the repos
44

55
## Development setup
66

7-
Use Python 3.13 to work on the package and its documentation. The SDK CI also tests Python 3.8 through 3.14. Create a virtual environment and install the package with its test and documentation dependencies:
7+
The SDK CI tests Python 3.8 through 3.15. Create a virtual environment and install the package with its test and documentation dependencies:
88

99
```sh
1010
uv venv --python 3.13
@@ -31,10 +31,10 @@ The commands below use `python` from that environment.
3131

3232
### Checks that need no API key
3333

34-
These tests cover response parsing, image uploads with simulated responses, timeouts, exceptions, documentation syntax, and the example runner:
34+
These tests cover response parsing, image uploads and extraction with simulated responses, timeouts, exceptions, documentation syntax, and the example runner:
3535

3636
```sh
37-
python -m pytest tests/test_output_formats.py tests/test_image_upload.py tests/test_timeout.py tests/test_exceptions.py tests/test_docs_examples.py tests/test_docs_example_runner.py -k 'not live' -q
37+
python -m pytest tests/test_output_formats.py tests/test_image_upload.py tests/test_extract.py tests/test_timeout.py tests/test_exceptions.py tests/test_docs_examples.py tests/test_docs_example_runner.py -k 'not live' -q
3838
```
3939

4040
The file list limits this command to tests that make no external requests. Applying `-k 'not live'` to the entire suite is not enough to run offline: some SDK integration tests do not have `live` in their names.
@@ -61,7 +61,7 @@ This includes live account, location, search, and pagination checks. To run ever
6161
python -m pytest -q
6262
```
6363

64-
Both commands make real SerpApi requests. The full suite can use more searches than a test run limited to the files you changed. PR CI also runs the standalone engine examples on Python 3.14. Documentation tests run in the package and documentation release workflows on release tags, and in manual documentation runs on `master`.
64+
Both commands make real SerpApi requests. The full suite can use more searches than a test run limited to the files you changed. PR CI also runs the standalone engine examples on Python 3.15. Documentation tests run in the package and documentation release workflows on release tags, and in manual documentation runs on `master`.
6565

6666
### Testing documentation examples
6767

‎README.md‎

Lines changed: 62 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,48 @@ Environment variables are a secure, safe, and easy way to manage secrets.
4545
Set `export SERPAPI_KEY=<secret_serpapi_key>` in your shell.
4646
Python accesses these variables from `os.environ["SERPAPI_KEY"]`.
4747

48+
### Extract content from a URL
49+
50+
The [Extract API](https://serpapi.com/extract-api) supports single-page scraping, URL discovery, and crawling. Call `client.extract()` with a parameter dictionary or keyword arguments. The client sets `engine="extract"` automatically.
51+
52+
Set `output="md"` to get the page's main content as a Markdown string:
53+
54+
```python
55+
import os
56+
import serpapi
57+
58+
client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60)
59+
markdown = client.extract(url="https://serpapi.com/", output="md")
60+
print(markdown)
61+
```
62+
63+
JSON is the default output format. It returns a dictionary-like `SerpResults` object with page information, structured data, links, and Markdown:
64+
65+
```python
66+
results = client.extract(url="https://serpapi.com/")
67+
print(results["page_information"]["title"])
68+
print(results["markdown"])
69+
```
70+
71+
Use `output="html"` to get the retrieved HTML as a string:
72+
73+
```python
74+
html = client.extract(url="https://serpapi.com/", output="html")
75+
```
76+
77+
Use `mode="map"` to discover URLs or `mode="crawl"` to extract multiple pages:
78+
79+
```python
80+
urls = client.extract(url="https://serpapi.com/", mode="map", include_sitemap=False)
81+
pages = client.extract(
82+
url="https://serpapi.com/", mode="crawl", max_pages=2, max_depth=1, timeout=180
83+
)
84+
```
85+
86+
Crawl mode can consume one search credit per successfully extracted page, including the starting page. Set `max_pages` to limit the number of pages.
87+
88+
See the [Extract API guide](https://github.com/serpapi/serpapi-python/blob/master/docs/user_guide/extract.md) for response examples and more options.
89+
4890
### Error handling
4991

5092
Unsuccessful requests raise `serpapi.HTTPError` or `serpapi.TimeoutError` exceptions. The returned status code will reflect the sort of error that occurred, please refer to [Status and Error Codes Documentation](https://serpapi.com/api-status-and-error-codes) for more details.
@@ -267,15 +309,30 @@ results = client.search({
267309
- API Documentation: [serpapi.com/google-autocomplete-api](https://serpapi.com/google-autocomplete-api)
268310

269311
### Search Google Immersive Product
312+
313+
Get a current `page_token` from Google Shopping before requesting product details:
314+
270315
```python
271316
import os
272317
import serpapi
273318

274319
client = serpapi.Client(api_key=os.getenv("API_KEY"))
275-
results = client.search({
276-
'engine': 'google_immersive_product',
277-
'page_token': 'eyJlaSI6Im5ZVmxaOXVVTDY2X3A4NFBqTnZELUFjIiwicHJvZHVjdGlkIjoiIiwiY2F0YWxvZ2lkIjoiNTE1NDU2NTc1NTc5MzcxMDY3NSIsImhlYWRsaW5lT2ZmZXJEb2NpZCI6IjI1MDkyMjcwMDUzMjk2NzQwODMiLCJpbWFnZURvY2lkIjoiMTYzOTg5MjU0MDcwMDU4MDA1NTQiLCJyZHMiOiJQQ18zNDg4MDE0MTg3ODgxNzc5NjU0fFBST0RfUENfMzQ4ODAxNDE4Nzg4MTc3OTY1NCIsInF1ZXJ5IjoibGcrdHYiLCJncGNpZCI6IjM0ODgwMTQxODc4ODE3Nzk2NTQiLCJtaWQiOiI1NzY0NjI3ODM3Nzc5MTUzMTMiLCJwdnQiOiJoZyIsInV1bGUiOm51bGx9=',
278-
})
320+
shopping = client.search(
321+
engine="google_shopping",
322+
q="lg tv",
323+
gl="us",
324+
hl="en",
325+
no_cache=True,
326+
)
327+
page_token = next(
328+
product["immersive_product_page_token"]
329+
for product in shopping["shopping_results"]
330+
if product.get("immersive_product_page_token")
331+
)
332+
results = client.search(
333+
engine="google_immersive_product",
334+
page_token=page_token,
335+
)
279336
```
280337
- API Documentation: [serpapi.com/google-immersive-product-api](https://serpapi.com/google-immersive-product-api)
281338

@@ -287,7 +344,7 @@ import serpapi
287344
client = serpapi.Client(api_key=os.getenv("API_KEY"))
288345
results = client.search({
289346
'engine': 'google_reverse_image',
290-
'image_url': 'https://i.imgur.com/5bGzZi7.jpg',
347+
'image_url': 'https://upload.wikimedia.org/wikipedia/commons/6/63/Wikipedia-logo.png',
291348
'max_results': '1',
292349
})
293350
```

‎docs/index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ Use the same search parameter names as the [SerpApi API documentation](https://s
9090
- [Getting Started](user_guide/getting-started.md) covers installation and your first search. [Client Usage](user_guide/client-usage.md) explains how to make requests and read responses.
9191
- [AI Agents](ai-agents.md) covers search tools, Markdown results, and API references for agents.
9292
- [Output Formats](user_guide/output-formats.md) explains when to use JSON, Markdown, or HTML.
93+
- [Extract API](user_guide/extract.md) covers page content in Markdown or JSON, URL discovery, and crawling.
9394
- [Migration Guide](user_guide/migrating-from-google-search-results.md) shows how to replace `google-search-results` with this package.
9495
- [Parameters and Engines](user_guide/parameters-and-engines.md) explains search parameters and how to test them in the [SerpApi Playground](https://serpapi.com/playground).
9596
- For scripts that collect results, see [pagination](user_guide/pagination.md), [timeouts and errors](user_guide/errors-and-timeouts.md), [request options](user_guide/request-options.md), and [account and locations](user_guide/account-and-locations.md).
@@ -113,6 +114,7 @@ user_guide/client-usage
113114
User Guide Overview <user_guide/index>
114115
user_guide/parameters-and-engines
115116
user_guide/output-formats
117+
user_guide/extract
116118
user_guide/pagination
117119
user_guide/errors-and-timeouts
118120
user_guide/account-and-locations

‎docs/user_guide/client-usage.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,10 +148,13 @@ For one additional page, call `next_page()`:
148148
next_results = results.next_page()
149149
```
150150

151-
Use `yield_pages()` in a loop to read the current page and fetch more pages. This example stops after ten pages, or earlier if there is no next page:
151+
Use `yield_pages()` in a loop to read the current page and fetch more pages. This example reads up to ten pages and stops if a page contains an `error` message:
152152

153153
```python
154154
for page_number, page in enumerate(results.yield_pages(max_pages=10), start=1):
155+
if "error" in page:
156+
print(page["error"])
157+
break
155158
current = page.get("serpapi_pagination", {}).get("current", page_number)
156159
print(current)
157160
```

‎docs/user_guide/extract.md‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
---
2+
title: "Extract API"
3+
description: "Extract page content, discover URLs, and crawl websites with the Python client."
4+
---
5+
6+
# Extract API
7+
8+
Use `client.extract()` to read content from a public URL. It supports single-page scraping, URL discovery, and crawling. Pass a parameter dictionary or keyword arguments, as you would with `client.search()`. The client sets `engine="extract"` automatically.
9+
10+
See the [Extract API documentation](https://serpapi.com/extract-api) for all parameters and response fields.
11+
12+
## Get Page Content as Markdown
13+
14+
Set `output="md"` to get the page's main content as a Markdown string.
15+
16+
After [setting your API key](getting-started.md#installation), run:
17+
18+
```python
19+
import os
20+
import serpapi
21+
22+
client = serpapi.Client(api_key=os.environ["SERPAPI_KEY"], timeout=60)
23+
markdown = client.extract(
24+
url="https://serpapi.com/",
25+
output="md",
26+
)
27+
28+
print(markdown)
29+
```
30+
31+
You can pass the returned string to an AI agent or save it to a file.
32+
33+
## Get Structured Data as JSON
34+
35+
JSON is the default output format. The SDK returns a dictionary-like `SerpResults` object with page information, meta tags, structured data, links, images, and Markdown. The sections returned depend on the target page.
36+
37+
```python
38+
results = client.extract(url="https://serpapi.com/")
39+
40+
print(results["page_information"]["title"])
41+
print(results["markdown"])
42+
```
43+
44+
The `markdown` field contains the page's main content. Set `include_markdown=False` to omit it from the JSON response.
45+
46+
The module-level `serpapi.extract(url=..., api_key=...)` function accepts the same parameters.
47+
48+
## Get HTML
49+
50+
Set `output="html"` to get the retrieved HTML as a string:
51+
52+
```python
53+
html = client.extract(url="https://serpapi.com/", output="html")
54+
```
55+
56+
## Discover URLs
57+
58+
Set `mode="map"` to discover URLs on a website:
59+
60+
```python
61+
results = client.extract(
62+
url="https://serpapi.com/",
63+
mode="map",
64+
include_sitemap=False,
65+
include_subdomains=False,
66+
)
67+
68+
for entry in results["urls"]:
69+
print(entry["url"])
70+
```
71+
72+
Map mode includes sitemap URLs by default. Set `include_sitemap=False` to discover URLs from the page's links only. Set `include_subdomains=True` to include subdomains.
73+
74+
## Crawl Multiple Pages
75+
76+
Set `mode="crawl"` to follow links and extract pages:
77+
78+
```python
79+
results = client.extract(
80+
url="https://serpapi.com/",
81+
mode="crawl",
82+
max_pages=2,
83+
max_depth=1,
84+
timeout=180,
85+
)
86+
87+
for page in results["pages"]:
88+
print(page["url"])
89+
print(page["markdown"])
90+
```
91+
92+
Each entry in `results["pages"]` contains the extracted page's data, including its `markdown` field. The starting page has depth `0`. `max_pages` includes the starting page and defaults to `10`. Each successfully extracted page can consume one search credit. Set a page limit for your crawl and allow enough time for the HTTP request to finish.
93+
94+
For requests you want to retrieve later, pass `"async": True` in a parameter dictionary and use the [Search Archive](async-search-archive.md).
95+
96+
## Timeouts and Content Filters
97+
98+
The `timeout=` keyword sets the SDK's HTTP timeout in seconds. Set it above the API fetch timeout, and allow time for all pages when crawling. Put the Extract API's fetch timeout in the parameter dictionary to send it in milliseconds:
99+
100+
```python
101+
results = client.extract(
102+
{"url": "https://serpapi.com/", "timeout": 30000},
103+
timeout=60,
104+
)
105+
```
106+
107+
Use `include_tags` and `exclude_tags` to keep or remove content with CSS selectors, such as `include_tags="article, main"` or `exclude_tags="nav, footer"`. See the [Extract API parameters](https://serpapi.com/extract-api) for details and [Errors and Timeouts](errors-and-timeouts.md) for exception handling.

‎docs/user_guide/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ Start with [Getting Started](getting-started.md) to install the package and run
1111

1212
- [Parameters and Engines](parameters-and-engines.md)
1313
- [Output Formats](output-formats.md)
14+
- [Extract API](extract.md)
1415
- [Pagination](pagination.md)
1516
- [Errors and Timeouts](errors-and-timeouts.md)
1617
- [Account and Locations](account-and-locations.md)

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ classifiers = [
2929
"Programming Language :: Python :: 3.12",
3030
"Programming Language :: Python :: 3.13",
3131
"Programming Language :: Python :: 3.14",
32+
"Programming Language :: Python :: 3.15",
3233
"Programming Language :: Python :: Implementation :: CPython",
3334
"Natural Language :: English",
3435
"Topic :: Utilities",

‎serpapi/core.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,32 @@ def search(self, params: dict = None, **kwargs):
5555

5656
return SerpResults.from_http_response(r, client=self)
5757

58+
def extract(self, params: dict = None, **kwargs):
59+
"""Extract content from a public URL using the Extract API.
60+
61+
Accepts a parameter dictionary or keyword arguments, like :meth:`search`.
62+
Sets ``engine=extract`` and returns a :class:`SerpResults <serpapi.SerpResults>` object for JSON, or text for HTML and Markdown.
63+
64+
.. code-block:: python
65+
66+
results = client.extract(url="https://example.com/")
67+
markdown = client.extract(url="https://example.com/", output="md")
68+
results = client.extract({"url": "https://example.com/", "timeout": 30000}, timeout=60)
69+
70+
:param params: Optional mapping of Extract API parameters, including the required ``url``. Put ``timeout`` here to set the API fetch timeout in milliseconds.
71+
:param kwargs: Additional API parameters or HTTP request options, as in :meth:`search`. The ``timeout`` keyword controls the HTTP client timeout in seconds.
72+
73+
``mode`` accepts ``scrape`` (default), ``map``, or ``crawl``. ``output`` accepts ``json`` (default), ``html``, or ``md``.
74+
JSON includes page content in the top-level ``markdown`` field by default,
75+
or in each ``pages`` entry for crawls. Set ``include_markdown=False`` to omit it.
76+
For a scraped page, ``output="md"`` returns the page Markdown without search metadata.
77+
78+
Crawl mode can consume one search credit per successfully extracted page. Use ``max_pages`` to limit the number of pages.
79+
"""
80+
params = dict(params) if params is not None else {}
81+
kwargs["engine"] = "extract"
82+
return self.search(params, **kwargs)
83+
5884
def search_archive(self, params: dict = None, **kwargs):
5985
"""Get a result from the SerpApi Search Archive API.
6086
@@ -190,6 +216,7 @@ def account(self, params: dict = None, **kwargs):
190216
# An un-authenticated client instance.
191217
_client = Client()
192218
search = _client.search
219+
extract = _client.extract
193220
search_archive = _client.search_archive
194221
upload_image = _client.upload_image
195222
locations = _client.locations

0 commit comments

Comments
 (0)