Skip to content

fix(scrape): improve error handling and response propagation for scraping failures - #118

Merged
HC-ONLINE merged 1 commit into
mainfrom
fix/scrape-failure-propagation
Oct 16, 2026
Merged

HC-ONLINE merged 1 commit into
mainfrom
fix/scrape-failure-propagation

Conversation

@HC-ONLINE

Copy link
Copy Markdown
Owner

Description

Scraping failures reported by the core engine (success=False — e.g. HTTP 4xx/5xx responses, connection errors, robots.txt blocks, browser timeouts) were previously surfaced as successful operations: ScrapeService returned success=True, the API answered HTTP 200 with success=true, the CLI could exit with code 0, and batches counted failed URLs as successes (leaving error=None when the whole batch failed).

This PR fixes the core/service boundary so engine failures propagate correctly:

  • _scrape_static() and _scrape_dynamic() inspect the core result and raise ExecutionError (with url/status_code details) when the engine reports failure; already-contextualized ExecutionErrors are re-raised instead of being wrapped again.
  • scrape_multiple() classifies individual failures as failures (warnings include the reason), and a fully failed batch returns an aggregate error with error_code=SCRAPE_ERROR.
  • The API no longer re-wraps its own HTTPExceptions (except HTTPException: raise), so error details stay clean; real failures return HTTP 500 instead of HTTP 200 with success=true.
  • The CLI scrape batch command prints failure warnings on partial batches.
  • Dynamic failures no longer fabricate status_code=200 metadata.

quick_service needed no code change: with the service fix it logs the real reason (e.g. Scrape failed: HTTP 404) and no longer incorporates failed scrapes into the report.

Related Issue

None.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update

Checklist

  • I have followed the style guidelines of this project (Ruff & Pyright)
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the documentation accordingly

Screenshots (if applicable)

Not applicable.

@HC-ONLINE
HC-ONLINE merged commit 378d377 into main Oct 16, 2026
8 checks passed
@HC-ONLINE
HC-ONLINE deleted the fix/scrape-failure-propagation branch October 16, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant