chore: sync fork with upstream/main while preserving custom overlays#3
Open
cayman-openclaw wants to merge 309 commits intomainfrom
Open
chore: sync fork with upstream/main while preserving custom overlays#3cayman-openclaw wants to merge 309 commits intomainfrom
cayman-openclaw wants to merge 309 commits intomainfrom
Conversation
…awl#2722) The "next" pagination URL in the v2 batch scrape status endpoint was hardcoded to use /v1/ instead of /v2/, causing clients using the v2 API to receive incorrect URLs for fetching subsequent pages. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev>
…l#2724) * fix(js-sdk): add Zod v4 compatibility for schema conversion Replace direct usage of deprecated `zod-to-json-schema` package with a new utility that supports both Zod v3 and v4. The `zod-to-json-schema` package is incompatible with Zod v4, causing schema conversion failures for users who have upgraded to Zod v4. Changes: - Create zodSchemaToJson.ts utility with Zod v3/v4 detection - Add isZodSchema() function to detect Zod schemas - Add safeZodSchemaToJsonSchema() for graceful conversion - Update validation.ts, extract.ts, agent.ts, and v1/index.ts - Add comprehensive unit tests for the new utility The utility: - Detects Zod v4 schemas by checking for _zod property - Falls back to zod-to-json-schema for Zod v3 schemas - Provides safeZodSchemaToJsonSchema for graceful fallback Co-Authored-By: rafaelsideguide <rafael@sideguide.dev> * refactor(js-sdk): simplify zodSchemaToJson utility and tests - Rename safeZodSchemaToJsonSchema to zodSchemaToJsonSchema (redundant prefix) - Remove unnecessary comments explaining obvious behavior - Condense 47 unit tests into 5 comprehensive tests - Clean up redundant code Co-Authored-By: rafaelsideguide <rafael@sideguide.dev> * mv test to right spot * bump version --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: rafaelsideguide <rafael@sideguide.dev> Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
* feat(crawl): add sitemap-only support Co-Authored-By: mogery <mogery@sideguide.dev> * chore(sdk): bump js/python versions Co-Authored-By: mogery <mogery@sideguide.dev> --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: mogery <mogery@sideguide.dev>
…r-mmjv-4gpg) (firecrawl#2728) - Upgrade lodash to 4.17.23 in apps/api (direct dependency) - Add pnpm override for lodash in apps/test-suite (transitive via artillery) This fixes the audit-ci failures caused by CVE-2025-13465, a prototype pollution vulnerability in lodash's _.unset and _.omit functions. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: mogery <mogery@sideguide.dev>
* Add manual pagination helpers for crawl and batch * Bump python sdk version --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com>
* feat(agent): include model in status responses Co-Authored-By: mogery <mogery@sideguide.dev> * fix(agent): narrow model response type Co-Authored-By: mogery <mogery@sideguide.dev> * bump SDK versions --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: mogery <mogery@sideguide.dev> Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
…planations (firecrawl#2697) * feat(errors): improve all user-facing error messages with detailed explanations Updated all scrape error messages to be much more descriptive and actionable: - DenialReason enum: Added detailed explanations for all crawl filter denial reasons (depth limit, include/exclude patterns, robots.txt, file types, backward crawling, external links, etc.) with specific parameter names and remediation steps - RacedRedirectError: Explains that this is expected behavior to prevent duplicate scrapes - MapTimeoutError: Explains timeout causes and suggests increasing timeout or using more specific URLs - ScrapeJobTimeoutError: Comprehensive explanation of timeout causes and solutions - NoEnginesLeftError: Lists engines tried and common failure causes - SiteError: Maps common error codes to human-readable explanations (ERR_NAME_NOT_RESOLVED, ERR_CONNECTION_REFUSED, etc.) - SSLError: Explains certificate issues and when to use skipTlsVerification - ProxySelectionError: Explains how to fix location/proxy configuration - ActionError: Explains action failures with selector debugging tips - PDFAntibotError/DocumentAntibotError: Explains anti-bot protection and workarounds - NoCachedDataError: Explains maxAge requirements and how to get fresh data - ZDRViolationError: Explains Zero Data Retention limitations - DNSResolutionError: Explains DNS failure causes - SitemapError: Explains sitemap fetch and parsing failures - BLOCKLISTED_URL_MESSAGE: Explains why URLs may be blocklisted - JobCancelledError: Explains cancellation behavior - UnknownError: More helpful internal error message Error messages now include: - Dynamic context (URL paths, patterns, depths, etc.) - Specific parameter names to adjust - Numbered remediation steps - Links to support when applicable Co-Authored-By: mogery <mogery@sideguide.dev> * fix(errors): revert antibot/action errors and fix ZDR/NoCachedData messages - Reverted PDFAntibotError, DocumentAntibotError, and ActionError to original messages - Fixed ZDRViolationError to remove redundant "caching" word and enterprise mention - Fixed NoCachedDataError to correctly reference minAge parameter instead of maxAge Co-Authored-By: mogery <mogery@sideguide.dev> * fix(errors): update discovery depth message to be verbose Co-Authored-By: mogery <mogery@sideguide.dev> * fixes --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: mogery <mogery@sideguide.dev> Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
…recrawl#2735) Add Abi as the code owner for all branding-related files in the repository to ensure proper review coverage. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: nickscamara <nick@sideguide.dev>
…recrawl#2746) Some pages override Array.from with custom implementations that don't handle Sets correctly, causing Array.from(picksSet) to return an array containing the Set itself instead of its elements. Replace Array.from(picksSet) with spread operator [...picksSet] which: - Works correctly even when Array.from is overridden - Is a standard JavaScript feature that pages can't easily break - Returns the actual elements from the Set Fixes TypeError: Failed to execute 'getComputedStyle' on 'Window': parameter 1 is not of type 'Element' when Set is passed to getStyleSnapshot.
…irecrawl#2754) Exports the Watcher class and WatcherOptions type from the main SDK entrypoint so users can access them for crawl/batch job monitoring. Fixes firecrawl#2731 Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: nickscamara <nick@sideguide.dev>
* fix(audit): Fix pnpm audit issues * fix(audit): allowlist js-yaml vulnerability in test-suite Add artillery>js-yaml path to allowlist for GHSA-mh29-5h37-fv8m since it's a test-only dependency where we control YAML input. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…n ingestion-ui (firecrawl#2765) Upgrade eslint from 8.x to 9.26.0+ to fix CVE-2025-50537 stack overflow vulnerability. This required migrating to the new flat config format. Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…2759) * feat(api): add enhanced proxy option as alias for stealth Add "enhanced" as a new proxy option that works identically to "stealth" under the hood. This is a non-breaking change that keeps "stealth" working for backwards compatibility while introducing a more descriptive name for the feature. Changes: - Add "enhanced" to proxy enum in API types (v1 and v2) - Handle "enhanced" the same as "stealth" in scrapeURL logic - Update timeout transformation to also consider "enhanced" - Update SDK types (JS and Python) to include "enhanced" - Update test helpers to verify "enhanced" behavior Co-Authored-By: nickscamara <nick@sideguide.dev> * Nick: --------- Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: nickscamara <nick@sideguide.dev> Co-authored-by: Nicolas <20311743+nickscamara@users.noreply.github.com>
Change notification emails to send from notifications@notifications.firecrawl.dev instead of firecrawl@getmendableai.com. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev>
* conc boost for agent interop * Update apps/api/src/controllers/v2/types.ts Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com> --------- Co-authored-by: Gergő Móricz <mo.geryy@gmail.com> Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Blacksmith CI provides test output directly, making the separate test reporting via dorny/test-reporter redundant. Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
…agent-sign-up-v0-plan feat(api): agent signup v0 endpoints and enforcement
…mail-debug-logs Add debug logs for agent sponsor email
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
…ter-link fix(docs): update social media links in README and footer/header components
…mendableai-with-firecrawl Replace mendableai gh org with firecrawl
* Nick: * Update package.json * Update pnpm-lock.yaml
…2989) Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev>
* feat: concurrency limit cap * chore: update queue full message
…crawl#2991) Remove the SUPABASE_ACUC_URL/SUPABASE_ACUC_SERVICE_TOKEN config and the supabase_acuc_only_service client. This was a separate Supabase instance used only for ACUC (auth/billing) queries, but it caused the dashboard to show wrong concurrency limits because firecrawl-web was calling the same ACUC SQL function against the main database while the API used the dedicated one, resulting in potentially different return values. With this change, all ACUC queries go through the main Supabase client (with read replica load balancing), ensuring the API and dashboard always see the same data. Co-authored-by: firecrawl-spring[bot] <254786068+firecrawl-spring[bot]@users.noreply.github.com> Co-authored-by: micahstairs <micah@sideguide.dev>
Add interactive live view URL support
…gnup-email-from fix(api): update sender email in agent signup controller
…gnup-email-from-2 fix(api): change sender email in agent signup controller to new notifications address
…mail-link fix(api): update sender email in agent signup controller to new notif…
…ivity and improve rate limiting logic
…specific rules for sideguide.dev test emails
…de.dev test emails
…rove maintainability and clarity
…estability fix(api): enhance agent signup controller to handle email case sensit…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary\n- Merges latest upstream
firecrawl/firecrawlmain into fork integration branch\n- Preserves custom local overlays (docs/workflow folders such as.agent,.cursor, and project-specific guides)\n- No manual conflict resolutions were required in this merge\n\n## Drift snapshot before merge\n- Fork was behind upstream by 308 commits and ahead by 21 commits\n\n## Notes\n- This PR is intentionally large and should be reviewed as an upstream sync\n- Recommend validating key local workflows and docker compose paths after merge\n