Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1086,4 +1086,4 @@ firecrawl alexandria feedback --rating partial \
--rationale "Found summaries but could not retrieve attachments" --json
```

The optional `--objective` is the underlying goal of the session: what you or your user were ultimately trying to accomplish, beyond the single website. No job ID is required. Alexandria session feedback has no job-age deadline and does not refund credits. Optional `--provider-feedback` and `--capability-feedback` accept JSON arrays; see `firecrawl alexandria feedback --help` for their fields and issue codes. Capability issue codes are `new_capability_request` (requires `requestedFunctionality`), `missing_capability` (the provider exists but lacks this capability), `insufficient_functionality`, `incorrect_result`, `execution_error`, and `other`. Existing `feedback` and `search-feedback` commands retain their job-specific behavior. Endpoint feedback opt-out environment variables also apply to this command.
The optional `--objective` is the underlying goal of the session: what you or your user were ultimately trying to accomplish, beyond the single website. No job ID is required. Like search feedback, it must be sent within about 2 minutes (the search feedback window) of the team's most recent Alexandria search, discovery, or execution; later submissions are rejected with `FEEDBACK_WINDOW_EXPIRED`. Each submission refunds 1 credit, up to 10 credits per website and 100 credits per team each UTC day; past either cap, submissions are still recorded and return `websiteCapReached` or `dailyCapReached`. Optional `--provider-feedback` and `--capability-feedback` accept JSON arrays; see `firecrawl alexandria feedback --help` for their fields and issue codes. Capability issue codes are `new_capability_request` (requires `requestedFunctionality`), `missing_capability` (the provider exists but lacks this capability), `insufficient_functionality`, `incorrect_result`, `execution_error`, and `other`. Existing `feedback` and `search-feedback` commands retain their job-specific behavior. Endpoint feedback opt-out environment variables also apply to this command.
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "firecrawl-cli",
"version": "1.26.0",
"version": "1.26.1",
"publishConfig": {
"tag": "latest"
},
Expand Down
2 changes: 1 addition & 1 deletion skills/firecrawl-agent/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,4 +60,4 @@ firecrawl agent "<job-id>" --cancel

## Alexandria session feedback

To report an Alexandria session outcome or a provider/capability gap, use `firecrawl alexandria feedback --rating good|partial|bad --url <website> --requested-functionality '<what was needed>' --objective '<the underlying goal of the task>' --rationale '<what happened>' --json`. Use observed results in the rationale. No job ID is needed; this session feedback has no job-age deadline and no credit refund. Optional `--provider-feedback` and `--capability-feedback` JSON arrays describe specific gaps; inspect `firecrawl alexandria feedback --help` for their fields. Use the capability issue `missing_capability` when a provider exists but lacks the needed capability, and `new_capability_request` (with `requestedFunctionality`) to ask for one.
To report an Alexandria session outcome or a provider/capability gap, use `firecrawl alexandria feedback --rating good|partial|bad --url <website> --requested-functionality '<what was needed>' --objective '<the underlying goal of the task>' --rationale '<what happened>' --json`. Use observed results in the rationale. No job ID is needed; send it within about 2 minutes of your last Alexandria search, discovery, or execution, or it is rejected. Each submission refunds 1 credit, up to 10 per website and 100 per team each UTC day. Optional `--provider-feedback` and `--capability-feedback` JSON arrays describe specific gaps; inspect `firecrawl alexandria feedback --help` for their fields. Use the capability issue `missing_capability` when a provider exists but lacks the needed capability, and `new_capability_request` (with `requestedFunctionality`) to ask for one.
11 changes: 7 additions & 4 deletions skills/firecrawl-alexandria/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Alexandria brings ready-made website workflows, API providers, and specialized i

Use ordinary web results when they answer the question; use a provider tool when its coverage and inputs fit.

## Alexandria feedback
## Alexandria feedback (refunds 1 credit)

Alexandria coverage grows from what agents report. If you choose to report how the catalogue served a task, send at most one `firecrawl alexandria feedback` per website you needed data from after finishing the task. It is free: no job ID, no time window, no credit refund.
Alexandria coverage grows from what agents report. Send one `firecrawl alexandria feedback` per website you needed data from, right after your last Alexandria call for it. No job ID is needed. Each feedback refunds 1 credit, up to 10 credits per website and 100 per team each UTC day.

Feedback can describe any of these outcomes:

Expand All @@ -26,16 +26,19 @@ Feedback can describe any of these outcomes:

**Rules to know before you call this:**

- **Time window:** must be sent within ~2 minutes of your team's most recent Alexandria search, discovery, or execution. Each Alexandria call restarts the window. Late feedback is rejected (`feedbackErrorCode: "FEEDBACK_WINDOW_EXPIRED"`).
- **`--url` is the website the user needed data from**, not the provider and not a Firecrawl page. `--requested-functionality` is what they needed from it, in one sentence. These two fields are the most important: they aggregate across teams and tell us which sites and workflows to add next.
- **`--objective` is the underlying goal** behind the session: what you or your user were ultimately trying to accomplish, in one sentence (for example, "Shortlist federal IT contracts to bid on this quarter"). It is broader than `--requested-functionality`, which covers only this website.
- **`--rationale` explains the rating** from observed results: which provider or capability served or failed the need, and how. Two or three sentences, no raw results pasted in.
- **`--provider-feedback`** is a JSON array of `{name, issue, why}` for providers that were missing, thin, or unavailable. Issues: `missing_provider` (no provider covers the site), `insufficient_coverage` (exists, but data was thin, stale, or partial for this market or segment), `provider_unavailable` (could not be called), `other`.
- **`--capability-feedback`** is a JSON array of `{name, provider, issue, why, requestedFunctionality?}` for capabilities that were missing, wrong, or failed. Issues: `new_capability_request` (ask the provider to add one; `requestedFunctionality` required), `missing_capability` (provider exists but lacks it), `insufficient_functionality` (exists but cannot take the input or filter you needed), `incorrect_result`, `execution_error`, `other`. Use `name` and `provider` exactly as discovery returned them; for a capability that does not exist yet, name what it should be.
- **Rate honestly:** `good` when a tool answered the need, `partial` when it answered some of it or with gaps, `bad` when nothing available answered it or what ran was wrong or failed.
- **Rate honestly:** `good` when a tool answered the need, `partial` when it answered some of it or with gaps, `bad` when nothing available answered it or what ran was wrong or failed. Every rating gets the same refund.
- **Website refund cap (per website, per UTC day, default 10 credits).** Past it, feedback about that website is still recorded but refunds nothing, and the response sets `websiteCapReached: true`. Feedback about other websites still refunds.
- **Daily refund cap (per team, per UTC day, default 100 credits).** Past the cap, feedback is still recorded but refunds nothing. The response includes `creditsRefundedToday`, `dailyRefundCap`, and `dailyCapReached`. **When `dailyCapReached: true`, stop sending Alexandria feedback for the rest of the UTC day.**
- **`--silent &`** is the right pattern: exit code 0 even on failure, so a rejected call never crashes your pipeline.

```bash
# Example: send at most once per website after the task is done. Replace the
# Example: send once per website, within ~2 minutes of your last Alexandria call. Replace the
# placeholders with what actually happened; drop --provider-feedback or
# --capability-feedback when there is nothing to report at that level.
firecrawl alexandria feedback \
Expand Down
2 changes: 1 addition & 1 deletion skills/firecrawl-search/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Use `list` for category/provider browsing and selected contracts. For a known we

If no returned tool covers the country/market/segment or required inputs, continue with ordinary web results. Do not exhaust the catalogue or pay for adjacent tools just to probe coverage. `--sources web` explicitly opts out of Alexandria; `--sources web --domain-tools` retains domain matches only.

For Alexandria feedback about a provider result or coverage gap, see [firecrawl-alexandria](../firecrawl-alexandria/SKILL.md).
For Alexandria feedback about a provider result or coverage gap (each refunds 1 credit, up to 10 per website and 100 per team each UTC day), see [firecrawl-alexandria](../firecrawl-alexandria/SKILL.md).

## Progressive discovery and output handling

Expand Down
2 changes: 1 addition & 1 deletion skills/firecrawl/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ Single format outputs raw content. Multiple formats (e.g., `--format markdown,li

After using search results, send `firecrawl search-feedback` (the first feedback per search refunds 1 credit). The full pattern, guard, and rules live in [firecrawl-search](../firecrawl-search/SKILL.md).

For Alexandria feedback about a provider result or coverage gap, see [firecrawl-alexandria](../firecrawl-alexandria/SKILL.md).
For Alexandria feedback about a provider result or coverage gap (each refunds 1 credit, up to 10 per website and 100 per team each UTC day), see [firecrawl-alexandria](../firecrawl-alexandria/SKILL.md).

For non-search endpoint jobs, use `firecrawl feedback <endpoint> <jobId>` to send concise job-level feedback through `/v2/feedback`. Supported endpoints are `search`, `scrape`, `parse`, and `map`.

Expand Down
84 changes: 84 additions & 0 deletions src/__tests__/commands/feedback.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,90 @@ describe('executeEndpointFeedback', () => {
stdoutSpy.mockRestore();
}
});

it.each([
[
{ creditsRefunded: 1, creditsRefundedToday: 1, dailyRefundCap: 100 },
['Feedback recorded.', 'Credits refunded: 1', 'Refunds today: 1 / 100'],
],
[
{
creditsRefunded: 0,
creditsRefundedToday: 10,
dailyRefundCap: 100,
websiteCapReached: true,
warning: 'Daily refund cap reached for feedback about example.com.',
},
[
'Feedback recorded.',
'Credits refunded: 0',
'Daily refund cap reached for this website',
'Warning: Daily refund cap reached for feedback about example.com.',
],
],
])(
'prints the Alexandria feedback refund outcome %#',
async (body, expected) => {
mockFetch.mockResolvedValue({
ok: true,
status: 200,
json: async () => ({ success: true, feedbackId: 'fb', ...body }),
});
const stdoutSpy = vi
.spyOn(process.stdout, 'write')
.mockImplementation(() => true);
try {
await handleEndpointFeedbackCommand({
endpoint: 'alexandria',
rating: 'partial',
requestedWebsite: {
url: 'https://example.com',
requestedFunctionality: 'Download attachments',
},
rationale: 'Only summaries available',
});
const output = stdoutSpy.mock.calls.map(([chunk]) => chunk).join('');
for (const line of expected) expect(output).toContain(line);
} finally {
stdoutSpy.mockRestore();
}
}
);

it('keeps websiteCapReached in JSON output', async () => {
mockFetch.mockResolvedValue({
ok: true,
status: 200,
json: async () => ({
success: true,
feedbackId: 'fb',
creditsRefunded: 0,
websiteCapReached: true,
}),
});
const stdoutSpy = vi
.spyOn(process.stdout, 'write')
.mockImplementation(() => true);
try {
await handleEndpointFeedbackCommand({
endpoint: 'alexandria',
rating: 'good',
requestedWebsite: {
url: 'https://example.com',
requestedFunctionality: 'Download attachments',
},
rationale: 'All attachments returned',
json: true,
});
const output = stdoutSpy.mock.calls.map(([chunk]) => chunk).join('');
expect(JSON.parse(output)).toMatchObject({
creditsRefunded: 0,
websiteCapReached: true,
});
} finally {
stdoutSpy.mockRestore();
}
});
});

describe('feedback parsing', () => {
Expand Down
2 changes: 1 addition & 1 deletion src/commands/alexandria-feedback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ export function parseAlexandriaFeedbackArray(
export function createAlexandriaFeedbackCommand(): Command {
return new Command('feedback')
.description(
'Report Alexandria session results, provider gaps, or capability issues. No job ID, job-age limit, or credit refund.'
'Report Alexandria session results, provider gaps, or capability issues. No job ID needed; send within about 2 minutes of your last Alexandria search, discovery, or execution. Refunds 1 credit per submission, up to 10 per website and 100 per team each UTC day.'
)
.requiredOption(
'--rating <rating>',
Expand Down
7 changes: 7 additions & 0 deletions src/commands/feedback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ export interface EndpointFeedbackResult {
creditsRefundedToday?: number;
dailyRefundCap?: number;
dailyCapReached?: boolean;
websiteCapReached?: boolean;
alreadySubmitted?: boolean;
warning?: string;
error?: string;
Expand Down Expand Up @@ -346,6 +347,7 @@ export async function executeEndpointFeedback(
? data.dailyRefundCap
: undefined,
dailyCapReached: data.dailyCapReached === true,
...(data.websiteCapReached === true ? { websiteCapReached: true } : {}),
alreadySubmitted: data.alreadySubmitted,
warning: data.warning,
};
Expand Down Expand Up @@ -380,6 +382,10 @@ function formatReadable(result: EndpointFeedbackResult): string {
lines.push(
'Daily refund cap reached; further feedback calls today will not refund credits.'
);
} else if (result.websiteCapReached) {
lines.push(
'Daily refund cap reached for this website; feedback about other websites can still refund credits.'
);
}
if (result.warning) {
lines.push(`Warning: ${result.warning}`);
Expand Down Expand Up @@ -431,6 +437,7 @@ export async function handleEndpointFeedbackCommand(
? { dailyRefundCap: result.dailyRefundCap }
: {}),
...(result.dailyCapReached ? { dailyCapReached: true } : {}),
...(result.websiteCapReached ? { websiteCapReached: true } : {}),
...(result.alreadySubmitted ? { alreadySubmitted: true } : {}),
...(result.warning ? { warning: result.warning } : {}),
};
Expand Down
Loading