From 3325e531c7e07b809bf5140824ba9e0fd485eb9f Mon Sep 17 00:00:00 2001 From: Manos Kaparos <> Date: Mon, 31 Aug 2026 01:20:38 +0300 Subject: [PATCH] fixes from the fourth adversarial review: repeat guard, pinned parquet, dev-set labels, narrative corrections Co-Authored-By: Claude Fable 5 --- README.md | 15 +++--- REPRODUCE.md | 10 ++-- .../astropy__astropy-12544/human-labels.md | 1 + skills/fairtask-eval/SKILL.md | 8 +-- src/lib/audit.ts | 2 +- src/report.test.ts | 10 ++++ src/report.ts | 12 +++-- src/show.ts | 2 +- .../v3-verify/astropy__astropy-12544.md | 50 +++++++++---------- 9 files changed, 64 insertions(+), 46 deletions(-) create mode 100644 src/report.test.ts diff --git a/README.md b/README.md index 190ac52..569b973 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ It is a Claude Agent SDK pipeline (TypeScript) that reads the actual repository, verifies every piece of evidence they cite against the code, and returns a scored verdict with quotes a reviewer can check. Its output is measured against **OpenAI's public human annotations** for the same tasks. -**Result on 30 human-annotated cases (same cases, same deciding model for every system):** on the headline +**Result on 30 human-annotated cases (same cases and same deciding model for every system — a development set: these same 30 cases were used to diagnose failures and design every iteration, and there is no held-out evaluation):** on the headline metric — agreement with the human usable/flag decision — the agent pipeline **does not reliably beat the one-prompt baseline**: the baseline scores 67% on two independent runs, the agent variants 60–70% with a ±2-case run-to-run spread. What the agent changes is measurable elsewhere: the evidence it cites is **verifiable 100% of the time** @@ -355,7 +355,7 @@ a complete command; rows with the same script differ by one flag. | `npm run score -- baseline v3-verify --detail` | Adds one row per instance: human scores, predicted scores, decision, cost. | | `npm run score -- baseline v3-verify --common` | Restricts every run to the instances all listed runs scored (for comparing partial runs). | | `npm run score -- baseline --json` | The summary as JSON instead of the table. | -| `node src/report.ts --baseline baseline --final v3-verify` | Prints the headline comparison of those two runs as an aligned terminal table (the same rows as §4). | +| `node src/report.ts --baseline baseline --final v3-verify` | Prints the headline comparison of those two runs as an aligned terminal table (the same rows as §4). Needs cloned workspaces (`npm run data:workspaces`, ~20 s) — the bad-evidence rows re-verify every quote against the repositories. | | `node src/report.ts --baseline baseline --final v3-verify --markdown` | The same as a Markdown pipe table — what `scripts/finalize-report.py` pastes into this README. | | `node src/report.ts --baseline baseline --final v5-cheap-probes --final-repeat v5-rerun` | Shows the final configuration as "first run · repeat run". The repeat must be a run of the same configuration; the script refuses otherwise. | | `node src/report.ts --baseline baseline --final v3-verify --runs baseline,v1-context,v3-verify` | Adds the all-systems table with those runs as rows. | @@ -409,7 +409,7 @@ the corrective turn). The representative ones, one per agent and one per kind of | Agent / situation | Trajectory | What to look at | |---|---|---| -| **Baseline** — one prompt, no tools | [`trajectories/baseline/astropy__astropy-12544.md`](trajectories/baseline/astropy__astropy-12544.md) | Instructions and task prompt, then the verdict straight away. It flags the task (humans: flag, `false_negative` 3) but also scores the issue 2 where humans gave 0 — the right decision for half the right reasons; every later configuration keeps the flag and the same over-score on the issue axis. | +| **Baseline** — one prompt, no tools | [`trajectories/baseline/astropy__astropy-12544.md`](trajectories/baseline/astropy__astropy-12544.md) | Instructions and task prompt, then the verdict straight away. It flags the task (humans: flag, `false_negative` 3) but also scores the issue 2 where humans gave 0 — the right decision for half the right reasons; every configuration keeps the flag, while the issue-axis score wobbles between 1 and 2 across runs (§6). | | **Single agent with repository tools** (v1) | [`trajectories/v1-context/astropy__astropy-12544.md`](trajectories/v1-context/astropy__astropy-12544.md) | The same prompt plus `Read`/`Grep`/`Glob`: what it reads, and how reading the code made it *more* lenient (§5, Iteration 1). | | **Judge + `spec-probe` + `test-probe`** (v3, the final default) | [`trajectories/v3-verify/astropy__astropy-12544.md`](trajectories/v3-verify/astropy__astropy-12544.md) | Instructions for all three agents (`### Subagent spec-probe`, `### Subagent test-probe`); the judge dispatching both in parallel; each probe's `Read`/`Grep` calls and the file contents that came back; the two probe reports with evidence; the judge re-opening the cited lines itself ("Now spot-checking the strongest claims myself"); the final verdict; `✅ Verification passed`. | | **Verifier feedback and a corrective turn** | [`trajectories/v4-calibrated/scikit-learn__scikit-learn-11574.md`](trajectories/v4-calibrated/scikit-learn__scikit-learn-11574.md) · [`trajectories/v5-cheap-probes/sphinx-doc__sphinx-8284.md`](trajectories/v5-cheap-probes/sphinx-doc__sphinx-8284.md) | `⛔ Verification failed (attempt 1)`: the verifier names the evidence item whose quote is not in the test patch, the exact feedback text sent to the judge, the judge's re-check with `Grep`, and the corrected verdict that passes. These are the only two corrective turns in 300 evaluation runs — the announcement that quotes will be checked did the work (§5, Iteration 3). | @@ -453,7 +453,7 @@ matched to the runs that used it. ## 4. Results -### Headline comparison (30 cases, same cases and same deciding model for both) +### Headline comparison (30 cases, development set — same cases and same deciding model for both) | Metric | Simple baseline (`baseline`) | Agent solution (`v3-verify`) | Change | |---|---|---|---| @@ -469,7 +469,7 @@ matched to the runs that used it. | Cost per task (USD, list price) | $0.14 | $0.95 | +$0.81 ▼ | | Challenging case (`astropy__astropy-12544`) | correct (us=2 fn=3 flag) | correct (us=2 fn=3 flag) | human: us=0 fn=3 flag | -### All systems on the same 30 cases +### All systems on the same 30 cases (development set) | Run | Decision acc. | κ | TPR / TNR | Missed / false alarms | Both axes | Bad evidence | Cost/task | Time/task | |---|---|---|---|---|---|---|---|---| @@ -501,8 +501,9 @@ lines of a flagged task. the axis. Humans: issue clear (0), tests unfair (3). Every configuration found the test-axis problem with the right evidence — the `mask_invalid` assertions quoted from the test patch against the issue's own `mask` wording — but the baseline and the default pipeline (`v3-verify`) also scored `underspecified = 2`, arguing that the issue leaves the -single-table and memmap behaviour open; only the Sonnet-probe configurations (`v5`, `v7`) put it at 1, on the human -side of the threshold. Right decision, one axis debatable — which is why the "both axes correct" metric exists. +single-table and memmap behaviour open. Four runs put it at 1, on the human side of the threshold (`v1`, `v2`, `v4` +and the first `v5` run); the baseline repeat, `v5-rerun`, `v6` and `v7` stayed at 2. Right decision, one axis +debatable — and unstable from run to run — which is why the "both axes correct" metric exists. --- diff --git a/REPRODUCE.md b/REPRODUCE.md index 48497a7..0fd92da 100644 --- a/REPRODUCE.md +++ b/REPRODUCE.md @@ -38,13 +38,13 @@ Two public inputs, both already committed under `data/`: `npm run data:annotations` re-downloads it from the pinned source commit and refuses a mismatching file. - `data/eval/instances.json` — the fixed evaluation set: 30 instances (seed `20260828`), each with its issue text, gold patch, test patch, FAIL_TO_PASS list and the human labels. `data/eval/calibration.json` holds the annotator - notes for the *other* 1,600+ instances, grouped by repository (used by the final variant; contains no eval instance). + notes for the *other* 1,600+ instances, grouped by repository (used only by the `v4`–`v6` experimental configurations, not by the `v3-verify` default; contains no eval instance). To rebuild them from scratch (needs the 12 MB SWE-bench parquet, not committed): ```bash -curl -L -o data/raw/swebench_test.parquet \ - "https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/main/data/test-00000-of-00001.parquet" +curl --fail -L -o data/raw/swebench_test.parquet \ + "https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6/data/test-00000-of-00001.parquet" npm run data:eval-set # -> data/eval/instances.json (prints the 30 instances and strata) npm run data:calibration # -> data/eval/calibration.json ``` @@ -116,8 +116,8 @@ of them with `scripts/render-trajectories.sh `. ## 5. Evaluate -`npm run score` needs only the committed results. `evidence-audit`, `code-check` and `finalize-report.py` re-read the -repositories at the base commit, so run `npm run data:workspaces` first (they refuse to run otherwise). +`npm run score` needs only the committed results. `report.ts`, `evidence-audit`, `code-check` and `finalize-report.py` +re-read the repositories at the base commit, so run `npm run data:workspaces` first (they refuse to run otherwise). ```bash npm run score -- baseline baseline-rerun v1-context v2-specialists v3-verify v4-calibrated v5-cheap-probes v5-rerun v6-target-aware v7-sonnet-nocal diff --git a/examples/astropy__astropy-12544/human-labels.md b/examples/astropy__astropy-12544/human-labels.md index 54c5416..397b1b2 100644 --- a/examples/astropy__astropy-12544/human-labels.md +++ b/examples/astropy__astropy-12544/human-labels.md @@ -6,6 +6,7 @@ Source: data/raw/ensembled_annotations_public.csv (provenance in data/raw/SOURCE |---|---| | underspecified | 0 | | false_negative | 3 | +| difficulty | 1-4 hours | | filter_out | true | ## Annotator note — issue diff --git a/skills/fairtask-eval/SKILL.md b/skills/fairtask-eval/SKILL.md index 8d35de0..1ea11e9 100644 --- a/skills/fairtask-eval/SKILL.md +++ b/skills/fairtask-eval/SKILL.md @@ -21,9 +21,9 @@ except `data` and `annotations`, which download pinned public files. None calls | Request mentions | Run from the repository root | |---|---| - | the evaluation set, the thirty cases, strata, which tasks | `npm run data:eval-set` (needs `data/raw/swebench_test.parquet`; if missing, run `curl -L -o data/raw/swebench_test.parquet https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/main/data/test-00000-of-00001.parquet` first and say you did) | + | the evaluation set, the thirty cases, strata, which tasks | `npm run data:eval-set` (needs `data/raw/swebench_test.parquet`; if missing, run `curl -L -o data/raw/swebench_test.parquet https://huggingface.co/datasets/princeton-nlp/SWE-bench/resolve/e48e2bd1e9fecd5bbd641e9414ac59da9f2e69f6/data/test-00000-of-00001.parquet` first and say you did) | | score, accuracy, kappa, recall, a run id, compare runs | `npm run score -- ` (default: `baseline v3-verify`; add `--detail` for per-instance rows) | - | the report, the headline table, baseline versus final | `node src/report.ts --baseline baseline --final v3-verify --runs ` — the run ids are the directory names under `results/`; the default (`v3-verify`) has no repeat run, so no `--final-repeat` (the script refuses a repeat of a different configuration). Prints aligned tables; `--markdown` gives the README's pipe tables | + | the report, the headline table, baseline versus final | `node src/report.ts --baseline baseline --final v3-verify --runs ` — the run ids are the directory names under `results/`; the default (`v3-verify`) has no repeat run, so no `--final-repeat` (the script refuses a repeat of a different configuration). Needs cloned workspaces for the bad-evidence rows — run `npm run data:workspaces` (~20 s) first if `workspaces/` is missing. Prints aligned tables; `--markdown` gives the README's pipe tables | | evidence audit, bad evidence, fabricated quotes | `npm run audit -- ` (needs cloned workspaces: `npm run data:workspaces` first, ~20 s) | | code check, novel identifiers, pre-check | `npm run code-check` (needs workspaces) | | provenance, annotations, checksum, data source | `npm run data:annotations -- --check` | @@ -47,7 +47,7 @@ except `data` and `annotations`, which download pinned public files. None calls - Committed run ids are complete: `npm run run -- --run-id baseline` does nothing. Reproduction runs use fresh ids (`baseline-repro`), cost money and need `claude login` or `ANTHROPIC_API_KEY`; this skill does not start them. Point the user at `REPRODUCE.md` instead. -- `audit`, `code-check` and `finalize-report.py` refuse to run without workspaces at the right commits; that is - intended. Run `npm run data:workspaces`, do not work around it. +- `report.ts`, `audit`, `code-check` and `finalize-report.py` refuse to run without workspaces at the right commits; + that is intended. Run `npm run data:workspaces`, do not work around it. - `data:eval-set` prints the table; it also rewrites `data/eval/instances.json` deterministically. In a clean checkout `git status` stays clean afterwards — if it does not, say so, because that is a reproducibility failure. diff --git a/src/lib/audit.ts b/src/lib/audit.ts index ee28886..9de49ce 100644 --- a/src/lib/audit.ts +++ b/src/lib/audit.ts @@ -28,7 +28,7 @@ export function requireWorkspaces(ids: Iterable): void { if (git("status", "--porcelain", "--untracked-files=all") !== "") problems.push(`${id}: working tree has modified or untracked files`); } catch (e) { problems.push(`${id}: ${(e as Error).message.split("\n")[0]}`); } } - if (problems.length) throw new Error(`${problems.length} workspace(s) cannot be used for verification:\n ${problems.slice(0, 5).join("\n ")}${problems.length > 5 ? "\n …" : ""}\nRun \`npm run data:workspaces\` to (re)clone them at the base commit.`); + if (problems.length) throw new Error(`${problems.length} workspace(s) cannot be used for verification (fix: npm run data:workspaces):\n ${problems.slice(0, 5).join("\n ")}${problems.length > 5 ? "\n …" : ""}\nRun \`npm run data:workspaces\` to (re)clone them at the base commit.`); } export function auditRun(runId: string): Audit { diff --git a/src/report.test.ts b/src/report.test.ts new file mode 100644 index 0000000..d9f1b73 --- /dev/null +++ b/src/report.test.ts @@ -0,0 +1,10 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; + +test("report refuses a --final-repeat of a different configuration", () => { + assert.throws( + () => execFileSync(process.execPath, ["src/report.ts", "--baseline", "baseline", "--final", "v3-verify", "--final-repeat", "v5-rerun"], { encoding: "utf8", stdio: "pipe" }), + (e: unknown) => String((e as { stderr?: string }).stderr).includes("is not a repeat of") && String((e as { stderr?: string }).stderr).includes("variant"), + ); +}); diff --git a/src/report.ts b/src/report.ts index b49673a..0962551 100644 --- a/src/report.ts +++ b/src/report.ts @@ -23,7 +23,13 @@ const delta = (a: number, b: number, fmt: (x: number) => string, betterHigh = tr const base = load(args.get("baseline") ?? "baseline"); const fin = load(args.get("final") ?? "v3-verify"); const rep = args.get("final-repeat") ? load(args.get("final-repeat")!) : undefined; -if (rep && rep.variant !== fin.variant) throw new Error(`--final-repeat ${rep.run_id} is a run of variant "${rep.variant}", not of the final configuration "${fin.variant}" (${fin.run_id}); a repeat must be the same configuration run again`); +if (rep) { + const mismatch: string[] = []; + if (rep.variant !== fin.variant) mismatch.push(`variant "${rep.variant}" vs "${fin.variant}"`); + if (rep.model !== fin.model) mismatch.push(`model "${rep.model}" vs "${fin.model}"`); + if (rep.n !== fin.n || rep.n_flag !== fin.n_flag || rep.n_usable !== fin.n_usable) mismatch.push(`case universe ${rep.n} (${rep.n_flag}/${rep.n_usable}) vs ${fin.n} (${fin.n_flag}/${fin.n_usable})`); + if (mismatch.length) throw new Error(`--final-repeat ${rep.run_id} is not a repeat of ${fin.run_id}: ${mismatch.join("; ")}. A repeat must be the same configuration on the same cases run again.`); +} const format: TableFormat = args.get("markdown") ? "markdown" : "table"; /** Show "first / repeat" when a repeat run of the final configuration exists. */ const both = (f: (s: Summary) => string) => (rep ? `${f(fin)} · ${f(rep)}` : f(fin)); @@ -44,12 +50,12 @@ const headline: string[][] = [ [`Cost per task (USD, list price)`, `${usd(base.mean_cost_usd)}`, `${usd(fin.mean_cost_usd)}`, `${delta(base.mean_cost_usd, fin.mean_cost_usd, usd, false)}`], [`Challenging case (\`${fin.challenging?.instance_id}\`)`, `${base.challenging?.correct ? "correct" : "wrong"} (${base.challenging?.pred})`, `${fin.challenging?.correct ? "correct" : "wrong"} (${fin.challenging?.pred})`, `human: ${fin.challenging?.human}`], ]; -console.log(renderTitle(`Headline comparison (${base.n} cases, same cases and same deciding model for both${rep ? "; final shown as first run · repeat run" : ""})`, format)); +console.log(renderTitle(`Headline comparison (${base.n} cases, development set — same cases and same deciding model for both${rep ? "; final shown as first run · repeat run" : ""})`, format)); console.log(renderTable(["Metric", `Simple baseline (\`${base.run_id}\`)`, `Agent solution (\`${fin.run_id}\`)`, "Change"], headline, format)); const runs = (args.get("runs") ?? "").split(",").filter(Boolean).map(load); if (runs.length) { - console.log("\n" + renderTitle(`All systems on the same ${base.n} cases`, format)); + console.log("\n" + renderTitle(`All systems on the same ${base.n} cases (development set)`, format)); console.log(renderTable( ["Run", "Decision acc.", "κ", "TPR / TNR", "Missed / false alarms", "Both axes", "Bad evidence", "Cost/task", "Time/task"], runs.map((r) => [`\`${r.run_id}\``, pct(r.decision_accuracy), r.kappa.toFixed(2), `${pct(r.tpr)} / ${pct(r.tnr)}`, `${r.missed_problems} / ${r.false_alarms}`, pct(r.both_axes_correct), badRate(r.run_id), usd(r.mean_cost_usd), `${Math.round(r.mean_duration_s)} s`]), diff --git a/src/show.ts b/src/show.ts index 7c219ea..cdf4a43 100644 --- a/src/show.ts +++ b/src/show.ts @@ -28,5 +28,5 @@ const links = [ writeFileSync(`${dir}/issue.md`, `# ${id} — the issue text (all a solver sees)\n\n${links}\n\n---\n\n${inst.problem_statement.trim()}\n`); writeFileSync(`${dir}/test.patch`, inst.test_patch); writeFileSync(`${dir}/gold.patch`, inst.patch); -writeFileSync(`${dir}/human-labels.md`, `# ${id} — human labels (OpenAI, 2024; max over three annotators)\n\nSource: data/raw/ensembled_annotations_public.csv (provenance in data/raw/SOURCES.md)${pr ? `; task: https://github.com/${inst.repo}/pull/${pr}` : ""}\n\n| axis | score |\n|---|---|\n| underspecified | ${inst.human.underspecified} |\n| false_negative | ${inst.human.false_negative} |\n| filter_out | ${inst.human.filter_out} |\n\n## Annotator note — issue\n\n${inst.human.underspecified_notes.trim()}\n\n## Annotator note — tests\n\n${inst.human.false_negative_notes.trim()}\n\n## Graded tests (FAIL_TO_PASS)\n\n${inst.FAIL_TO_PASS.map((t) => `- \`${t}\``).join("\n")}\n`); +writeFileSync(`${dir}/human-labels.md`, `# ${id} — human labels (OpenAI, 2024; max over three annotators)\n\nSource: data/raw/ensembled_annotations_public.csv (provenance in data/raw/SOURCES.md)${pr ? `; task: https://github.com/${inst.repo}/pull/${pr}` : ""}\n\n| axis | score |\n|---|---|\n| underspecified | ${inst.human.underspecified} |\n| false_negative | ${inst.human.false_negative} |\n| difficulty | ${inst.human.difficulty} |\n| filter_out | ${inst.human.filter_out} |\n\n## Annotator note — issue\n\n${inst.human.underspecified_notes.trim()}\n\n## Annotator note — tests\n\n${inst.human.false_negative_notes.trim()}\n\n## Graded tests (FAIL_TO_PASS)\n\n${inst.FAIL_TO_PASS.map((t) => `- \`${t}\``).join("\n")}\n`); console.log(`wrote ${dir}/{issue.md,test.patch,gold.patch,human-labels.md}`); diff --git a/trajectories/v3-verify/astropy__astropy-12544.md b/trajectories/v3-verify/astropy__astropy-12544.md index 2089beb..8abdeb2 100644 --- a/trajectories/v3-verify/astropy__astropy-12544.md +++ b/trajectories/v3-verify/astropy__astropy-12544.md @@ -117,29 +117,29 @@ Repository: astropy/astropy @ 3a0cd2d8cd7b459cdc1e1b97a14f3040ccc1fffc (version ## Issue text (this is ALL the solver will see) Can Table masking be turned off? - - - - - - -### Description - - - - -As of Astropy 5, when `astropy.table.Table.read()` encounters values such as `NaN`, it automatically creates a `MaskedColumn` and the whole table becomes a `MaskedTable`. While this might be useful for individual end-users, it is very inconvenient for intermediate data in pipelines. - -Here's the scenario: data are being passed via files and `Table.read()`. A downstream function needs to replace `NaN` with valid values. Previously those values could be easily identified (*e.g.* `np.isnan()` and replaced. However, now additional work is need to look "underneath" the mask, extracting the actual values, replacing them, and then possibly creating a new, unmasked column, or even an entirely new table. - + + + + + + +### Description + + + + +As of Astropy 5, when `astropy.table.Table.read()` encounters values such as `NaN`, it automatically creates a `MaskedColumn` and the whole table becomes a `MaskedTable`. While this might be useful for individual end-users, it is very inconvenient for intermediate data in pipelines. + +Here's the scenario: data are being passed via files and `Table.read()`. A downstream function needs to replace `NaN` with valid values. Previously those values could be easily identified (*e.g.* `np.isnan()` and replaced. However, now additional work is need to look "underneath" the mask, extracting the actual values, replacing them, and then possibly creating a new, unmasked column, or even an entirely new table. + Ideally, a keyword like `Table.read(filename, ..., mask=False)` would disable this behavior, for people who don't need this masking. @@ -587,7 +587,6 @@ No matches found { "underspecified": 2, "underspecified_rationale": "The issue's high-level ask (a keyword on Table.read to disable automatic masking) is legible, and the NaN-masking site is discoverable (astropy/io/fits/connect.py:L249-L252 is the only such site in astropy/io). But three test-determinative decisions are left blank: (1) the keyword name — the issue proposes `mask=False`, the gold API is `mask_invalid`, and grep shows no pre-existing `mask_invalid` convention anywhere in the package to guide the solver; (2) whether the flag should also suppress empty-string masking (the issue only ever names NaN) while leaving the integer `col.null` branch masked (gold leaves it unguarded) — the issue does not resolve which masking paths \"this behavior\" covers; (3) the memmap interaction is never mentioned at all, yet gold makes `memmap=True` silently imply no masking. The issue is also phrased generically about `Table.read()` while the fix is FITS-only.", - "false_negative": 3, "false_negative_rationale": "The FAIL_TO_PASS tests pin three gold-patch-only choices, at least two of which defeat reasonable solutions. (a) `tab = Table.read(filename, mask_invalid=False)` requires the exact kwarg name `mask_invalid`, while the issue literally proposes `mask=False`; a faithful implementation raises TypeError and fails both tests. (b) `test_mask_nans_on_read` adds `tab = Table.read(filename, memmap=True); assert tab.mask is None` — behaviour never requested in the issue. Verified at base commit that connect.py:L217-227 uses memmap only for `fits_open` and does not even forward it to the recursive `read_table_fits` call, so masking is currently applied regardless of memmap; a solver who adds only the kwarg fails this assertion. (c) `test_mask_str_on_read` requires the flag to also gate `elif issubclass(coltype, np.character): mask = col.array == b''`, though the issue mentions only NaN and gold itself inconsistently leaves the `col.null` integer branch unguarded.", "evidence": [ { @@ -640,7 +639,8 @@ No matches found "quote": " if memmap:\n # using memmap is not compatible with masking invalid value by\n # default so we deactivate the masking\n mask_invalid = False" } ], - "decision": "flag", + "decision": "flag", + "false_negative": 3, "confidence": 4 } ```