TAB-976 Harden form fills against prompt context exfiltration#478
Open
srbiv wants to merge 7 commits into
Open
TAB-976 Harden form fills against prompt context exfiltration#478srbiv wants to merge 7 commits into
srbiv wants to merge 7 commits into
Conversation
This reverts commit 6def109.
lmorchard
approved these changes
May 26, 2026
Collaborator
There was a problem hiding this comment.
I think this is okay to land - though, it does break some form-filling stuff I was doing for automated survey testing. Not sure if we have any real customers relying on something similar.
The solution for the survey testing would be to use a separate companion agent to provide the survey answers in interactive mode. But that's probably a better design anyway, since that agent won't necessarily be exposed to prompt injection from the page itself as easily.
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a structural “action firewall” to harden browser automation against prompt-context exfiltration by gating agent-driven form fills and form submissions using DOM-derived field metadata and provenance tracking (rather than payload text matching).
Changes:
- Add
FieldMetadata/FormSubmissionContextbrowser APIs and implement them in Playwright + Extension browsers. - Enforce a security policy in
webActionToolsto block unauthorized agent fills and to gate submit-like actions when a form contains unauthorized agent-filled refs. - Add unit/regression tests for firewall behavior, provenance requirements, and snapshot/provenance lifetime interactions.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/extension/src/background/ExtensionBrowser.ts | Adds DOM introspection helpers for field metadata + form submission context. |
| packages/core/src/browser/ariaBrowser.ts | Extends the AriaBrowser contract with metadata/submission context types and methods. |
| packages/core/src/browser/playwrightBrowser.ts | Implements the new browser APIs using Playwright locator.evaluate. |
| packages/core/src/security/actionFirewall.ts | Adds fill/submission assessment logic and constants. |
| packages/core/src/tools/webActionTools.ts | Integrates firewall checks + provenance tracking into tools (fill/click/enter). |
| packages/core/src/tools/interactiveTools.ts | Simplifies ApprovedRefs implementation (now a Set). |
| packages/core/src/webAgent.ts | Wires provenance sets into tool context; adjusts snapshot refresh policy for fill. |
| packages/core/src/core.ts | Re-exports the new browser types from the public core entrypoint. |
| packages/core/test/security/actionFirewall.test.ts | Adds unit coverage for the firewall’s allow/block decisions. |
| packages/core/test/tools/webActionTools.test.ts | Adds regression tests for blocked fills, submit gating, and provenance requirements. |
| packages/core/test/webAgent.test.ts | Adds regression coverage for snapshot stability after fill. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if (error instanceof BrowserException) { | ||
| return failedActionResult(action, error.message, context, ref); | ||
| } | ||
| throw error; |
| if (error instanceof BrowserException) { | ||
| return failedActionResult(PageAction.Fill, error.message, context, ref); | ||
| } | ||
| throw error; |
Comment on lines
+899
to
+971
| return locator.evaluate( | ||
| (element, { submitterRef, trigger }): FormSubmissionContext | null => { | ||
| const el = element as HTMLElement; | ||
| if (!canSubmitForm(el, trigger)) return null; | ||
|
|
||
| const form = getSubmissionForm(el); | ||
| if (!form) return null; | ||
|
|
||
| const fields = Array.from(form.elements) | ||
| .filter( | ||
| (field): field is HTMLInputElement | HTMLTextAreaElement | HTMLSelectElement => | ||
| field instanceof HTMLInputElement || | ||
| field instanceof HTMLTextAreaElement || | ||
| field instanceof HTMLSelectElement, | ||
| ) | ||
| .filter((field) => !field.disabled) | ||
| .map((field) => ({ | ||
| ref: field.getAttribute("data-pilo-ref"), | ||
| name: field.name || null, | ||
| tagName: field.tagName.toLowerCase(), | ||
| inputType: field instanceof HTMLInputElement ? field.type.toLowerCase() : null, | ||
| autocomplete: "autocomplete" in field ? field.autocomplete || null : null, | ||
| })); | ||
|
|
||
| return { | ||
| submitterRef, | ||
| formId: form.id || null, | ||
| actionUrl: form.action || null, | ||
| method: form.method?.toLowerCase() || null, | ||
| fields, | ||
| }; | ||
|
|
||
| function getSubmissionForm(node: HTMLElement): HTMLFormElement | null { | ||
| if ( | ||
| node instanceof HTMLButtonElement || | ||
| node instanceof HTMLInputElement || | ||
| node instanceof HTMLTextAreaElement || | ||
| node instanceof HTMLSelectElement | ||
| ) { | ||
| return node.form; | ||
| } | ||
| return node.closest("form"); | ||
| } | ||
|
|
||
| function canSubmitForm(node: HTMLElement, submitTrigger: FormSubmissionTrigger): boolean { | ||
| if (submitTrigger === "click") { | ||
| if (node instanceof HTMLButtonElement) { | ||
| return node.type === "submit"; | ||
| } | ||
| if (node instanceof HTMLInputElement) { | ||
| return node.type === "submit" || node.type === "image"; | ||
| } | ||
| return false; | ||
| } | ||
|
|
||
| if (node instanceof HTMLTextAreaElement || node instanceof HTMLSelectElement) | ||
| return false; | ||
| if (!(node instanceof HTMLInputElement)) return false; | ||
| return ![ | ||
| "button", | ||
| "checkbox", | ||
| "color", | ||
| "file", | ||
| "hidden", | ||
| "radio", | ||
| "range", | ||
| "reset", | ||
| "submit", | ||
| ].includes(node.type); | ||
| } | ||
| }, | ||
| { submitterRef: ref, trigger }, | ||
| ); |
Comment on lines
+316
to
+321
| func: (elementRef: string) => { | ||
| const element = document.querySelector(`[data-pilo-ref="${elementRef}"]`); | ||
| if (!(element instanceof HTMLElement)) { | ||
| throw new Error(`Element with ref ${elementRef} not found in DOM`); | ||
| } | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test Plan