feat: eval the Row Level Security guide in supabase.com/docs - #168
Open
czenko wants to merge 14 commits into
Open
feat: eval the Row Level Security guide in supabase.com/docs#168czenko wants to merge 14 commits into
czenko wants to merge 14 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
barryroodt
requested changes
Aug 7, 2026
barryroodt
left a comment
barryroodt
left a comment
Contributor
There was a problem hiding this comment.
Thanks, this is a useful eval and the core shape looks good. pnpm format:check and pnpm typecheck pass. I found several false-pass paths that need closing before merge, and I’ve left the exact cases inline.
The PR also needs refreshed eval results. The handwritten solution proves one passing path. Real agent results show whether the eval discriminates.
pnpm check failed in one results-export test; attribution is unresolved.
czenko
marked this pull request as draft
August 7, 2026 19:00
czenko
changed the title
czenko
added a commit
that referenced
this pull request
Aug 7, 2026
The RLS eval previously copied the guide's "AI Prompt" panel verbatim into PROMPT.md, so the eleven rules that spell out the answer were the prompt and the scorer checked the agent against them. That measures spec-following, not the guide. Rewrite it so the guide is the subject under test. The prompt is now a vibe coder's request naming two separate apps plus the guide's url, with RLS, policy, security, role, tenant and test stripped from the vocabulary. The eleven rules became the grading criteria. The seed carries the app domain and says nothing about who may read or write what. Seed covers the three access shapes most apps need: owner-private todos, membership-shared lists, and a public weather feed no client writes to. list_members forces a security definer helper, since a policy that joins it is the join to avoid and a policy that queries itself raises 42P17. Move both docs-prompt evals into a build-docs series, named for the documentation under test rather than the interface: build-cli-004-quickstart-nextjs -> build-docs-001-homepage-quickstart build-cli-005-quickstart-rls -> build-docs-002-rls-guide Address the review on #168: - Allowed writes now prove state changed, including delete, and denied writes prove the row survived. - The helper check resolves the exact function the list_items policies call and requires security definer plus search_path pinned empty. - pgTAP coverage is bound to the seeded tables before the judge runs. - Every rule the seed implies has a live probe, plus a 42P17 probe. - The index check requires btree and rejects partial indexes. - The scorer is split into EVAL.ts, catalog.ts, access.ts and tests.ts. Narrow apps/framework/tsconfig.json from evals/**/EVAL.ts to evals/*/*.ts so sibling scorer modules are typechecked, while still excluding local/, remote/ and tests/ fixtures. Drop two grant checks that could only fail on over-revoke, since Supabase grants all four privileges to anon and authenticated by default and the live probes cover the outcome. The two remaining grant checks measure narrowing those defaults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
czenko
added
the
run-evals-changed
czenko
changed the title
The RLS eval previously copied the guide's "AI Prompt" panel verbatim into PROMPT.md, so the eleven rules that spell out the answer were the prompt and the scorer checked the agent against them. That measures spec-following, not the guide. Rewrite it so the guide is the subject under test. The prompt is now a vibe coder's request naming two separate apps plus the guide's url, with RLS, policy, security, role, tenant and test stripped from the vocabulary. The eleven rules became the grading criteria. The seed carries the app domain and says nothing about who may read or write what. Seed covers the three access shapes most apps need: owner-private todos, membership-shared lists, and a public weather feed no client writes to. list_members forces a security definer helper, since a policy that joins it is the join to avoid and a policy that queries itself raises 42P17. Move both docs-prompt evals into a build-docs series, named for the documentation under test rather than the interface: build-cli-004-quickstart-nextjs -> build-docs-001-homepage-quickstart build-cli-005-quickstart-rls -> build-docs-002-rls-guide Address the review on #168: - Allowed writes now prove state changed, including delete, and denied writes prove the row survived. - The helper check resolves the exact function the list_items policies call and requires security definer plus search_path pinned empty. - pgTAP coverage is bound to the seeded tables before the judge runs. - Every rule the seed implies has a live probe, plus a 42P17 probe. - The index check requires btree and rejects partial indexes. - The scorer is split into EVAL.ts, catalog.ts, access.ts and tests.ts. Narrow apps/framework/tsconfig.json from evals/**/EVAL.ts to evals/*/*.ts so sibling scorer modules are typechecked, while still excluding local/, remote/ and tests/ fixtures. Drop two grant checks that could only fail on over-revoke, since Supabase grants all four privileges to anon and authenticated by default and the live probes cover the outcome. The two remaining grant checks measure narrowing those defaults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Narrowing the include to evals/*/*.ts was unnecessary. TypeScript typechecks transitively imported files, so EVAL.ts importing ./catalog.js, ./access.js and ./tests.js already pulls the sibling scorer modules into the program. Verified by planting a type error in access.ts and running tsc with the original evals/**/EVAL.ts include. The error is reported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The check scanned tool call requests for the guide url, which misses a search_docs hit. There the url comes back in the result, not the request, so an agent that reached the guide through the docs MCP server was scored as never having read it. Seen in CI: one run web_fetched the url and passed, the next used search_docs, pulled 72,200 characters with the guide as the top hit, and failed. Reuse buildDocsResult, which the harness already uses to resolve pages across search_docs, web_fetch, web_search and shell_fetch, and require the call to have actually retrieved page content. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
czenko
changed the title
czenko
marked this pull request as ready for review
August 7, 2026 22:43
persona is not in evalFrontmatterSchema. The preprocessor builds a fixed object from known keys, so it was silently dropped and surfaced nowhere. It also had no precedent: across 44 evals it appeared once, in this file. motivation is not in the schema either, but 29 evals use it, so that one is convention. Fold the pointer into motivation. The persona itself is already documented in the eval README, which is where it belongs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… for a helper The check required a security definer helper in a private schema and failed anything else. Membership has more than one safe implementation, and the guide itself teaches one that needs no function at all, keeping membership in the JWT. An agent following that example wrote secure policies, passed every access probe, and still failed this check. That is checking process, not end state. Invert it. Whether membership actually holds is already proven by the access probes. This now fails only the shapes that are genuinely dangerous: - a security definer function in public or graphql_public, which PostgREST exposes as an RPC callable by anon - a security definer function anywhere that does not pin search_path No such function is a pass, because the agent solved it another way and the probes covered the outcome. Widen the function loader to keep the exposed schemas in scope. It previously filtered public out as pre-existing, which is exactly where the dangerous case lands. A stock stack has no security definer functions in either exposed schema, so anything found there is the agent's. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Audited every check against the guide's actual text. Two tested beyond it. Remove the force-row-level-security check. The guide never mentions forcing RLS. Its Bypassing section covers service keys and the bypassrls role privilege only, so an agent had no way to learn this from the guide. Remove the search_path condition on security definer functions. The guide cautions only that they must not live in an exposed schema, and its own private.has_good_role() example does not pin search_path. Pinning is good practice, but testing it measures the model's background knowledge rather than the guide, so it belongs in a doc recommendation instead. What remains traces to the guide: enabling RLS, specifying roles, per-operation policies, using plus with check on updates, granting only what a role needs, indexing policy columns, wrapping auth.uid() in a select, minimizing joins, and keeping security definer functions out of exposed schemas. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… gap The scoping rule is whether something is the guide's job, not whether the guide currently says it. A gap is a failure, because the question is whether the document does its job. Teaching a pattern includes teaching it safely. The guide tells a reader to create a security definer function and gives a copy-pasteable example that leaves search_path unpinned, which is exploitable: anyone able to create objects can shadow an unqualified name and change what the elevated function reads. An agent following the guide faithfully inherits that. Same shape as the pgTAP gap, so it is scored the same way. The force-row-level-security check stays removed. Protecting a table from its own owner is not what this guide is for, and the access probes already catch the breakage it would cause. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes DOCS-1259
Problem
We point agents at the Row Level Security guide. Nobody has measured whether reading it is enough to produce secure policies.
The previous version of this eval could not answer that. It copied the guide's AI prompt panel into
PROMPT.md, so the eleven rules that spell out the answer were the prompt itself.Solution
The guide is now the subject under test. The prompt is a normal feature request plus the guide's url. The eleven rules became the grading criteria.
The prompt is written as a vibe coder. It never says RLS, policy, security, role, tenant or test. An agent that only writes secure policies when the prompt says "security" has not been served by the guide.
The seed is two separate apps, so one access pattern cannot be applied to everything.
todos.lists,list_members,list_items.weather_stations,weather_readings.using (true)is correct on the weather feed and catastrophic on todos.list_membersforces asecurity definerhelper, because a policy that joins it is a join inside a policy, and a policy that queries itself raises42P17.Nothing in the migration says who may read or write what.
The prompt does not ask for tests. Whether the agent arrives at pgTAP is part of the measurement.
Rename
Both docs-prompt evals move into a series named for the documentation under test.
build-cli-004-quickstart-nextjsbuild-docs-001-homepage-quickstartbuild-cli-005-quickstart-rlsbuild-docs-002-rls-guideRenaming breaks the gh-pages history series for 004, which is keyed by eval id.
Testing
pnpm typecheckandpnpm format:checkare clean.pnpm eval:dryconfirms discovery and frontmatter for both ids. Results are refreshing in CI.The scorer has not been run end to end. Local runs were blocked by another project's stack holding the Supabase ports.