Skip to content

FEAT: Add remove_seeds_from_memory_async to memory - #1

Open
blahdeblahde wants to merge 561 commits into
mainfrom
blahdeblahde-remove-seeds-from-memory-api
Open

FEAT: Add remove_seeds_from_memory_async to memory#1
blahdeblahde wants to merge 561 commits into
mainfrom
blahdeblahde-remove-seeds-from-memory-api

Conversation

@blahdeblahde

@blahdeblahde blahdeblahde commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Description

PyRIT memory can add seed prompt datasets (via add_seed_datasets_to_memory_async / add_seeds_to_memory_async) and query them with get_seeds, but there is no counterpart for removing seeds. Today users have to drop into raw SQL, which is error-prone and differs across the supported backends (DuckDB, SQLite, Azure SQL).

This PR adds remove_seeds_from_memory_async, which accepts the same filter parameters as get_seeds (dataset_name, dataset_name_pattern, added_by, harm_categories, authors, groups, source, value_sha256, data_types, seed_type, parameters, metadata, prompt_group_ids) and returns the number of seeds removed.

Recommended workflow — preview, then delete with the same filters:

# See what matches before deleting
to_delete = memory.get_seeds(dataset_name="illegal")

# Remove them; returns the count deleted
removed = await memory.remove_seeds_from_memory_async(dataset_name="illegal")

Implementation

  • Extracted the filter-building logic out of get_seeds into a shared private helper (_build_seed_filter_conditions) so both query and removal build conditions from a single source and cannot drift.
  • remove_seeds_from_memory_async deletes matching rows in a single transaction using the SQLAlchemy ORM (rollback on error), so it behaves consistently across DuckDB, SQLite, and Azure SQL.

Safety

  • At least one filter must be provided; a no-filter call raises ValueError to prevent accidentally wiping the entire seed database.

Tests and Documentation

Teststests/unit/memory/memory_interface/test_interface_remove_seeds.py covers:

  • exact dataset_name match
  • dataset_name_pattern (SQL LIKE)
  • multi-filter narrowing (dataset_name + added_by)
  • no-filter call raises ValueError
  • zero-match returns 0 and deletes nothing
  • harm_categories (list field) and source filters

Documentation — added a "Removing Seeds from the Database" section to doc/code/memory/8_seed_database.py (and the synced .ipynb) demonstrating the preview-then-remove workflow.

hannahwestra25 and others added 30 commits June 18, 2026 21:06
…#2047)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… scenario E2E tests (microsoft#2048)

Co-authored-by: Behnam Ousat <behnamousat@microsoft.com>
…h 2 updates (microsoft#2052)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: hannahwestra25 <hannahwestra@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: hannahwestra25 <hannahwestra@users.noreply.github.com>
…t#2034)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…patch group across 1 directory (microsoft#2055)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
…ft#1902)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…7 to 0.15.19 (microsoft#2073)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…icrosoft#2076)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#2077)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
…t#2095)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
microsoft#2046)

Co-authored-by: Behnam Ousat <behnamousat@microsoft.com>
…icrosoft#2098)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
)

Co-authored-by: biefan <70761325+biefan@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
romanlutz and others added 14 commits August 4, 2026 19:06
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2c45e075-9e85-40b5-8352-fb0bcab7261c
…endabot alerts (microsoft#2323)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9009f005-342b-41a1-8513-74cbf282a9a4
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 647cdf27-3786-40a2-918c-b88f784fbc6d
…atch group across 1 directory (microsoft#2324)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1c85c49c-2d6c-4050-b3e3-ce085c97c5ec
…t and Responses targets) (microsoft#2166)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: varunj-msft <vjoginpalli@microsoft.com>
Copilot-Session: 00b73356-ae65-4d83-8220-ebad37cc5850
Copilot-Session: ebba1398-76c6-4659-97c5-a08afa209749
…ft#2330)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… to 0.16.1 (microsoft#2328)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…tes (microsoft#2332)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…crosoft#2314)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 13a6d940-c11c-4ba7-a4d4-bc6b9d75c757
…tes (microsoft#2326)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a14e9aad-db5a-46bc-9da4-4018369a8eab
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7a2feb74-e66a-427e-8b5c-c503b91d98d8
@blahdeblahde
blahdeblahde force-pushed the blahdeblahde-remove-seeds-from-memory-api branch from 7dbf60f to 1873066 Compare August 6, 2026 17:48
romanlutz and others added 7 commits August 7, 2026 01:15
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 88783b89-8853-43e4-8d60-483268ab0579
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 4f742b92-0427-45a9-924f-3fedeb8b915a
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 41c8a28e-3109-4b71-b97a-3d8ec8834136
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 48ec2fc7-3fd7-4188-a3ff-df3cc2a64790
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0d1f97bd-49a5-4a6d-b814-fecef1e2a8cb
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 92dc0f48-0fb4-4cbd-866c-ce5f41796439
Adds two memory APIs to remove seed prompts using the same filter
parameters as get_seeds (value, exact, value_sha256, dataset_name,
dataset_name_pattern, data_types, harm_categories, added_by, authors,
groups, source, seed_type, parameters, metadata, prompt_group_ids):

- remove_seeds_from_memory removes the individual seeds that match.
- remove_seed_groups_from_memory expands any match to its whole
  prompt_group_id so partial groups are never left behind.

Implementation:
- Extract filter-building logic from get_seeds into a shared
  _build_seed_filter_conditions helper so query and removal cannot drift.
- value matching is controlled by an exact bool flag (replacing the
  earlier value_exact string, per review). The remove methods default to
  exact=True (full-string equality) so a short or common value cannot
  over-delete; get_seeds keeps substring matching. Pass exact=False to
  opt into substring deletion. The intentional difference is documented.
- Require at least one filter (raises ValueError if none provided).
- Single transaction with rollback on SQLAlchemyError.
- Group removal selects matching prompt_group_ids with a server-side
  subquery (not a materialized IN(...) list) so a broad filter cannot
  exceed the backend bound-parameter limit; NULL group ids are excluded,
  so ungrouped seeds are skipped.
- Methods are synchronous, mirroring get_seeds: they do only DB work and
  never await I/O.

Only database records are removed; file-backed seeds (image_path,
audio_path, video_path) leave the serialized file on disk (documented).

Unit tests cover every individual filter, exact-by-default vs opt-in
substring matching, multi-filter narrowing, no-filter ValueError,
zero-match, rollback-on-error, and for groups: whole-group removal,
groups spanning datasets, non-matching groups preserved, and the
ungrouped (NULL group id) skip behavior.

Docs: add a "Removing Seeds from the Database" section to the seed
database notebook (8_seed_database.py/.ipynb) demonstrating the
preview-then-remove workflow, whole-group removal, exact-by-default
matching, the ungrouped-skip, and the file-backed caveat.

Resolves AB#5688

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f9e3cc10-5c77-4384-92d9-d27dad9baea2
@blahdeblahde
blahdeblahde force-pushed the blahdeblahde-remove-seeds-from-memory-api branch from 1873066 to c6060d9 Compare August 7, 2026 17:29
romanlutz and others added 7 commits August 7, 2026 20:03
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0d1f97bd-49a5-4a6d-b814-fecef1e2a8cb
Copilot-Session: 67c023ca-8b50-48de-88f1-0e2cc3025996
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 2b3468bc-0737-461b-bb82-18dad97d400e
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 058569a0-6218-4cf4-947e-67fa51001d85
…h 2 updates (microsoft#2355)

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…icrosoft#2261)

Co-authored-by: Roman Lutz <romanlutz13@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b84937f-52e6-42a6-9b35-ff544cfd16eb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.