fix(dedup): let the defining file win an ID collision, and warn only about real loss#1852
Open
bchan84x wants to merge 1 commit into
Open
Conversation
…about real loss Node IDs are <source-path>_<entity>, so a doc that merely references an entity mints the entity's own ID and collides with the entity's node by construction. The pre-dedup pass kept whichever arrived first, so chunk order decided whether an entity kept its own attributes or a passing cross-reference's, and the warning that fired (Graphify-Labs#1504) told the user a same-name-different-directory clash had lost their data — while the one drop that really is lossy, two labels for one ID from the same file, was silent because the warning was gated on source_file differing. The survivor is now the node whose source_file is the file its ID encodes (any trailing slice of the path, so absolute, repo-relative, and pre-Graphify-Labs#1504 bare-stem IDs all resolve), falling back to first-seen. Reporting follows what the drop actually costs: a cross-reference folding into the node it references loses nothing (edges are keyed by ID and rewire to the survivor) and is silent; a same-file relabel notes the discarded label; two files that both encode the ID are distinct entities and still WARNING, now stating the ID-scheme cause rather than a filename clash. Refs Graphify-Labs#1851
|
A scan of this PR to explain it - using a tool I'm working on. Hope that it helps: https://www.lenzon.ai/viewer/cmrju75nq0011odzhizb1hp1p?voice=google-chirp3 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the ID-collision handling described in #1851.
The problem
A node ID is
<slug(source_path)>_<slug(entity)>, so a doc that merely references an entity mints that entity's own ID and collides with the entity's defining node by construction. The pre-dedup pass indeduplicate_entitieshandled that badly:seen_idskept whichever node arrived first, so chunk order decided whether an entity kept its own attributes or a passing cross-reference's thinner label.existing_sf != new_sf, so two same-ID nodes from the same file — the extractor emitting two different labels — were dropped with no output at all.The change
graphify/dedup.pyonly.source_fileis the file the ID encodes; otherwise first-seen, as before._id_prefixes()accepts any trailing slice of the path, so absolute paths, repo-relative paths, and pre-Silent data loss: doc files with the same name in different directories produce colliding node IDs across extraction chunks #1504 bare-filename-stem IDs all resolve.[graphify] note:naming the discarded label (this was the previously-silent case).extractper-subfolder +merge-graphsadvice. This is the Silent data loss: doc files with the same name in different directories produce colliding node IDs across extraction chunks #1504 case, and its existing test still passes unchanged.Tests
New tests in
tests/test_dedup.py: definer-wins in both insertion orders, reference collisions stay silent and keep their edges, absolutesource_filepaths still resolve, same-file relabels emit the note, plus a unit test for_defines_id. Both existing collision tests (test_cross_chunk_id_collision_emits_warning,test_same_id_same_source_file_no_warning) are untouched and green.pytest tests/test_dedup.py→ 40 passed.ruff checkclean.Notes
test_falkordb_integration.py(no FalkorDB module in my local Redis) andtest_skillgen.py(git showof baseline blobs, which a--depth 1clone lacks; CI usesfetch-depth: 0).Refs #1851