Fix Delta JSON round-trip crash when orjson is not installed#613
Open
chuenchen309 wants to merge 1 commit into
Open
Fix Delta JSON round-trip crash when orjson is not installed#613chuenchen309 wants to merge 1 commit into
chuenchen309 wants to merge 1 commit into
Conversation
Delta serializes each iterable Opcode NamedTuple, then rebuilds it in the JSON deserializer with Opcode(**op_code), which assumes a mapping. That only holds under orjson: its default callback runs JSON_CONVERTOR and emits Opcodes as dicts. The stdlib json fallback (used on any install without the optional orjson extra) encodes NamedTuples natively as positional arrays, so Opcode(**op_code) raises TypeError and the documented round-trip Delta(dump, deserializer=json_loads) crashes for any list diff that goes through difflib opcodes (reorders / duplicates). Accept both shapes on deserialization: mapping -> Opcode(**op_code), array -> Opcode(*op_code) (fields are in NamedTuple order). Added a regression test that forces the builtin-json path via force_use_builtin_json=True so it guards the array shape even where orjson is installed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Delta's documented JSON round-trip crashes on a default install (one without the optionalorjsonextra) for any list diff that goes through difflib opcodes — i.e. reorders or duplicates:This is the repo's own contract:
tests/test_delta.py::TestDeltaCompareFunc::test_list_of_alphabet_and_its_deltaalready asserts exactly this round-trip. It passes in CI only because the dev/test environment installsorjson. On a plainpip install deepdiffthat test fails, becauseorjsonis optional ([project.dependencies]is justorderly-set;orjsonlives in theoptimize/dev/testextras).Root cause
A serialize/deserialize asymmetry around the
OpcodeNamedTuple:orjson, the encoder'sdefaultcallback runsJSON_CONVERTOR[tuple](serialization.py), turning eachOpcodeinto a mapping via_asdict(). The stdlibjsonfallback (serialization.py,json_dumpswhenorjsonisNone) serializes anytuple/NamedTuple subclass natively as a JSON array and never invokesdefault, so anOpcodebecomes a positional list like["equal", 1, 3, 0, 2, null, null].delta.pyunconditionally doesOpcode(**op_code), assuming a mapping. On anorjson-less install the opcode is a list, so**listraisesTypeError.A serialize-side fix alone can't work: stdlib
jsonnever callsdefaultfor tuple subclasses, so the robust fix is on deserialization.The fix
Accept both encodings when rebuilding opcodes — mapping →
Opcode(**op_code), array →Opcode(*op_code)(the array is in NamedTuple field order, so positional reconstruction is exact).Verification (re-runnable from the diff)
orjson, before the fix:test_list_of_alphabet_and_its_deltafails atdelta.pywith theTypeErrorabove. After the fix it passes, and the fulltests/test_delta.pygoes from 25 failures to 1.orjson(the CI condition):tests/test_delta.pyis 129 passed, 0 failed — the mapping path is untouched, so no regression.test_delta_iterable_opcodes_json_roundtrip_builtin_json, which forces the stdlib-json path viaforce_use_builtin_json=True. It fails on the current code even withorjsoninstalled, so the round-trip is now guarded regardless of the install, not just whenorjsonhappens to be absent.One thing left out of scope
The remaining no-
orjsonfailure istest_delta_repr: it assertsorjson's compact separators ({"iterable_item_added":{...}}), while stdlibjsonemits spaces ({"iterable_item_added": {...}}). That's a separate repr/formatting assumption, unrelated to this crash — I left it alone to keep the diff focused, but happy to address it here or in a follow-up if you'd prefer.Base branch is
devperAGENTS.md.This PR was authored by an AI coding agent (Claude Code) running on this account: the AI found the bug, ran the repro, wrote the tests, and wrote this description. The human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.