Skip to content

feat: add the sync skeleton, Str.ascii and Str.squish, and AppLifecycle.states - #204

Merged
anilcancakir merged 2 commits into
masterfrom
feature/sync-and-support
Sep 26, 2026
Merged

anilcancakir merged 2 commits into
masterfrom
feature/sync-and-support

Conversation

@anilcancakir

Copy link
Copy Markdown
Member

What is added

  • SyncFeed, SyncLedger, SyncBookmarks, SyncPushRow, SyncReport and CreateSyncCursorsTable: a client-side push-then-pull sync skeleton over a documented wire protocol. Ledger writes are issued in a SAVEPOINT so they nest under a caller's own transaction.
  • Str.ascii (Latin-only diacritic fold, case preserved) and Str.squish (Laravel semantics). The User-Agent app-name folding now runs through Str.ascii.
  • AppLifecycle.states(), a lazy lifecycle stream.
  • Exports, doc/digging-deeper/sync.md, the helpers doc, a README row, a CHANGELOG entry under Unreleased, the magic-framework skill references (including the starter guest claim marked unreleased) and the SKILL.md stamp.

Sibling-compile note

No version bump and no publish. This targets master for local override consumption by magic_starter and watchools while those sibling PRs are open.

example/ is deliberately not updated

example/ is knowingly not updated here, against magic's own CLAUDE.md rule that a structural or feature change gets an example wired to it. The sync skeleton needs a backend that speaks its wire protocol, which example/ does not have; Str.ascii, Str.squish and AppLifecycle.states are plain helpers with no UI surface to demonstrate in the example app.

Local gates

1836 tests, analyze and format clean.

…le.states

Add SyncFeed, SyncLedger, SyncBookmarks, SyncPushRow, SyncReport and
CreateSyncCursorsTable: a client-side push-then-pull sync skeleton over
a documented wire protocol, with ledger writes issued in a SAVEPOINT so
they nest under a caller's own transaction.

Add Str.ascii (Latin-only diacritic fold, case preserved) and
Str.squish (Laravel semantics), and route the User-Agent app-name
folding through Str.ascii instead of its own logic.

Add AppLifecycle.states(), a lazy lifecycle stream.

Update exports, doc/digging-deeper/sync.md, the helpers doc, the
README row, CHANGELOG under Unreleased, the magic-framework skill
references (including the starter guest claim marked unreleased) and
the SKILL.md stamp.
@codecov

codecov Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.97080% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
lib/src/sync/sync_feed.dart 93.82% 5 Missing ⚠️
lib/src/sync/create_sync_cursors_table.dart 70.00% 3 Missing ⚠️
lib/src/sync/sync_ledger.dart 84.21% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@kodizm

kodizm Bot commented Sep 26, 2026

Copy link
Copy Markdown

Note

Kodizm (AI-generated). May contain mistakes; verify before acting.

Mostly sound, but one push-mark edge case can leave local rows unsent forever. Worth fixing before consumers build on this skeleton.

Major

lib/src/sync/sync_feed.dart:241: (data-loss) After each successful batch the mark moves to slice.last.mark, but nothing requires marks to be unique across a batch boundary. Here is how rows get lost:

  1. A bulk write stamps 600 rows with the same updated_at_ms.
  2. Batch 1 (500 rows) succeeds, so the mark becomes M.
  3. Batch 2 fails, and run persists M (line 157).
  4. The next run calls pending(sinceMillis: M). The documented reader in doc/digging-deeper/sync.md:62 uses updated_at_ms > ?, so the remaining 100 rows at exactly M are never sent.

Possible fixes: only advance to a mark strictly below the next unsent row's mark, or document pending as inclusive (>=) and rely on the server's >= idempotency the docs already lean on. test/sync/sync_feed_test.dart:34 notes that no scenario uses a real sinceMillis filter, so no test covers this.

Minor

lib/src/sync/sync_feed.dart:144: (correctness) If pending or adoptRow throws after a batch was accepted (for example from the _adopt at line 244), the outer catch returns without writing the ledger. The report then says pushed: 0 even though the server took rows. The resend is harmless under the server's >=, but the count is wrong.

Other notes

  • SyncLedger.write's savepoint handling mirrors the autocommit guard already in Migrator. It looks correct both nested inside a caller's transaction and on its own.
  • Str.ascii keeps the User-Agent folding behaviour and adds three new code points to it: the Romanian comma-below letters, ẞ and U+212B (Angstrom sign).
  • AppLifecycle.states() removes each subscription's observer when that subscription is cancelled.
  • The PR says it skips example/, which departs from the CLAUDE.md post-change-sync rule 5. That was declared up front with a reason, so I'm noting it here rather than flagging it.

Tests

There are new tests for SyncFeed, SyncLedger, Str.ascii/squish (including the fold-parity walk), AppLifecycle and exports. None of them cover equal marks across a batch boundary.

CI

  • Lint & Test: success
  • codecov/patch: success (91.47% of the diff covered)
  • Internal Links & Anchors: success
  • External Links: skipped
  • Dependabot auto-merge: skipped

After a successful push batch, SyncFeed.run advanced the mark to
slice.last.mark unconditionally. Marks are a local clock, not a row
id, so a bulk write can stamp many rows with the same mark; when that
run straddles a batch boundary, advancing to it makes the next run's
sinceMillis filter (a strict >) skip the still-unsent rows at that
mark forever.

_push now advances only to the greatest mark in the batch that is
strictly below a shared value on the next unsent row, or leaves the
mark where it was when every row in the batch carries it. The rows
the mark stops short of are resent next run, absorbed by the server's
own >= as a duplicate rather than lost.

Also: a throw from pending, adoptRow or the ledger after a batch
landed used to report pushed: 0. run() now tracks accepted rows in a
variable outside the try and reports that instead.
@anilcancakir

Copy link
Copy Markdown
Member Author

Fixed both findings in 787edd7 (fix: never step the push mark past rows that share its value).

Major (data loss): _push no longer advances the mark to slice.last.mark unconditionally. It now checks whether the next unsent row shares that mark; if so it advances only to the greatest mark in the batch strictly below the shared value, or leaves the mark where it was when every row in the batch carries it. This is the new _markAfterBatch helper. The rows the mark stops short of are resent on the next run, absorbed by the server's own >= as a duplicate rather than lost. Added a class-doc line on pending noting that equal marks may straddle a batch boundary, next to the existing "oldest first" rule.

Minor: run() now tracks accepted rows in pushedSoFar, a variable declared outside the try block and updated via an onBatchAccepted callback passed into _push as each batch clears its 2xx. The catch block reports pushed: pushedSoFar instead of a hardcoded 0, so a throw from pending, adoptRow or the ledger after a batch landed reports what the server actually accepted. The ledger is still never written from the catch.

Tests added in test/sync/sync_feed_test.dart (TDD: both failed for the right reason before the fix):

  • 600 rows sharing one mark, batch size 500: first batch's mark stays below the shared value, second run resends the remainder (and may resend the first batch).
  • Existing distinct-marks test stays green (mark still advances to the batch's last mark when the next row's mark is strictly greater).
  • An adoptRow that throws after a batch's 2xx now reports pushed equal to that batch's size.

Added one clause to the existing [Unreleased] SyncFeed entry in CHANGELOG.md describing the mark-safety and partial-progress-on-throw behavior (kept to one clause, no new bullet).

flutter analyze: no issues. flutter test: full suite green (1838 tests). dart format --output=none --set-exit-if-changed .: exit 0.

@kodizm review

@kodizm

kodizm Bot commented Sep 26, 2026

Copy link
Copy Markdown

Note

Kodizm (AI-generated). May contain mistakes; verify before acting.

Both earlier findings are fixed in 787edd7 and I found nothing new, so this looks ready to merge.

This is a re-review. The only new commit is 787edd7, which touches lib/src/sync/sync_feed.dart, test/sync/sync_feed_test.dart and CHANGELOG.md. The rest of the PR hasn't changed since my last review.

  • Major (push mark skipping rows that share its value): fixed. _markAfterBatch (lib/src/sync/sync_feed.dart:285) moves the mark to the batch's last value only when that value is finished: either no rows are left, or the next row's mark is strictly higher. Otherwise it moves to the highest mark in the batch that is strictly lower, or keeps the previous mark. I checked three cases:

    • Shared mark filling a whole batch: the last batch of that run still ends at the end of changed, so the mark moves to M only once every row at M has been sent.
    • Order: the mark never goes backwards, as long as pending returns rows oldest first as documented.
    • Server: rows sent again are absorbed as duplicates by the server's >=.

    The > reader in doc/digging-deeper/sync.md is still correct with this change. One limit remains: if more than batchSize rows share a mark and a later batch keeps failing, no run makes progress. The rows are delayed, not lost, so I'm not flagging it.

  • Minor (pushed: 0 after a throw): fixed. The onBatchAccepted callback updates pushedSoFar, and the catch now reports that count. The ledger is still never written from the catch, which is correct.

Tests

Two tests were added in test/sync/sync_feed_test.dart:

  • 600 rows sharing one mark, sent in batches of 500: the first run reports pushed: 500 and is incomplete, and the second run finishes.
  • adoptRow throws after a batch was accepted: the report counts that batch's rows as pushed.

CI

  • Lint & Test: success
  • codecov/patch: success (91.97% of the diff covered)
  • Internal Links & Anchors: success
  • External Links: skipped
  • Auto-merge low-risk Dependabot PRs: skipped

@anilcancakir
anilcancakir merged commit c7d7f06 into master Sep 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant