Skip to content

feat(synapse-core): add SP-to-SP piece pull functionality#544

Open
rvagg wants to merge 5 commits intomasterfrom
rvagg/sp-sp-fetch
Open

feat(synapse-core): add SP-to-SP piece pull functionality#544
rvagg wants to merge 5 commits intomasterfrom
rvagg/sp-sp-fetch

Conversation

@rvagg
Copy link
Collaborator

@rvagg rvagg commented Jan 12, 2026

@github-project-automation github-project-automation bot moved this to 📌 Triage in FOC Jan 12, 2026
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Jan 12, 2026

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
synapse-dev 2f8c0ca Commit Preview URL

Branch Preview URL
Feb 05 2026, 12:26 PM

@BigLep BigLep moved this from 📌 Triage to ⌨️ In Progress in FOC Jan 15, 2026
@BigLep BigLep linked an issue Jan 16, 2026 that may be closed by this pull request
5 tasks
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from c2011f6 to 7af205b Compare January 17, 2026 06:01
@rvagg
Copy link
Collaborator Author

rvagg commented Jan 17, 2026

Added a temporary utils/example-sp-fetch-e2e.js testing file to test the sp-sp fetch flow. It stores pieces with one SP, confirms they are parked, then asks the second SP to fetch from the first and confirms they are parked. It's working currently on devnet using FilOzone/foc-devnet#34 👌.

@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from ac3552a to 3c5cdb9 Compare January 20, 2026 09:08
@rvagg rvagg changed the base branch from master to hugomrdias/move-tests January 20, 2026 09:08
@hugomrdias hugomrdias force-pushed the hugomrdias/move-tests branch 2 times, most recently from 0a20df0 to a231d32 Compare January 26, 2026 12:20
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from 3c5cdb9 to 429c2e3 Compare January 28, 2026 10:29
@rvagg rvagg changed the base branch from hugomrdias/move-tests to hugomrdias/session-key January 28, 2026 10:30
@rvagg
Copy link
Collaborator Author

rvagg commented Jan 28, 2026

Rebased on top of the latest @hugomrdias stack #571

@rjan90 rjan90 added this to the M4.0: mainnet staged milestone Jan 30, 2026
Base automatically changed from hugomrdias/session-key to hugomrdias/payments January 30, 2026 12:03
Base automatically changed from hugomrdias/payments to hugomrdias/move-tests February 2, 2026 19:19
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from 429c2e3 to 8ec5942 Compare February 3, 2026 03:23
@hugomrdias hugomrdias force-pushed the hugomrdias/move-tests branch from a5f8971 to d75fb75 Compare February 3, 2026 11:52
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from 8ec5942 to b6d3479 Compare February 4, 2026 10:44
@rvagg rvagg changed the base branch from hugomrdias/move-tests to master February 4, 2026 10:45
@rvagg rvagg changed the title (WIP) feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers Feb 4, 2026
@rvagg rvagg marked this pull request as ready for review February 4, 2026 10:45
@rvagg rvagg requested a review from hugomrdias as a code owner February 4, 2026 10:45
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch 2 times, most recently from b221f97 to ff14a1d Compare February 4, 2026 11:31
@rjan90 rjan90 moved this from ⌨️ In Progress to 🔎 Awaiting review in FOC Feb 4, 2026
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from ff14a1d to 13ed974 Compare February 5, 2026 10:30
@rvagg rvagg changed the title feat(synapse-core): add sp-fetch module for SP-to-SP piece transfers feat(synapse-core): add SP-to-SP piece pull functionality Feb 5, 2026
rvagg added 4 commits February 5, 2026 21:31
Add low-level Pull module for initiating piece pulls from external storage
providers to the local SP via Curio's POST /pdp/piece/pull endpoint.

Features:
- fetchPieces: Initiate pull requests with EIP-712 signed authorization
- waitForFetchStatus: Poll until completion with status callbacks
- Support for both existing data sets (signAddPieces) and new data sets
  (signCreateDataSetAndAddPieces)
- Configurable timeout and abort signal support

The pull endpoint is idempotent - repeated calls with the same extraData
return status updates rather than creating duplicates.
@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from 13ed974 to 2d1a65e Compare February 5, 2026 10:32
@rvagg
Copy link
Collaborator Author

rvagg commented Feb 5, 2026

latest fixup in here adds the ability to supply a pre-built extraData since we use it for auth but then use it again to addPieces and we don't want the user to have to sign twice for the same thing if we can help it

@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from 2d1a65e to caebe09 Compare February 5, 2026 12:15
@rvagg
Copy link
Collaborator Author

rvagg commented Feb 5, 2026

And another fix, I realised I still had a ton of "fetch" in here. Now all "pull".

@rvagg rvagg force-pushed the rvagg/sp-sp-fetch branch from caebe09 to 2f8c0ca Compare February 5, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: 🔎 Awaiting review

Development

Successfully merging this pull request may close these issues.

GA DURABILITY: Multi-copy upload via SP-to-SP pull

2 participants