-
-
Notifications
You must be signed in to change notification settings - Fork 109
xftp server: support storage time and BBS proofs of badge credential to extend it #1856
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
epoberezkin
wants to merge
19
commits into
master
Choose a base branch
from
ep/badges-xftp
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
4989bd8
xftp server: support storage time and BBS proofs of badge credential …
evgeny-simplex add6542
update
evgeny-simplex 0df161a
types
evgeny-simplex c089462
implementation
evgeny-simplex fddc151
update
evgeny-simplex ed9786a
remove permanent and FTTL
evgeny-simplex 9507b2d
refactor
evgeny-simplex 83accac
refactor
evgeny-simplex 3aad156
refactor
evgeny-simplex 121628a
simplify
epoberezkin 46bb52d
simplify
epoberezkin 8a998a7
simpler
epoberezkin af4bed8
remove comments
evgeny-simplex 1ce2df9
add file expiration time to agent event
evgeny-simplex f064513
add test of upload with entitlement
evgeny-simplex 173916b
update schema
epoberezkin 6541745
fix header and test
evgeny-simplex 370c4e6
fix store log encoding, add xftp store log tests
evgeny-simplex 775fb8b
:send entitlement proof in handshake
evgeny-simplex File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| # Implementation plan: XFTP variable file storage time | ||
|
|
||
| Proposal: `../rfcs/2026-08-22-xftp-file-storage-time.md`. | ||
|
|
||
| ## simplexmq: entitlement crypto | ||
|
|
||
| New module `Simplex.Messaging.Crypto.Entitlement`, over `Simplex.Messaging.Crypto.BBS`: | ||
|
|
||
| Types: | ||
|
|
||
| ``` | ||
| newtype MasterKey = MasterKey ByteString | ||
|
|
||
| data Entitlement = Entitlement | ||
| { entitlementName :: Text, | ||
| expiresAt :: UTCTime, | ||
| extraInfo :: Text | ||
| } | ||
|
|
||
| data EntitlementCredential = EntitlementCredential | ||
| { issuerKeyIdx :: Int, | ||
| masterKey :: MasterKey, | ||
| issuerSignature :: BBSSignature, | ||
| entitlement :: Entitlement | ||
| } | ||
|
|
||
| data EntitlementProof = EntitlementProof | ||
| { issuerKeyIdx :: Int, | ||
| proof :: BBSProof, | ||
| entitlement :: Entitlement | ||
| } | ||
| ``` | ||
|
|
||
| Functions and constants: | ||
|
|
||
| - the disclosed-message encoding: the master key is message 0 and stays undisclosed; `expiresAt`, `entitlementName`, and `extraInfo` are messages 1 to 3 and are disclosed | ||
| - the BBS header string `"SimpleX badges v1"` (shared with chat's badges, which sign under it), the message count, and the disclosed indexes | ||
| - `generateEntitlementProof :: BBSPublicKey -> EntitlementCredential -> BBSPresHeader -> IO (Either String EntitlementProof)` | ||
| - `verifyEntitlement :: Map Int BBSPublicKey -> BBSPresHeader -> EntitlementProof -> IO (Maybe Bool)` (the caller supplies the presentation header; the server reconstructs it, the proof never includes it) | ||
| - the issuer public keys constant `Map Int BBSPublicKey` | ||
|
|
||
| ## simplexmq: protocol, new XFTP version | ||
|
|
||
| In `Simplex.FileTransfer.Transport`: | ||
|
|
||
| - add the next `VersionXFTP` and set `currentXFTPVersion` to 4 | ||
| - add `entitlementProof :: Maybe EntitlementProof` to `XFTPClientHandshake`, encoded before the `Tail` and only from this version | ||
| - the presentation header is the session id alone | ||
|
|
||
| In `Simplex.FileTransfer.Protocol`: | ||
|
|
||
| - add `GrantedStorageTime` and its encoding; retain the one-character sum prefix for future variants: | ||
|
|
||
| ``` | ||
| data GrantedStorageTime = GSTExpires {epochSeconds :: Int64} | ||
| ``` | ||
|
|
||
| - add the storage time (`Maybe Int64`: `Nothing` requests the server maximum, `Just` a number of hours) to `FNEW` | ||
| - add the granted storage to `FRSndIds` as `Maybe GrantedStorageTime` (`Nothing` when decoding a response from a server below this version) | ||
|
|
||
| ## simplexmq: server configuration | ||
|
|
||
| In `Simplex.FileTransfer.Server.Env` and `Simplex.FileTransfer.Server.Main`: | ||
|
|
||
| - make `fileExpiration` non-optional (`ExpirationConfig`, no longer `Maybe`); the server always expires files, so the server maximum is always a concrete number of seconds | ||
| - read a maximum storage time (a number of hours) for each entitlement name from the `[STORE_LOG]` INI section, from the keys `expire_files_hours_for_supporter` and `expire_files_hours_for_legend`; an absent key is skipped (that name gets the default), a present but malformed value fails startup | ||
| - exit at startup if any name's maximum is below the default file expiration | ||
| - add `entitlementKeys :: Map Word16 BBSPublicKey` to the server config (default = the shared constant, set from `Main`); the handshake verifies the proof against it, so the trusted keys never come from the sender | ||
|
|
||
| ## simplexmq: server session | ||
|
|
||
| In `Simplex.FileTransfer.Server`: | ||
|
|
||
| - `processClientHandshake` verifies the proof from the handshake, once per session, and resolves the maximum storage time for the entitlement name | ||
| - verify only when the answer can change: the name is configured with a maximum above the default, and the entitlement expired less than 24 hours ago. A proof that fails these checks or fails to verify is logged, and the session gets the default maximum | ||
| - `HandshakeAccepted` holds the resolved maximum for the session, and `processXFTPRequest` takes it from there, so no proof is verified while a command is processed | ||
| - `createFile` caps the requested storage time by the session maximum | ||
|
|
||
| ## simplexmq: server store and expiration | ||
|
|
||
| The `files` table gets a nullable `expires_at`. Every new file stores a concrete `expires_at`. It is NULL only for pre-feature rows, which the migration must not re-date (it has no access to the operator's configured TTL); those are expired at query time as `created_at + ttl`. | ||
|
|
||
| Common to both stores, in `Simplex.FileTransfer.Server.Store`: | ||
|
|
||
| - add `expiresAt :: Maybe RoundedFileTime` to `FileRec` | ||
| - in `createFile`, verify the proof against `sessionId <> sndKey <> digest`, cap the requested hours at the entitlement's maximum, round the expiry up to the hour, store it, and return that same value as the granted storage | ||
| - a valid proof raises the maximum to the entitlement's configured value; a proof that fails verification, carries an unknown issuer key, or whose entitlement expired more than 24 hours ago falls back to the default maximum. The entitlement is honoured for 24 hours after its `expiresAt`. | ||
| - `expiredFiles` receives `now` and `old` (= `now - ttl`). A stored expiry is deleted when `expires_at < now` (no grace — it is already rounded up); a legacy row (no `expires_at`) is deleted when `created_at + fileTimePrecision < old` (the grace covers `created_at` being floored to the hour) | ||
| - retain `created_at` for statistics, export, and the legacy fallback | ||
|
|
||
| STM store: | ||
|
|
||
| - in `expiredFiles`, expire a new file when `roundedSeconds expiresAt < now`, and a legacy file (no `expiresAt`) when `created_at + fileTimePrecision < old` | ||
|
|
||
| PostgreSQL store, in `Simplex.FileTransfer.Server.Store.Postgres` and its migrations: | ||
|
|
||
| - add the nullable column `expires_at BIGINT` (no backfill) | ||
| - add one composite index `idx_files_expiry ON files (expires_at, created_at)` | ||
| - `expiredFiles` query: `WHERE (expires_at < ?) OR (expires_at IS NULL AND created_at < ?) LIMIT ?` with `(now, old - fileTimePrecision)`. The first arm deletes stored (already rounded-up) expiries; the second drains legacy rows, with the grace folded into `old - fileTimePrecision` so the columns stay bare and sargable. Keep the `OR` at the top level so each disjunct is independently indexable (BitmapOr on the composite index): `expires_at` covers arm 1's range and arm 2's `IS NULL` group, and `created_at` orders arm 2 within that group. A `COALESCE(expires_at, created_at + ttl)` predicate is avoided (not sargable, would force a sequential scan). No `ORDER BY` — the batch loop deletes all expired rows regardless of order. | ||
|
|
||
| Store log, in `Simplex.FileTransfer.Server.StoreLog`: | ||
|
|
||
| - add the optional expiration to the `AddFile` record; a record without it parses to `Nothing` (the configured default), never a hardcoded value | ||
|
|
||
| ## simplexmq: agent | ||
|
|
||
| The credential belongs to the user, so the agent holds it the way it holds the user's servers: in memory, supplied when the agent is created and replaced through an API. It is not stored by the agent. | ||
|
|
||
| Per-user state in `Simplex.Messaging.Agent.Env.SQLite` and `Simplex.Messaging.Agent.Client`: | ||
|
|
||
| - add `entitlements :: Map UserId EntitlementCredential` to `InitialAgentServers`, beside the servers | ||
| - add `userEntitlements :: TMap UserId EntitlementCredential` to `AgentClient`, filled from it by `newAgentClient` | ||
| - add `entitlementKeys :: Map Word16 BBSPublicKey` to `AgentConfig` (default = the shared constant), for the issuer key that proof generation needs | ||
|
|
||
| Public API in `Simplex.Messaging.Agent`: | ||
|
|
||
| - add storage time (`Maybe Int64` hours) to `xftpSendFile` | ||
| - add `setUserEntitlement :: AgentClient -> UserId -> Maybe EntitlementCredential -> IO ()`, in the shape of `setProtocolServers`: it replaces the entry, and closes that user's XFTP clients, so the next upload presents the new credential | ||
|
|
||
| Store, in both the SQLite and PostgreSQL agent stores: | ||
|
|
||
| - add a nullable storage time column (integer hours; NULL means the server maximum) to `snd_files` | ||
| - add the migration to both stores | ||
| - in `createSndFile`, store the storage time | ||
|
|
||
| Upload, in `Simplex.Messaging.Agent.Client` and `Simplex.FileTransfer.Client`: | ||
|
|
||
| - `getXFTPClient` takes a proof for the session as a parameter, `SessionId -> IO (Maybe EntitlementProof)`, beside the callback it already takes for a closed client. The client config holds no credential and no keys | ||
| - `getXFTPServerClient` passes a function that reads the user's credential, looks the issuer key up, and generates the proof over the session id. A missing credential or a failure to generate gives `Nothing`, with the failure logged | ||
| - `xftpClientHandshakeV1` calls it with the session id from the connection, and sends the result in the handshake | ||
| - `agentXFTPNewChunk` reads the storage time from the send record and sends FNEW with it | ||
| - `createXFTPChunk` returns the granted expiry (epoch seconds); `agentXFTPNewChunk` stores it on `NewSndChunkReplica` | ||
|
|
||
| Completion: | ||
|
|
||
| - `createXFTPChunk` returns the granted expiry as `Maybe GrantedStorageTime`; `SndFileChunkReplica` and `NewSndChunkReplica` carry `expiresAt :: Maybe GrantedStorageTime` | ||
| - persist it in a nullable `replica_expires_at` column on `snd_file_chunk_replicas` (added to the entitlement migration): `createSndFileReplica` stores `epochSeconds`, `getSndFile` reads it back into `GSTExpires` | ||
| - on `SFDONE`, report the file expiry: a chunk expires when its last replica expires (`max` over replicas, absent replicas ignored, `Nothing` only if none report); the file expires when its first chunk expires (`min` over chunks, `Nothing` if any chunk is unknown). `GrantedStorageTime` derives `Ord` | ||
| - `SFDONE` gains a trailing `Maybe GrantedStorageTime` (not str-encoded); chat consumes it (wired later) | ||
|
|
||
| Testing: | ||
|
|
||
| - e2e test in `tests/XFTPAgent.hs`: generate a BBS keypair, sign a supporter credential (issuer key index 1), run the server with `entitlementKeys = {1: testPk}` and a supporter maximum above the default, run the sender agent with the same `entitlementKeys` and the credential for the user, send a file requesting a number of hours above the default and below that maximum, and assert `SFDONE`'s granted expiry rounds up `now + requested` (proof of the entitlement raising the max above the default) | ||
| - the same upload without the credential is capped at the default maximum | ||
| - store log round trip in `tests/CoreTests/StoreLogTests.hs`, in the shape of the SMP store log test: a file record survives a write, a read into the store, and compaction, including a file blocked with a notice, where the record has a field after the blocking info | ||
|
|
||
| ## simplex-chat | ||
|
|
||
| - remove lifetime badges: make `badgeExpiry` a `UTCTime`, drop the `"lifetime"` encoding, and remove the lifetime option from the UI and the CLI | ||
| - map `BadgeInfo` to `Entitlement` (`entitlementName = textEncode badgeType`, `expiresAt = badgeExpiry`, `extraInfo = badgeExtra`) when calling the agent | ||
| - pass the user's credential to the agent when it is created, and through `setUserEntitlement` when the badge changes, in the same places that pass and update the user's servers | ||
| - pass the storage time to `xftpSendFile` | ||
| - retain the `maxXFTPFileSize` size limit | ||
| - reuse `verifyEntitlement` for peer-badge verification | ||
| - import the issuer public keys from the shared simplexmq constant | ||
|
|
||
| ## State | ||
|
|
||
| Steps 2 to 5 are implemented in simplexmq. Step 1 and step 6 belong to the chat branch that carries badges. | ||
|
|
||
| ## Order | ||
|
|
||
| 1. Add the entitlement crypto module; move chat's badge verification onto it and remove lifetime badges. | ||
| 2. Add the new XFTP version, the FNEW storage time, and the response. | ||
| 3. Change the server configuration, store, expiration, and store log. | ||
| 4. Move the proof to the handshake: the handshake field, the session state on the server, and the proof for the session on the client. | ||
| 5. Hold the credential per user in the agent, and add the API to replace it. | ||
| 6. Wire chat to pass the credential and the storage time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| # XFTP variable file storage time | ||
|
|
||
| ## Summary | ||
|
|
||
| The server stores a storage time for each file. The sender sets it in the FNEW command. The client may present a proof of an entitlement in the handshake to raise the maximum storage time the server allows. The proof is bound to the TLS session, so it cannot be reused for another session. | ||
|
|
||
| An entitlement belongs to the user, not to a file: the client presents it once per connection, and the server applies it to everything the client does in that session. The server can also vary other limits, such as throttling, by the entitlement. | ||
|
|
||
| ## Entitlement | ||
|
|
||
| An entitlement is a name, an expiration, and an extra string. It is the disclosed content of a BBS proof: the holder's secret remains undisclosed, and the three fields are revealed. The server reads `entName` to select a maximum storage time, checks `entExpires`, and ignores `entExtra`; the interpretation of `entExtra` is out of scope here. The protocol references only the entitlement, never a badge; chat maps its own badge to an entitlement before it asks the agent to send. | ||
|
|
||
| The proof discloses the entitlement and includes the issuer key index and the BBS proof. The holder's secret and the BBS signature remain with the sender and are never transmitted. The origin of the sender's signed entitlement, from the entitlement service, is out of scope here. | ||
|
|
||
| ``` | ||
| entitlement = entName entExpires entExtra | ||
| entName = shortString ; e.g. "supporter", "legend" | ||
| entExpires = shortString ; expiration as a UTCTime ISO8601 string | ||
| entExtra = shortString ; opaque, interpretation out of scope | ||
|
|
||
| entitlementProof = issuerKeyIndex bbsProof entitlement | ||
| issuerKeyIndex = 2*2 OCTET ; Word16, network byte order | ||
| bbsProof = largeString ; BBS proof bytes | ||
| ``` | ||
|
|
||
| The presentation header that the BBS proof is generated over is not transmitted; the server takes it from the session (see [Binding](#binding)), which is what binds the proof. | ||
|
|
||
| ## Storage time | ||
|
|
||
| ``` | ||
| fileStorageTime = %s"0" / (%s"1" storageHours) | ||
| storageHours = 8*8 OCTET ; Int64, network byte order | ||
| ``` | ||
|
|
||
| The storage time is an optional number of hours. Absent (`%s"0"`) requests the maximum the server allows for the presented entitlement, or the default maximum when no proof is present. A value (`%s"1"` with hours) requests a specific number of hours. | ||
|
|
||
| ## Handshake, new XFTP version | ||
|
|
||
| The client handshake carries the entitlement proof. | ||
|
|
||
| ``` | ||
| clientHandshake = xftpVersion keyHash optEntitlementProof | ||
| optEntitlementProof = %s"0" / (%s"1" entitlementProof) | ||
| ``` | ||
|
|
||
| `xftpVersion` and `keyHash` are defined by the current XFTP protocol. Version 3 and earlier encode no proof. | ||
|
|
||
| The server verifies the proof once, when it accepts the handshake, and keeps the resulting maximum storage time for the session. A proof that fails to verify, names an entitlement the server does not configure, or names one whose expiration passed more than 24 hours ago, is logged and ignored, and the session gets the default maximum. The client learns nothing about which entitlements the server accepts. | ||
|
|
||
| ## Commands | ||
|
|
||
| The new protocol version extends FNEW with the storage time. | ||
|
|
||
| ``` | ||
| fnew = %s"FNEW " fileInfo rcvKeys optBasicAuth fileStorageTime | ||
| ``` | ||
|
|
||
| `fileInfo`, `rcvKeys`, and `optBasicAuth` are defined by the current XFTP protocol. Version 3 and earlier encode no `fileStorageTime`, and the server applies the default storage time. | ||
|
|
||
| ## Responses | ||
|
|
||
| FNEW extends the SIDS response with the granted storage. | ||
|
|
||
| ``` | ||
| sndIds = %s"SIDS " senderId rcvIds optGrantedStorageTime | ||
| optGrantedStorageTime = %s"0" / (%s"1" grantedStorageTime) | ||
| grantedStorageTime = grantedExpires | ||
| grantedExpires = %s"T" expiresAt | ||
| expiresAt = 8*8 OCTET ; Int64, seconds since epoch (absolute UTC instant), network byte order | ||
| ``` | ||
|
|
||
| `grantedExpires` returns the absolute expiration — the same value stored for the file. The sum encoding retains a one-character prefix so further variants can be added. Version 3 and earlier omit `optGrantedStorageTime` entirely; a client decoding such a response reads it as absent. `senderId` and `rcvIds` are defined by the current XFTP protocol. | ||
|
|
||
| ## Binding | ||
|
|
||
| The presentation header binds the proof to the TLS session, so a proof presented on any other session fails to verify. | ||
|
|
||
| ``` | ||
| presHeader = sessionId | ||
| ``` | ||
|
|
||
| `sessionId` is the TLS session identifier, the TLS unique channel binding. Both sides take it from the connection: the client has it once TLS is established, and the client checks that the identifier the server sends in its handshake matches. | ||
|
|
||
| Binding to the session is what stops a proof being replayed by another client. A proof is not bound to a file, because the entitlement belongs to the user and authorises everything the client does in that session. | ||
|
|
||
| ## Maximum storage time | ||
|
|
||
| The server configures a maximum storage time for each entitlement name, and a default maximum for requests with no proof. Each maximum is a number of hours. The server exits at startup if any name's maximum is below the default, so a proof never reduces the allowed time. The server honours an entitlement for 24 hours after its expiration; past that grace it is treated as no proof. | ||
|
|
||
| If the requested time exceeds the maximum, the server stores the file for the maximum and does not reject the request. The expiration is rounded up to the hour, stored, and returned as `grantedExpires`. | ||
|
|
||
| ## Encoding primitives | ||
|
|
||
| ``` | ||
| shortString = length *OCTET ; 0-255 bytes | ||
| largeString = length2 *OCTET | ||
| length = 1*1 OCTET | ||
| length2 = 2*2 OCTET ; Word16, network byte order | ||
| ``` | ||
|
|
||
| `senderId`, `rcvIds`, `fileInfo`, `sndKey`, `digest`, `rcvKeys`, `optBasicAuth`, and `sessionId` are defined by the current XFTP and SMP protocols. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.