-
Notifications
You must be signed in to change notification settings - Fork 23
feat: show remaining CDN & cache-miss egress #538
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
bajtos
wants to merge
26
commits into
master
Choose a base branch
from
feat-egress-allowance
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
26 commits
Select commit
Hold shift + click to select a range
fd491d0
feat: show remaining CDN & cache-miss egress
bajtos 9aa16b7
chore: document synapse-react in AGENTS.md
bajtos 2132a05
fixup! cache-miss icon RefreshCW -> FolderSync
bajtos f029bc4
fixup! npm run lint:fix
bajtos b1f0f04
fixup! code cleanup + better error logging
bajtos 10c3811
fixup! forgotten err
bajtos c0bbd0e
Merge branch 'master' into feat-egress-allowance
bajtos ac134f7
fix(playground): improve egress quota display in Data Sets UI
bajtos ad49466
refactor: move FilBeam egress quota logic to synapse-core
bajtos 164bc70
test(synapse-core): add tests for FilBeam stats module
bajtos 725f20b
fix: add PiB and EiB to formatBytes helper
bajtos 30952c2
docs: clarify CDN & cache-miss quotas in API docs
bajtos abd0d39
Merge branch 'master' into feat-egress-allowance
bajtos 47d0a25
refactor(playground): extract CdnDetails component
bajtos 730aeea
fix(synapse-react): use plural query key for egress quotas
bajtos f2bc852
refactor(synapse-sdk): delegate FilBeamService.getDataSetStats to syn…
bajtos 3914329
test: move getDataSetStats tests to synapse-core
bajtos 5905842
refactor(synapse-core): organize filbeam stats code top-down
bajtos 6f7f718
Fix formatBytes to support negative values
bajtos 0063a33
fix: harden the implementation of formatBytes
bajtos 1bd4f2c
Update AGENTS.md
bajtos 53c49e5
Update AGENTS.md
bajtos abc7ecb
Merge branch 'master' into feat-egress-allowance
bajtos 501b661
refactor(core): move FilBeam stats URL to chain config
bajtos a7f6bb4
Update apps/synapse-playground/src/lib/utils.ts
bajtos 8071a70
test(core): add test for FilBeam unsupported chain error
bajtos File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
apps/synapse-playground/src/components/warm-storage/cdn-details.tsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| import { useEgressQuota } from '@filoz/synapse-react' | ||
| import { Globe } from 'lucide-react' | ||
| import { formatBytes } from '@/lib/utils.ts' | ||
| import { Tooltip, TooltipContent, TooltipTrigger } from '../ui/tooltip.tsx' | ||
|
|
||
| export function CdnDetails({ dataSetId }: { dataSetId: bigint }) { | ||
| const { data: egressQuota } = useEgressQuota({ dataSetId }) | ||
|
|
||
| return ( | ||
| <span className="flex items-center gap-1 text-sm text-muted-foreground"> | ||
| <Tooltip> | ||
| <TooltipTrigger> | ||
| <Globe className="w-4" /> | ||
| </TooltipTrigger> | ||
| <TooltipContent> | ||
| <p>This data set is using CDN</p> | ||
| </TooltipContent> | ||
| </Tooltip> | ||
| {egressQuota && ( | ||
| <> | ||
| Egress remaining: {formatBytes(egressQuota.cdnEgressQuota)} delivery{' · '} | ||
bajtos marked this conversation as resolved.
Show resolved
Hide resolved
bajtos marked this conversation as resolved.
Show resolved
Hide resolved
bajtos marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| {formatBytes(egressQuota.cacheMissEgressQuota)} cache-miss | ||
| </> | ||
| )} | ||
| </span> | ||
| ) | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| import { isSynapseError, SynapseError } from './base.ts' | ||
|
|
||
| export class GetDataSetStatsError extends SynapseError { | ||
| override name: 'GetDataSetStatsError' = 'GetDataSetStatsError' | ||
|
|
||
| static override is(value: unknown): value is GetDataSetStatsError { | ||
| return isSynapseError(value) && value.name === 'GetDataSetStatsError' | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| /** | ||
| * FilBeam API | ||
| * | ||
| * @example | ||
| * ```ts | ||
| * import * as FilBeam from '@filoz/synapse-core/filbeam' | ||
| * ``` | ||
| * | ||
| * @module filbeam | ||
| */ | ||
| export * from './stats.ts' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| /** | ||
| * FilBeam Stats API | ||
| * | ||
| * @example | ||
| * ```ts | ||
| * import { getDataSetStats } from '@filoz/synapse-core/filbeam' | ||
| * ``` | ||
| * | ||
| * @module filbeam | ||
| */ | ||
|
|
||
| import { HttpError, request } from 'iso-web/http' | ||
| import type { Chain } from '../chains.ts' | ||
| import { GetDataSetStatsError } from '../errors/filbeam.ts' | ||
|
|
||
| /** | ||
| * Data set statistics from FilBeam. | ||
| * | ||
| * These quotas represent the remaining pay-per-byte allocation available for data retrieval | ||
| * through FilBeam's trusted measurement layer. The values decrease as data is served and | ||
| * represent how many bytes can still be retrieved before needing to add more credits. | ||
| */ | ||
| export interface DataSetStats { | ||
| /** The remaining quota for all requests served by FilBeam (both cache-hit and cache-miss) in bytes */ | ||
| cdnEgressQuota: bigint | ||
| /** The remaining quota for cache-miss requests served by the Storage Provider in bytes */ | ||
| cacheMissEgressQuota: bigint | ||
| } | ||
|
|
||
| export interface GetDataSetStatsOptions { | ||
| /** The chain configuration containing FilBeam stats URL */ | ||
| chain: Chain | ||
| /** The data set ID to query */ | ||
| dataSetId: bigint | number | string | ||
| /** Optional override for request.json.get (for testing) */ | ||
| requestGetJson?: typeof request.json.get | ||
| } | ||
|
|
||
| /** | ||
| * Retrieves remaining pay-per-byte statistics for a specific data set from FilBeam. | ||
| * | ||
| * Fetches the remaining CDN and cache miss egress quotas for a data set. These quotas | ||
| * track how many bytes can still be retrieved through FilBeam's trusted measurement layer | ||
| * before needing to add more credits: | ||
| * | ||
| * - **CDN Egress Quota**: Remaining bytes for all requests served by FilBeam (both cache-hit and cache-miss) | ||
| * - **Cache Miss Egress Quota**: Remaining bytes for cache-miss requests served by the Storage Provider | ||
| * | ||
| * @param options - The options for fetching data set stats | ||
| * @returns A promise that resolves to the data set statistics with remaining quotas as BigInt values | ||
| * | ||
| * @throws {GetDataSetStatsError} If the chain doesn't support FilBeam, data set is not found, the API returns an invalid response, or network errors occur | ||
| * | ||
| * @example | ||
| * ```typescript | ||
| * import { mainnet } from '@filoz/synapse-core/chains' | ||
| * | ||
| * const stats = await getDataSetStats({ chain: mainnet, dataSetId: 12345n }) | ||
| * console.log(`Remaining CDN Egress: ${stats.cdnEgressQuota} bytes`) | ||
| * console.log(`Remaining Cache Miss: ${stats.cacheMissEgressQuota} bytes`) | ||
| * ``` | ||
| */ | ||
| export async function getDataSetStats(options: GetDataSetStatsOptions): Promise<DataSetStats> { | ||
| if (!options.chain.filbeam) { | ||
| throw new GetDataSetStatsError(`Chain ${options.chain.id} (${options.chain.name}) does not support FilBeam`) | ||
| } | ||
bajtos marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| const baseUrl = options.chain.filbeam.statsBaseUrl | ||
| const url = `${baseUrl}/data-set/${options.dataSetId}` | ||
| const requestGetJson = options.requestGetJson ?? request.json.get | ||
|
|
||
| const response = await requestGetJson<unknown>(url) | ||
|
|
||
| if (response.error) { | ||
| if (HttpError.is(response.error)) { | ||
| const status = response.error.response.status | ||
| if (status === 404) { | ||
| throw new GetDataSetStatsError(`Data set not found: ${options.dataSetId}`, { | ||
| cause: response.error, | ||
| }) | ||
| } | ||
| const errorText = await response.error.response.text().catch(() => 'Unknown error') | ||
| throw new GetDataSetStatsError(`Failed to fetch data set stats`, { | ||
| details: `HTTP ${status} ${response.error.response.statusText}: ${errorText}`, | ||
| cause: response.error, | ||
| }) | ||
| } | ||
| throw new GetDataSetStatsError('Unexpected error', { cause: response.error }) | ||
| } | ||
|
|
||
| return validateStatsResponse(response.result) | ||
|
Comment on lines
+71
to
+90
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Following up on #538 (comment) and #538 (comment). If we apply both suggestions, then we can simplify this block as follows - is that what we want? const response = await requestGetJson<unknown>(url)
if (response.error) {
throw new GetDataSetStatsError(
'Cannot get DataSet stats from FilBeam API.',
{ cause: response.error }
)
}
return {
cdnEgressQuota: BigInt(response.result.cdnEgressQuota),
cacheMissEgressQuota: BigInt(response.result.cacheMissEgressQuota),
} |
||
| } | ||
|
|
||
| /** | ||
| * Validates that a string can be converted to a valid BigInt | ||
| */ | ||
| function parseBigInt(value: string, fieldName: string): bigint { | ||
| // Check if the string is a valid integer format (optional minus sign followed by digits) | ||
| if (!/^-?\d+$/.test(value)) { | ||
| throw new GetDataSetStatsError('Invalid response format', { | ||
| details: `${fieldName} is not a valid integer: "${value}"`, | ||
bajtos marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }) | ||
| } | ||
| return BigInt(value) | ||
bajtos marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
|
|
||
| /** | ||
| * Validates the response from FilBeam stats API and returns DataSetStats | ||
| */ | ||
| export function validateStatsResponse(data: unknown): DataSetStats { | ||
| if (typeof data !== 'object' || data === null) { | ||
| throw new GetDataSetStatsError('Invalid response format', { | ||
| details: 'Response is not an object', | ||
| }) | ||
| } | ||
|
|
||
| const response = data as Record<string, unknown> | ||
|
|
||
| if (typeof response.cdnEgressQuota !== 'string') { | ||
| throw new GetDataSetStatsError('Invalid response format', { | ||
| details: 'cdnEgressQuota must be a string', | ||
| }) | ||
| } | ||
|
|
||
| if (typeof response.cacheMissEgressQuota !== 'string') { | ||
| throw new GetDataSetStatsError('Invalid response format', { | ||
| details: 'cacheMissEgressQuota must be a string', | ||
| }) | ||
| } | ||
|
|
||
| return { | ||
| cdnEgressQuota: parseBigInt(response.cdnEgressQuota, 'cdnEgressQuota'), | ||
| cacheMissEgressQuota: parseBigInt(response.cacheMissEgressQuota, 'cacheMissEgressQuota'), | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The component doesn't handle loading or error states from the useEgressQuota hook. If the API request is slow or fails, the egress quota information will simply not appear, which could be confusing to users. Consider adding a loading skeleton (similar to the pattern used in services.tsx line 32-34) or an error state display to provide better user feedback. At minimum, showing a loading indicator while fetching would improve the user experience.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do we feel about this?
Are we okay to ship the current version, since it is still an improvement, or is it required to handle loading & error states before we can ship the remaining egress indicators?
@juliangruber @hugomrdias
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is good enough for the playground components