diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 73ecf4ce..06138335 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -30,7 +30,7 @@ STAGE 0/1 export_client → JSONL → GeoParquet → isamples_export_*_geo.parquet (Export format; ~300MB, 6.7M; Zenodo doi:10.5281/zenodo.15278211) ▼ STAGE 2 pqg/pqg/sql_converter.py (export → base PQG; 7-stage DuckDB SQL) - → narrow (…_narrow.parquet, ~844MB, 106M rows) and wide (…_wide.parquet, ~282MB, 20M rows) + → narrow (…_narrow.parquet, ~860MB, 101,387,180 rows) and wide (…_wide.parquet, ~292MB, 20.7M rows) ▼ STAGE 3 sidecar/enrichment merges (LEFT JOIN by pid) ← Eric's independently-maintained OC PQG (GCS) 3a scripts/enrich_wide_with_oc_thumbnails.py → isamples_202604_wide.parquet (+47K thumbnails) diff --git a/README.md b/README.md index 8b472dc7..adbda4e9 100644 --- a/README.md +++ b/README.md @@ -76,10 +76,10 @@ Updating dependencies using `pip -U <>` and regenerate `requiremen All tutorials query parquet files hosted on Cloudflare R2: ```javascript -// Wide format (recommended) - 280 MB, 20M rows +// Wide format (recommended) - 292 MB, 20.7M rows const WIDE_URL = "https://data.isamples.org/isamples_202601_wide.parquet"; -// Narrow format (advanced) - 850 MB, 106M rows +// Narrow format (advanced) - 860 MB, 101,387,180 rows const NARROW_URL = "https://data.isamples.org/isamples_202512_narrow.parquet"; ``` diff --git a/how-to-use.qmd b/how-to-use.qmd index 84859d69..5860e302 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -75,9 +75,9 @@ The two main files carrying the sample records themselves: | File | Size | Shape | Rows | Use when you need… | |---|---:|---|---:|---| -| [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20 M | General entity queries, UI filtering, description text | -| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 292 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20 M | Geospatial queries with H3 clustering at arbitrary zoom | -| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 820 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 106 M | Graph traversals, relationship-centric analysis, PQG work | +| [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20.7 M | General entity queries, UI filtering, description text | +| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 307 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20.7 M | Geospatial queries with H3 clustering at arbitrary zoom | +| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 860 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 101,387,180 | Graph traversals, relationship-centric analysis, PQG work | ∗ `/current/wide.parquet` is a stable alias that HTTP 302-redirects to the latest dated file (currently @@ -87,7 +87,7 @@ immutable; the alias rotates atomically when we rebuild. Use the alias for interactive work, the dated URL when you want a pinned, reproducible reference. The original [`isamples_202601_wide.parquet`](https://data.isamples.org/isamples_202601_wide.parquet) -(278 MB, no thumbnails) is kept available for historical pinning. +(292 MB, no thumbnails) is kept available for historical pinning. All three represent the same underlying data (SESAR + OpenContext + GEOME + Smithsonian) with identical semantics — they differ only in serialization @@ -98,7 +98,7 @@ for a performance comparison. ### Pre-aggregated helpers {.unnumbered} Small lookup tables computed ahead of time so a page can render facets -and counts instantly, without touching the 278 MB primary file: +and counts instantly, without touching the 292 MB primary file: | File | Size | Contents | Use when… | |---|---:|---|---| diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index 5452f84b..00ec3911 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -18,9 +18,9 @@ The iSamples property graph data can be serialized in three different parquet fo | Format | Description | File Size | Row Count | Sources | |--------|-------------|-----------|-----------|---------| -| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~850 MB | ~106M rows | All 4 sources | -| **Wide** | Stores relationships as `p__*` columns on entity rows | ~278 MB | ~20M rows | All 4 sources | -| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~292 MB | ~20M rows | All 4 sources | +| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~860 MB | 101,387,180 rows | All 4 sources | +| **Wide** | Stores relationships as `p__*` columns on entity rows | ~292 MB | ~20.7M rows | All 4 sources | +| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~307 MB | ~20.7M rows | All 4 sources | All three formats represent the **same underlying data** (SESAR, OpenContext, GEOME, Smithsonian) with identical semantics. The wide format eliminates edge rows; the H3 variant adds pre-computed hexagonal spatial indices for accelerated geospatial queries. @@ -249,7 +249,7 @@ validityCheck ? (validityCheck.error ? html` Wide+H3 samples:${validityCheck.wideH3Samples.toLocaleString()} Sample count match (all 3):${validityCheck.sampleMatch ? '✅ Yes' : '❌ No'} -

The wide schema has ~79% fewer rows because edge rows are eliminated. The H3 variant adds 3 spatial index columns (h3_res4/6/8).

+

The wide schema has ~80% fewer rows because edge rows are eliminated. The H3 variant adds 3 spatial index columns (h3_res4/6/8).

`) : html`

Click "Run All Benchmarks" to check data validity

` ``` @@ -883,7 +883,7 @@ benchmark5 ? (benchmark5.error ? html` -Full scan (280MB) +Full scan (292MB) ${benchmark5.fullScanCold.toFixed(0)} ms ${benchmark5.fullScanMedian.toFixed(0)} ms ${benchmark5.fullScanAll.map(t => t.toFixed(0)).join(', ')} ms @@ -992,7 +992,7 @@ ${allResults.geoBenchmark && !allResults.geoBenchmark.error ? html` ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` -Full scan (280MB wide) +Full scan (292MB wide) ${allResults.facetBenchmark.fullScanMedian.toFixed(0)} baseline @@ -1008,7 +1008,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html`

Key Findings

@@ -1037,7 +1037,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` | **Browser caching** | First run is "cold" (metadata not cached), subsequent runs are "warm" | | **Network variability** | We run 3 iterations and report the warm-run median (exclude cold run) | | **JIT compilation** | First run includes JIT overhead; warm runs are more representative | -| **Memory limits** | 691 MB narrow file may stress browser memory; wide format is safer | +| **Memory limits** | 860 MB narrow file may stress browser memory; wide format is safer | ### Schema Differences @@ -1059,7 +1059,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` {..., h3_res4: 595536348953485311, h3_res6: 604265133842685951, h3_res8: 613003918731886591} ``` -The wide format eliminates ~9M edge rows (60% file size reduction). The H3 variant adds ~14MB for spatial index columns that enable cell-based geospatial filtering. +The wide format eliminates ~81M edge rows (~66% file size reduction: 860 MB → 292 MB). The H3 variant adds ~15MB for spatial index columns that enable cell-based geospatial filtering. **Facet summaries** (2KB) pre-compute common aggregations: ```sql