From e8ae4abe6f82f1a67b4369a7811f3235cc4e8b5f Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 27 Aug 2026 08:36:42 -0400 Subject: [PATCH 1/4] =?UTF-8?q?Correct=20the=20narrow-file=20row=20count:?= =?UTF-8?q?=20106=20M=20=E2=86=92=20101,387,180?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four public docs said the 202512 narrow parquet has "106 M" rows. The file has 101,387,180 (scripts/isamples_facts.py, verified 2026-07-18 and 2026-08-25 against both the local copy and the R2-served file; the Zenodo staging README/MANIFEST already carry the correct figure). Numbers only; no other wording changed. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01GcdPqoUz6h6iyHeyXe1uni --- DATA_PROVENANCE.md | 2 +- README.md | 2 +- how-to-use.qmd | 2 +- tutorials/narrow_vs_wide_performance.qmd | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 73ecf4ce..ba12400d 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -30,7 +30,7 @@ STAGE 0/1 export_client → JSONL → GeoParquet → isamples_export_*_geo.parquet (Export format; ~300MB, 6.7M; Zenodo doi:10.5281/zenodo.15278211) ▼ STAGE 2 pqg/pqg/sql_converter.py (export → base PQG; 7-stage DuckDB SQL) - → narrow (…_narrow.parquet, ~844MB, 106M rows) and wide (…_wide.parquet, ~282MB, 20M rows) + → narrow (…_narrow.parquet, ~844MB, 101,387,180 rows) and wide (…_wide.parquet, ~282MB, 20M rows) ▼ STAGE 3 sidecar/enrichment merges (LEFT JOIN by pid) ← Eric's independently-maintained OC PQG (GCS) 3a scripts/enrich_wide_with_oc_thumbnails.py → isamples_202604_wide.parquet (+47K thumbnails) diff --git a/README.md b/README.md index 8b472dc7..bc863084 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ All tutorials query parquet files hosted on Cloudflare R2: // Wide format (recommended) - 280 MB, 20M rows const WIDE_URL = "https://data.isamples.org/isamples_202601_wide.parquet"; -// Narrow format (advanced) - 850 MB, 106M rows +// Narrow format (advanced) - 850 MB, 101,387,180 rows const NARROW_URL = "https://data.isamples.org/isamples_202512_narrow.parquet"; ``` diff --git a/how-to-use.qmd b/how-to-use.qmd index 84859d69..8dd8d902 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -77,7 +77,7 @@ The two main files carrying the sample records themselves: |---|---:|---|---:|---| | [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20 M | General entity queries, UI filtering, description text | | [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 292 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20 M | Geospatial queries with H3 clustering at arbitrary zoom | -| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 820 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 106 M | Graph traversals, relationship-centric analysis, PQG work | +| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 820 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 101,387,180 | Graph traversals, relationship-centric analysis, PQG work | ∗ `/current/wide.parquet` is a stable alias that HTTP 302-redirects to the latest dated file (currently diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index 5452f84b..ee97369a 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -18,7 +18,7 @@ The iSamples property graph data can be serialized in three different parquet fo | Format | Description | File Size | Row Count | Sources | |--------|-------------|-----------|-----------|---------| -| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~850 MB | ~106M rows | All 4 sources | +| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~850 MB | 101,387,180 rows | All 4 sources | | **Wide** | Stores relationships as `p__*` columns on entity rows | ~278 MB | ~20M rows | All 4 sources | | **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~292 MB | ~20M rows | All 4 sources | From 94e0e737239830bc233f4dec2ba722faeee02ee2 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 27 Aug 2026 12:13:28 -0400 Subject: [PATCH 2/4] Fix figures derived from the old row count, and put file sizes in decimal MB Codex review of #349 flagged numbers that were computed from '106 M' or from an older, smaller narrow file, plus sizes that were MiB labelled as MB. Live sizes (Content-Length, 2026-08-27): narrow 860,063,694 B; wide 291,818,634 B; wide_h3 306,553,295 B. Edge rows = 101,387,180 - 20,729,358 = 80,657,822 (79.6 %). - tutorial: ~9M edge rows -> ~81M; 60% -> ~66% size reduction; ~79% -> ~80% fewer rows; 691 MB -> 860 MB; table sizes 850/278/292 -> 860/292/307 MB - how-to-use: 820 MB -> 860 MB (narrow), 292 -> 307 MB (wide_h3) - README: 280/850 MB -> 292/860 MB - DATA_PROVENANCE: ~844MB/~282MB -> ~860MB/~292MB Co-Authored-By: Claude Fable 5 --- DATA_PROVENANCE.md | 2 +- README.md | 4 ++-- how-to-use.qmd | 4 ++-- tutorials/narrow_vs_wide_performance.qmd | 12 ++++++------ 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index ba12400d..e189a2a7 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -30,7 +30,7 @@ STAGE 0/1 export_client → JSONL → GeoParquet → isamples_export_*_geo.parquet (Export format; ~300MB, 6.7M; Zenodo doi:10.5281/zenodo.15278211) ▼ STAGE 2 pqg/pqg/sql_converter.py (export → base PQG; 7-stage DuckDB SQL) - → narrow (…_narrow.parquet, ~844MB, 101,387,180 rows) and wide (…_wide.parquet, ~282MB, 20M rows) + → narrow (…_narrow.parquet, ~860MB, 101,387,180 rows) and wide (…_wide.parquet, ~292MB, 20M rows) ▼ STAGE 3 sidecar/enrichment merges (LEFT JOIN by pid) ← Eric's independently-maintained OC PQG (GCS) 3a scripts/enrich_wide_with_oc_thumbnails.py → isamples_202604_wide.parquet (+47K thumbnails) diff --git a/README.md b/README.md index bc863084..72074219 100644 --- a/README.md +++ b/README.md @@ -76,10 +76,10 @@ Updating dependencies using `pip -U <>` and regenerate `requiremen All tutorials query parquet files hosted on Cloudflare R2: ```javascript -// Wide format (recommended) - 280 MB, 20M rows +// Wide format (recommended) - 292 MB, 20M rows const WIDE_URL = "https://data.isamples.org/isamples_202601_wide.parquet"; -// Narrow format (advanced) - 850 MB, 101,387,180 rows +// Narrow format (advanced) - 860 MB, 101,387,180 rows const NARROW_URL = "https://data.isamples.org/isamples_202512_narrow.parquet"; ``` diff --git a/how-to-use.qmd b/how-to-use.qmd index 8dd8d902..b515dfce 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -76,8 +76,8 @@ The two main files carrying the sample records themselves: | File | Size | Shape | Rows | Use when you need… | |---|---:|---|---:|---| | [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20 M | General entity queries, UI filtering, description text | -| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 292 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20 M | Geospatial queries with H3 clustering at arbitrary zoom | -| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 820 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 101,387,180 | Graph traversals, relationship-centric analysis, PQG work | +| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 307 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20 M | Geospatial queries with H3 clustering at arbitrary zoom | +| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 860 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 101,387,180 | Graph traversals, relationship-centric analysis, PQG work | ∗ `/current/wide.parquet` is a stable alias that HTTP 302-redirects to the latest dated file (currently diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index ee97369a..1e4ad2dd 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -18,9 +18,9 @@ The iSamples property graph data can be serialized in three different parquet fo | Format | Description | File Size | Row Count | Sources | |--------|-------------|-----------|-----------|---------| -| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~850 MB | 101,387,180 rows | All 4 sources | -| **Wide** | Stores relationships as `p__*` columns on entity rows | ~278 MB | ~20M rows | All 4 sources | -| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~292 MB | ~20M rows | All 4 sources | +| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~860 MB | 101,387,180 rows | All 4 sources | +| **Wide** | Stores relationships as `p__*` columns on entity rows | ~292 MB | ~20M rows | All 4 sources | +| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~307 MB | ~20M rows | All 4 sources | All three formats represent the **same underlying data** (SESAR, OpenContext, GEOME, Smithsonian) with identical semantics. The wide format eliminates edge rows; the H3 variant adds pre-computed hexagonal spatial indices for accelerated geospatial queries. @@ -249,7 +249,7 @@ validityCheck ? (validityCheck.error ? html` Wide+H3 samples:${validityCheck.wideH3Samples.toLocaleString()} Sample count match (all 3):${validityCheck.sampleMatch ? '✅ Yes' : '❌ No'} -

The wide schema has ~79% fewer rows because edge rows are eliminated. The H3 variant adds 3 spatial index columns (h3_res4/6/8).

+

The wide schema has ~80% fewer rows because edge rows are eliminated. The H3 variant adds 3 spatial index columns (h3_res4/6/8).

`) : html`

Click "Run All Benchmarks" to check data validity

` ``` @@ -1037,7 +1037,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` | **Browser caching** | First run is "cold" (metadata not cached), subsequent runs are "warm" | | **Network variability** | We run 3 iterations and report the warm-run median (exclude cold run) | | **JIT compilation** | First run includes JIT overhead; warm runs are more representative | -| **Memory limits** | 691 MB narrow file may stress browser memory; wide format is safer | +| **Memory limits** | 860 MB narrow file may stress browser memory; wide format is safer | ### Schema Differences @@ -1059,7 +1059,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` {..., h3_res4: 595536348953485311, h3_res6: 604265133842685951, h3_res8: 613003918731886591} ``` -The wide format eliminates ~9M edge rows (60% file size reduction). The H3 variant adds ~14MB for spatial index columns that enable cell-based geospatial filtering. +The wide format eliminates ~81M edge rows (~66% file size reduction: 860 MB → 292 MB). The H3 variant adds ~14MB for spatial index columns that enable cell-based geospatial filtering. **Facet summaries** (2KB) pre-compute common aggregations: ```sql From 905b1c1deecb86d1e8b830e90560834071a3daa6 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 27 Aug 2026 12:16:02 -0400 Subject: [PATCH 3/4] Round 2: remaining stale sizes in the tutorial's results panel; 20M -> 20.7M; H3 delta ~15MB Codex round-2 findings on #349. 20,729,358 rows is 20.7 M, not 20 M; wide_h3 - wide = 14,734,661 B = 14.7 MB -> ~15MB. Co-Authored-By: Claude Fable 5 --- DATA_PROVENANCE.md | 2 +- README.md | 2 +- how-to-use.qmd | 4 ++-- tutorials/narrow_vs_wide_performance.qmd | 10 +++++----- 4 files changed, 9 insertions(+), 9 deletions(-) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index e189a2a7..06138335 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -30,7 +30,7 @@ STAGE 0/1 export_client → JSONL → GeoParquet → isamples_export_*_geo.parquet (Export format; ~300MB, 6.7M; Zenodo doi:10.5281/zenodo.15278211) ▼ STAGE 2 pqg/pqg/sql_converter.py (export → base PQG; 7-stage DuckDB SQL) - → narrow (…_narrow.parquet, ~860MB, 101,387,180 rows) and wide (…_wide.parquet, ~292MB, 20M rows) + → narrow (…_narrow.parquet, ~860MB, 101,387,180 rows) and wide (…_wide.parquet, ~292MB, 20.7M rows) ▼ STAGE 3 sidecar/enrichment merges (LEFT JOIN by pid) ← Eric's independently-maintained OC PQG (GCS) 3a scripts/enrich_wide_with_oc_thumbnails.py → isamples_202604_wide.parquet (+47K thumbnails) diff --git a/README.md b/README.md index 72074219..adbda4e9 100644 --- a/README.md +++ b/README.md @@ -76,7 +76,7 @@ Updating dependencies using `pip -U <>` and regenerate `requiremen All tutorials query parquet files hosted on Cloudflare R2: ```javascript -// Wide format (recommended) - 292 MB, 20M rows +// Wide format (recommended) - 292 MB, 20.7M rows const WIDE_URL = "https://data.isamples.org/isamples_202601_wide.parquet"; // Narrow format (advanced) - 860 MB, 101,387,180 rows diff --git a/how-to-use.qmd b/how-to-use.qmd index b515dfce..136f96fb 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -75,8 +75,8 @@ The two main files carrying the sample records themselves: | File | Size | Shape | Rows | Use when you need… | |---|---:|---|---:|---| -| [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20 M | General entity queries, UI filtering, description text | -| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 307 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20 M | Geospatial queries with H3 clustering at arbitrary zoom | +| [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20.7 M | General entity queries, UI filtering, description text | +| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 307 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20.7 M | Geospatial queries with H3 clustering at arbitrary zoom | | [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 860 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 101,387,180 | Graph traversals, relationship-centric analysis, PQG work | ∗ `/current/wide.parquet` is a stable alias that HTTP 302-redirects to the diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index 1e4ad2dd..3da679b7 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -19,8 +19,8 @@ The iSamples property graph data can be serialized in three different parquet fo | Format | Description | File Size | Row Count | Sources | |--------|-------------|-----------|-----------|---------| | **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~860 MB | 101,387,180 rows | All 4 sources | -| **Wide** | Stores relationships as `p__*` columns on entity rows | ~292 MB | ~20M rows | All 4 sources | -| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~307 MB | ~20M rows | All 4 sources | +| **Wide** | Stores relationships as `p__*` columns on entity rows | ~292 MB | ~20.7M rows | All 4 sources | +| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~307 MB | ~20.7M rows | All 4 sources | All three formats represent the **same underlying data** (SESAR, OpenContext, GEOME, Smithsonian) with identical semantics. The wide format eliminates edge rows; the H3 variant adds pre-computed hexagonal spatial indices for accelerated geospatial queries. @@ -992,7 +992,7 @@ ${allResults.geoBenchmark && !allResults.geoBenchmark.error ? html` ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` -Full scan (280MB wide) +Full scan (292MB wide) ${allResults.facetBenchmark.fullScanMedian.toFixed(0)} baseline @@ -1008,7 +1008,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html`

Key Findings

  • Schema speedup: ${allResults.avgSchemaSpeedup != null ? `Average ${allResults.avgSchemaSpeedup.toFixed(1)}x faster with wide format` : 'N/A'}
    • -
  • File sizes: Narrow ~850MB, Wide ~278MB, Wide+H3 ~292MB
    • +
  • File sizes: Narrow ~860MB, Wide ~292MB, Wide+H3 ~307MB
  • Geospatial: H3 cell-based filtering ${allResults.geoBenchmark && !allResults.geoBenchmark.error ? `is ${allResults.geoBenchmark.speedup.toFixed(1)}x vs lat/lon bbox` : 'N/A'}
  • Facet summary: Pre-computed 2KB file ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? `is ${allResults.facetBenchmark.speedup.toFixed(0)}x faster than full scan` : 'N/A'}
@@ -1059,7 +1059,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html` {..., h3_res4: 595536348953485311, h3_res6: 604265133842685951, h3_res8: 613003918731886591} ``` -The wide format eliminates ~81M edge rows (~66% file size reduction: 860 MB → 292 MB). The H3 variant adds ~14MB for spatial index columns that enable cell-based geospatial filtering. +The wide format eliminates ~81M edge rows (~66% file size reduction: 860 MB → 292 MB). The H3 variant adds ~15MB for spatial index columns that enable cell-based geospatial filtering. **Facet summaries** (2KB) pre-compute common aggregations: ```sql From 9cd741bc3e98b6e1a6377dde988965b9348ccfa4 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Thu, 27 Aug 2026 12:17:54 -0400 Subject: [PATCH 4/4] Round 3: last three stale wide-file sizes (278/280 MB -> 292 MB) Co-Authored-By: Claude Fable 5 --- how-to-use.qmd | 4 ++-- tutorials/narrow_vs_wide_performance.qmd | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/how-to-use.qmd b/how-to-use.qmd index 136f96fb..5860e302 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -87,7 +87,7 @@ immutable; the alias rotates atomically when we rebuild. Use the alias for interactive work, the dated URL when you want a pinned, reproducible reference. The original [`isamples_202601_wide.parquet`](https://data.isamples.org/isamples_202601_wide.parquet) -(278 MB, no thumbnails) is kept available for historical pinning. +(292 MB, no thumbnails) is kept available for historical pinning. All three represent the same underlying data (SESAR + OpenContext + GEOME + Smithsonian) with identical semantics — they differ only in serialization @@ -98,7 +98,7 @@ for a performance comparison. ### Pre-aggregated helpers {.unnumbered} Small lookup tables computed ahead of time so a page can render facets -and counts instantly, without touching the 278 MB primary file: +and counts instantly, without touching the 292 MB primary file: | File | Size | Contents | Use when… | |---|---:|---|---| diff --git a/tutorials/narrow_vs_wide_performance.qmd b/tutorials/narrow_vs_wide_performance.qmd index 3da679b7..00ec3911 100644 --- a/tutorials/narrow_vs_wide_performance.qmd +++ b/tutorials/narrow_vs_wide_performance.qmd @@ -883,7 +883,7 @@ benchmark5 ? (benchmark5.error ? html` -Full scan (280MB) +Full scan (292MB) ${benchmark5.fullScanCold.toFixed(0)} ms ${benchmark5.fullScanMedian.toFixed(0)} ms ${benchmark5.fullScanAll.map(t => t.toFixed(0)).join(', ')} ms