Skip to content

Correct the narrow-file row count (106 M → 101,387,180) and the figures derived from it #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rdhyee merged 4 commits into from
Aug 27, 2026
Merged

Correct the narrow-file row count (106 M → 101,387,180) and the figures derived from it #349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e8ae4ab
Correct the narrow-file row count: 106 M → 101,387,180
rdhyee Aug 27, 2026
94e0e73
Fix figures derived from the old row count, and put file sizes in dec…
rdhyee Aug 27, 2026
905b1c1
Round 2: remaining stale sizes in the tutorial's results panel; 20M -…
rdhyee Aug 27, 2026
9cd741b
Round 3: last three stale wide-file sizes (278/280 MB -> 292 MB)
rdhyee Aug 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion DATA_PROVENANCE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ STAGE 0/1 export_client → JSONL → GeoParquet
→ isamples_export_*_geo.parquet (Export format; ~300MB, 6.7M; Zenodo doi:10.5281/zenodo.15278211)
▼
STAGE 2 pqg/pqg/sql_converter.py (export → base PQG; 7-stage DuckDB SQL)
→ narrow (…_narrow.parquet, ~844MB, 106M rows) and wide (…_wide.parquet, ~282MB, 20M rows)
→ narrow (…_narrow.parquet, ~860MB, 101,387,180 rows) and wide (…_wide.parquet, ~292MB, 20.7M rows)
▼
STAGE 3 sidecar/enrichment merges (LEFT JOIN by pid) ← Eric's independently-maintained OC PQG (GCS)
3a scripts/enrich_wide_with_oc_thumbnails.py → isamples_202604_wide.parquet (+47K thumbnails)
Expand Down
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,10 +76,10 @@ Updating dependencies using `pip -U <<package name>>` and regenerate `requiremen
All tutorials query parquet files hosted on Cloudflare R2:

```javascript
// Wide format (recommended) - 280 MB, 20M rows
// Wide format (recommended) - 292 MB, 20.7M rows
const WIDE_URL = "https://data.isamples.org/isamples_202601_wide.parquet";

// Narrow format (advanced) - 850 MB, 106M rows
// Narrow format (advanced) - 860 MB, 101,387,180 rows
const NARROW_URL = "https://data.isamples.org/isamples_202512_narrow.parquet";
```

Expand Down
10 changes: 5 additions & 5 deletions how-to-use.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,9 @@ The two main files carrying the sample records themselves:

| File | Size | Shape | Rows | Use when you need… |
|---|---:|---|---:|---|
| [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20 M | General entity queries, UI filtering, description text |
| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 292 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20 M | Geospatial queries with H3 clustering at arbitrary zoom |
| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 820 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 106 M | Graph traversals, relationship-centric analysis, PQG work |
| [`current/wide.parquet`](https://data.isamples.org/current/wide.parquet) ∗ | 292 MB | Wide (one row per entity, nested relationships in `p__*` array columns) | 20.7 M | General entity queries, UI filtering, description text |
| [`isamples_202601_wide_h3.parquet`](https://data.isamples.org/isamples_202601_wide_h3.parquet) | 307 MB | Wide + H3 BIGINT indices (`h3_res4`, `h3_res6`, `h3_res8`) | 20.7 M | Geospatial queries with H3 clustering at arbitrary zoom |
| [`isamples_202512_narrow.parquet`](https://data.isamples.org/isamples_202512_narrow.parquet) | 860 MB | Narrow (graph: nodes + explicit `_edge_` rows, s/p/o/n fields) | 101,387,180 | Graph traversals, relationship-centric analysis, PQG work |

∗ `/current/wide.parquet` is a stable alias that HTTP 302-redirects to the
latest dated file (currently
Expand All @@ -87,7 +87,7 @@ immutable; the alias rotates atomically when we rebuild. Use the alias for
interactive work, the dated URL when you want a pinned, reproducible
reference. The original
[`isamples_202601_wide.parquet`](https://data.isamples.org/isamples_202601_wide.parquet)
(278 MB, no thumbnails) is kept available for historical pinning.
(292 MB, no thumbnails) is kept available for historical pinning.

All three represent the same underlying data (SESAR + OpenContext + GEOME
+ Smithsonian) with identical semantics — they differ only in serialization
Expand All @@ -98,7 +98,7 @@ for a performance comparison.
### Pre-aggregated helpers {.unnumbered}

Small lookup tables computed ahead of time so a page can render facets
and counts instantly, without touching the 278 MB primary file:
and counts instantly, without touching the 292 MB primary file:

| File | Size | Contents | Use when… |
|---|---:|---|---|
Expand Down
18 changes: 9 additions & 9 deletions tutorials/narrow_vs_wide_performance.qmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ The iSamples property graph data can be serialized in three different parquet fo

| Format | Description | File Size | Row Count | Sources |
|--------|-------------|-----------|-----------|---------|
| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~850 MB | ~106M rows | All 4 sources |
| **Wide** | Stores relationships as `p__*` columns on entity rows | ~278 MB | ~20M rows | All 4 sources |
| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~292 MB | ~20M rows | All 4 sources |
| **Narrow** | Stores relationships as separate edge rows (`otype='_edge_'`) | ~860 MB | 101,387,180 rows | All 4 sources |
| **Wide** | Stores relationships as `p__*` columns on entity rows | ~292 MB | ~20.7M rows | All 4 sources |
| **Wide + H3** | Wide format + pre-computed H3 spatial indices (`h3_res4/6/8`) | ~307 MB | ~20.7M rows | All 4 sources |

All three formats represent the **same underlying data** (SESAR, OpenContext, GEOME, Smithsonian) with identical semantics. The wide format eliminates edge rows; the H3 variant adds pre-computed hexagonal spatial indices for accelerated geospatial queries.

Expand Down Expand Up @@ -249,7 +249,7 @@ validityCheck ? (validityCheck.error ? html`
<tr><td><strong>Wide+H3 samples:</strong></td><td>${validityCheck.wideH3Samples.toLocaleString()}</td></tr>
<tr><td><strong>Sample count match (all 3):</strong></td><td>${validityCheck.sampleMatch ? '✅ Yes' : '❌ No'}</td></tr>
</table>
<p><em>The wide schema has ~79% fewer rows because edge rows are eliminated. The H3 variant adds 3 spatial index columns (h3_res4/6/8).</em></p>
<p><em>The wide schema has ~80% fewer rows because edge rows are eliminated. The H3 variant adds 3 spatial index columns (h3_res4/6/8).</em></p>
</div>
`) : html`<p><em>Click "Run All Benchmarks" to check data validity</em></p>`
```
Expand Down Expand Up @@ -883,7 +883,7 @@ benchmark5 ? (benchmark5.error ? html`
</tr>
</thead>
<tbody>
<tr><td style="text-align: left;"><strong>Full scan (280MB)</strong></td>
<tr><td style="text-align: left;"><strong>Full scan (292MB)</strong></td>
<td>${benchmark5.fullScanCold.toFixed(0)} ms</td>
<td>${benchmark5.fullScanMedian.toFixed(0)} ms</td>
<td>${benchmark5.fullScanAll.map(t => t.toFixed(0)).join(', ')} ms</td></tr>
Expand Down Expand Up @@ -992,7 +992,7 @@ ${allResults.geoBenchmark && !allResults.geoBenchmark.error ? html`
<tbody>
${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html`
<tr style="border-bottom: 1px solid #155724;">
<td style="padding: 8px;">Full scan (280MB wide)</td>
<td style="padding: 8px;">Full scan (292MB wide)</td>
<td style="padding: 8px; text-align: center;">${allResults.facetBenchmark.fullScanMedian.toFixed(0)}</td>
<td style="padding: 8px; text-align: center;">baseline</td>
</tr>
Expand All @@ -1008,7 +1008,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html`
<h4>Key Findings</h4>
<ul>
<li><strong>Schema speedup:</strong> ${allResults.avgSchemaSpeedup != null ? `Average ${allResults.avgSchemaSpeedup.toFixed(1)}x faster with wide format` : 'N/A'}</li>
<li><strong>File sizes:</strong> Narrow ~850MB, Wide ~278MB, Wide+H3 ~292MB</li>
<li><strong>File sizes:</strong> Narrow ~860MB, Wide ~292MB, Wide+H3 ~307MB</li>
<li><strong>Geospatial:</strong> H3 cell-based filtering ${allResults.geoBenchmark && !allResults.geoBenchmark.error ? `is ${allResults.geoBenchmark.speedup.toFixed(1)}x vs lat/lon bbox` : 'N/A'}</li>
<li><strong>Facet summary:</strong> Pre-computed 2KB file ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? `is ${allResults.facetBenchmark.speedup.toFixed(0)}x faster than full scan` : 'N/A'}</li>
</ul>
Expand Down Expand Up @@ -1037,7 +1037,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html`
| **Browser caching** | First run is "cold" (metadata not cached), subsequent runs are "warm" |
| **Network variability** | We run 3 iterations and report the warm-run median (exclude cold run) |
| **JIT compilation** | First run includes JIT overhead; warm runs are more representative |
| **Memory limits** | 691 MB narrow file may stress browser memory; wide format is safer |
| **Memory limits** | 860 MB narrow file may stress browser memory; wide format is safer |

### Schema Differences

Expand All @@ -1059,7 +1059,7 @@ ${allResults.facetBenchmark && !allResults.facetBenchmark.error ? html`
{..., h3_res4: 595536348953485311, h3_res6: 604265133842685951, h3_res8: 613003918731886591}
```

The wide format eliminates ~9M edge rows (60% file size reduction). The H3 variant adds ~14MB for spatial index columns that enable cell-based geospatial filtering.
The wide format eliminates ~81M edge rows (~66% file size reduction: 860 MB → 292 MB). The H3 variant adds ~15MB for spatial index columns that enable cell-based geospatial filtering.

**Facet summaries** (2KB) pre-compute common aggregations:
```sql
Expand Down
Loading