Skip to content

Correct the narrow-file row count (106 M → 101,387,180) and the figures derived from it - #349

Merged
rdhyee merged 4 commits into
isamplesorg:mainfrom
rdhyee:docs/narrow-row-count
Aug 27, 2026
Merged

rdhyee merged 4 commits into
isamplesorg:mainfrom
rdhyee:docs/narrow-row-count

Conversation

@rdhyee

@rdhyee rdhyee commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: fix a known stale figure in the public docs during the 2026-08 solidify pass, and get this PR to a CC×Codex LGTM before a human merges. What I did: replaced "106 M" / "~106M rows" with the verified count 101,387,180 in the four places it appears (commit 1); then, after an adversarial Codex review, fixed the figures in the same four files that had been derived from the wrong count or from an older, smaller file, and standardized their sizes to decimal MB (commit 2).

Evidence

  • isamples_202512_narrow.parquet has 101,387,180 rows — scripts/isamples_facts.py (2026-07-18, 2026-08-25) and a live DuckDB count(*) against data.isamples.org on 2026-08-27.
  • Live sizes (Content-Length, 2026-08-27): narrow 860,063,694 B; wide 291,818,634 B; wide_h3 306,553,295 B. The old "292 MB / 820 MB" figures were MiB labelled as MB; "691 MB" and "~9M edge rows" were from an earlier file.
  • Edge rows = 101,387,180 − 20,729,358 = 80,657,822 → wide has ~80 % fewer rows; wide is ~66 % smaller than narrow (860 → 292 MB).

Files

  • how-to-use.qmd (file table: rows, sizes)
  • README.md (JavaScript snippet comments)
  • tutorials/narrow_vs_wide_performance.qmd (format table; "~79 %" → ~80 %; "~9M edge rows (60 %)" → ~81M (~66 %); "691 MB" → 860 MB)
  • DATA_PROVENANCE.md (pipeline diagram)

Numbers only; no prose or code changes. Draft, for a human to merge.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QJzwMsuFoSgTxX2p9paToy

rdhyee and others added 2 commits August 27, 2026 08:36
Four public docs said the 202512 narrow parquet has "106 M" rows. The
file has 101,387,180 (scripts/isamples_facts.py, verified 2026-07-18 and
2026-08-25 against both the local copy and the R2-served file; the Zenodo
staging README/MANIFEST already carry the correct figure). Numbers only;
no other wording changed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GcdPqoUz6h6iyHeyXe1uni
…imal MB

Codex review of isamplesorg#349 flagged numbers that were computed from '106 M' or from an
older, smaller narrow file, plus sizes that were MiB labelled as MB.
Live sizes (Content-Length, 2026-08-27): narrow 860,063,694 B; wide 291,818,634 B;
wide_h3 306,553,295 B. Edge rows = 101,387,180 - 20,729,358 = 80,657,822 (79.6 %).

- tutorial: ~9M edge rows -> ~81M; 60% -> ~66% size reduction; ~79% -> ~80% fewer rows;
  691 MB -> 860 MB; table sizes 850/278/292 -> 860/292/307 MB
- how-to-use: 820 MB -> 860 MB (narrow), 292 -> 307 MB (wide_h3)
- README: 280/850 MB -> 292/860 MB
- DATA_PROVENANCE: ~844MB/~282MB -> ~860MB/~292MB

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rdhyee rdhyee changed the title Correct the narrow-file row count: 106 M → 101,387,180 Correct the narrow-file row count (106 M → 101,387,180) and the figures derived from it Aug 27, 2026
rdhyee and others added 2 commits August 27, 2026 12:16
…> 20.7M; H3 delta ~15MB

Codex round-2 findings on isamplesorg#349. 20,729,358 rows is 20.7 M, not 20 M; wide_h3 - wide =
14,734,661 B = 14.7 MB -> ~15MB.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rdhyee

rdhyee commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: have Codex adversarially review this PR and iterate until both reviewers say LGTM before he merges. What I did: four Codex rounds (gpt-5.4 default model, reviewing a clean clone of this branch), fixing what each round found and pushing; then applied the CC+Codex+LGTM label and marked the PR ready for review.

Codex verdict at 9cd741b — LGTM

  1. All four files: figures match established row/byte facts.
  2. tutorials/narrow_vs_wide_performance.qmd:252,1062: arithmetic verified — 79.55 % → ~80 %; 66.07 % → ~66 %; 80,657,822 → ~81M; H3 adds 14.73 MB → ~15MB.
  3. Table pipe counts consistent. Diff contains only intended numeric corrections; git diff --check clean.

What the rounds caught (all fixed): figures derived from the old 106 M (~9M edge rows, 60 %, ~79 %); sizes that were MiB labelled as MB (292/820 → 307/860 MB) or from an older file (691 MB, 278/280 MB); 20M for 20,729,358 rows → 20.7M; H3 delta 14 → 15 MB. Facts used: live DuckDB count(*) = 101,387,180 and live Content-Length for the three files, both checked 2026-08-27.

Round transcripts are local (/tmp/pr349_codex_r{2,3,4,5}.txt); nothing else was posted.

@rdhyee rdhyee added the CC+Codex+LGTM Claude and Codex both reached LGTM; ready for human review label Aug 27, 2026
@rdhyee
rdhyee marked this pull request as ready for review August 27, 2026 16:19
@rdhyee
rdhyee merged commit 372dafb into isamplesorg:main Aug 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CC+Codex+LGTM Claude and Codex both reached LGTM; ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant