Repository navigation
Data collection: new records (workstream) #295
Description
Purpose
Tracker for new records: products the dataset does not have yet. It is one of the workstreams split out of the #1 roadmap on 2026-09-27.
PRs that add records (feat(data…), label data without bug) link here with Closes #<this>. The tracker stays open; auto-close is disabled for tracking issues.
Scope
- New CPUs, GPUs, SoCs, smartphones, tablets, watches, PDAs, laptops, monitors
- Benchmark backfill for existing records (filling empty score fields)
- New source integrations (Wikipedia list pages, manufacturer spec pages, CC datasets)
Not in scope: correcting existing values (see the accuracy tracker) and engine/bot changes (see the automation tracker).
Rules
- Every record cites
source_urls. Nothing is invented; if no source states a value, it stays null or the record is skipped. - Bulk imports land as
verified: false. The verify layer promotes them later. - Source policy (SPEC.md): Wikipedia, Wikidata, manufacturer pages and CC datasets are fine. GSMArena and PhoneDB may only be used to correct existing values (ADR-013). Namuwiki is banned. DeviceSpecifications is paused.
- Dedup against existing slugs in both directions and ignoring punctuation (e.g.
geforce3-ti500=geforce3-ti-500).
Lanes and status (2026-09-27)
| Lane | Status | Notes |
|---|---|---|
| Wikipedia GPU list pages | 7 Nvidia additions pending (#327); otherwise exhausted for complete rows | Rechecked 2026-09-29: AMD 0, Intel 0, Nvidia 7 genuinely new complete models vs 813/343/425 reported gaps; remaining entries are existing aliases, incomplete, out of scope, or non-model cells |
| Wikipedia CPU lists (AMD EPYC/Ryzen, Intel sub-pages) | Mostly done | About 390 older Xeon/Celeron rows lack a thread count; 83 were filled from Intel spec pages |
| Wikipedia phone/tablet/watch categories + manufacturer phone lists | Nearly exhausted for complete rows; 1 Apple addition pending (#328) | Manufacturer lists rechecked 2026-09-29: Apple 1, Google 0, Samsung 0 complete new models vs 231/8/109 reported gaps; four absent Pixel 11 models need a Tensor G6 SoC FK (and missing source fields), Samsung timeline lacks required specs; earlier category gaps include Xring O3/Exynos 1610 and Wh-only batteries |
| Apple M-series, MacBooks, Surface laptops | Done | M1-M6, 31 MacBooks, 13 Surface |
| Benchmarks (topcpu, SPEC CPU2006, Blender, PassMark, Time Spy) | Done for current records | Re-run after large imports; DX12 guard needed |
Reporting
One comment per batch: how many records were added, the sources, and the remaining gaps. The Development links list the individual PRs; per-PR status lives on Project #2.
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
feat(ingest): fill absent software and website fields from Wikidata
fix(ingest): annotate compact entity cache payload
fix(ingest): retry truncated Wikidata entity batches
feat(data): fill missing software and website fields from their cited…
Seungpyo1007 commentedon Sep 29, 2026
GPU Wikipedia coverage rechecked live on 2026-09-29 against TechAPI
01d65657e026.gpu/ati)The previous Exhausted claim was slightly overstated: seven Nvidia model names were absent. The other 1,574 reported gaps (99.6%) are not actionable complete additions. The AMD and Intel raw counts reproduce the report exactly; the two Nvidia pages union to the same 1,095 upstream entries and 425 exact-slug gaps.
Reconciliation expanded rowspan/colspan grids, used actual model columns rather than each row's first physical cell, checked names and slugs ignoring punctuation, compared both substring directions, included ATI records, and manually reviewed ambiguous matches. A shared prefix was reviewed rather than blindly treated as identity: MX 440 SE and 8x are distinct from MX 440, FX 5700 LE is distinct from FX 5700, plain FX 5900 differs from Ultra/XT, and plain/XT 6800 differ from the existing GT/GS/Ultra entries.
Added GeForce4 MX 420, MX 440 SE, MX 440 8x, GeForce FX 5700 LE, FX 5900, GeForce 6800, and GeForce 6800 XT in PR #327. Their names span 17 complete table configuration rows; the PR adds one explicitly documented configuration per model without mixing variants. All records cite the Nvidia Wikipedia list and use
verified: false; no missing bus width, TDP, or benchmark scores were guessed. The final commit refreshes the GPU static API dump (2,083 GPUs, 1,867 scored).Hand-checked false-positive examples:
radeon-x1550-agp/radeon-x1550-pci; HD 2350 →radeon-hd-2350-pro; embedded E4690/E6760 → existing MXM/PCIe entries.geforce-fx-5700-engineering-sample; MX 4000 →geforce4-mx-4000-rev-2; GTX 1060/1650 Ti and RTX 2050 mobile rows → existing mobile records.t500-mobile; RTX PRO 6000 Blackwell Max-Q Workstation Edition →rtx-pro-6000-blackwell-max-q.arc-a770(16GB); A60M →arc-pro-a60m. Intel CPU/chipset names and integrated graphics are not new discrete GPUs.1024:64:16, dates, memory sizes, and8 bitare non-model cells exposed by the coverage parser's first-cell heuristic.Remaining source-only rows are incomplete or out of scope. For example, Intel Arc Pro B70 has a combined clock cell without clearly identified required base and boost values, and many older mobile cards lack TDP; these need better sources, not inferred fields. No additional complete distinct models remain identified by this audit after #327's seven additions.
Sources: AMD GPU list, Intel GPU list, Nvidia GPU list, Quadro.
Validation:
python -m app.validatepassed; all seven records match their selected source configurations; GPU slugs are unique; generated detail fields and manifest/index/seed counts agree; new benchmark scores remain null;git diff --checkpassed.Seungpyo1007 commentedon Sep 29, 2026
Rechecked the three manufacturer Wikipedia smartphone pages live on 2026-09-29 against TechAPI
01d65657e026. The original missing counts reproduce exactly, but only one complete new model is importable.Added iPhone Duo in PR #328. Its comparison column supplies A20 Pro, 12 GB RAM, 256/512/1024/2048 GB storage, 5400 mAh, 254 g, and iOS 27.1. The stated October 23, 2026 release date is future and preserved exactly;
verified: false, existingapple-a20-pro, and a single base-model record with storage options. No unsourced fields were invented. Apple has 55 distinct comparison-column models: 54 were already present. Of its 231 raw gaps, 208 are non-model cells and 22 are already-covered grouped/generation aliases.Google has 26 table rows representing 38 model identities: 34 exist and Pixel 11, Pixel 11 Pro, Pixel 11 Pro XL, Pixel 11 Pro Fold are absent. All four specify Tensor G6, which has no curated SoC record, so the SoC foreign key blocks all four. The page also lacks clear weights and contains a Fold-model typo; these are not otherwise fully complete rows. The eight reported gaps consist of three headers, two existing slash groups, and three labels representing these four models.
Samsung's 109 reported gaps are existing names/families or non-model/out-of-scope cells. Its entire 176-row timeline was separately checked across every model column. It supplies model names and month/year, but no SoC or required hardware specifications, yielding zero complete additions. Unmatched source-only names such as C8/C7 (2017)/J7+, Trend 2 Lite and Z Flip 8 need better sources; redlinks/unsubstantiated labels such as A5+, F05 5G, J3 Duo, J4 Prime 2, J4 Pro, On2 and On3 were not forced into the dataset. This does not claim Samsung's whole catalog is exhausted.
Reconciliation expanded row/column spans, read actual model columns, expanded slash groups, searched names/slugs/base-model variants in both directions ignoring punctuation, and separated storage/color/carrier markers from distinct editions. Hand-checked examples: SE generations and separate Pro Max/Pro XL models already exist; Beam i8530 matches I8530 Galaxy Beam, Ace 2 x matches Ace II X S7560M, Note Fan Edition matches Note FE, Rugby Smart matches Rugby Smart I847, and Z Fold SE matches Z Fold Special Edition. Shorter siblings were reviewed rather than accepted as identity. RAM sizes, SoC names, years, months and family headings are not new phones.
Sources: iPhone models, Pixel, Samsung Galaxy list. Live revisions: 1377138205 / 1376853504 / 1376180026.
Validation passed:
python -m app.validate, source-field/FK/base-model checks, unique dump slugs, generated detail/score consistency, index/manifest counts, andgit diff --check. The final PR commit refreshes the smartphone dump through TechEngine serializers to 93,408 smartphones / 93,408 scored; existing detail pages remain unchanged, and Duo's performance score remains null. The phone status line is updated to distinguish this specific list-page audit from the earlier category-page work.data(brand): add 1881 device makers seen in Google Play's device list
data(device_catalog): add 2885 Play-listed devices for brands 1-d
data(device_catalog): add 2802 Play-listed devices for brands e-k
data(device_catalog): add 2897 Play-listed devices for brands l-q
data(device_catalog): add 2968 Play-listed devices for brands r-y
data(device_catalog): add 231 Play-listed devices for brands z
data(brand): add 1881 device makers seen in Google Play's device list
data(device_catalog): add 2885 Play-listed devices for brands 1-d
data(device_catalog): add 2802 Play-listed devices for brands e-k
data(device_catalog): add 2897 Play-listed devices for brands l-q
data(device_catalog): add 2968 Play-listed devices for brands r-y
data(device_catalog): add 231 Play-listed devices for brands z
Seungpyo1007 commentedon Oct 6, 2026
Google Play expansion merged (#360): 11,783
device_catalogentries (6,517 phones / 5,223 tablets / 43 watches) and 1,881 new device makers underbrand/unknown/. Specs come from the Play Console catalog (RAM/SoC/GPU/screen/SDK); there are no release dates by design (ADR-016). Major-brand new devices are held for review.data(brand): add 934 device makers seen in Google Play's device list
data(device_catalog): add 2840 Play-listed devices for brands 1-g
data(device_catalog): add 2878 Play-listed devices for brands h-o
data(device_catalog): add 2568 Play-listed devices for brands p-z
Seungpyo1007 commentedon Oct 6, 2026
Google Play hold-queue round merged (#362): 8,286
device_catalogentries + 934 makers, allverified: false, gated by TechEngineapp.verify candidates(#123). These entries are below the 95% bar (blind audits 72–86%) and were included deliberately because Play is an authoritative device listing; major-brand rows without a Console entry (43%) are excluded.data(device_catalog): add 297 Play-listed devices for brands a-z
Seungpyo1007 commentedon Oct 6, 2026
Major-brand rows without a Play Console entry merged as well (#363): 297
device_catalogentries,verified: false. Of the 321 gate accepts, 24 TV/box/player/camera or carrier-branded rows were removed. The blind audit put this pool at 43%, so expect carrier/regional variants of existing devices in it.