Skip to content

Data accuracy: corrections, duplicates and dates (workstream) #296

Open

Description

@Seungpyo1007

Purpose

Tracker for data accuracy: correcting existing values, merging duplicates, fixing dates and links. It is one of the workstreams split out of the #1 roadmap on 2026-09-27.

PRs that fix data (fix(data…), or label data + bug) link here with Closes #<this>. The tracker stays open.

Scope

  • Cross-checks against Wikipedia and manufacturer spec pages (CPU, GPU, SoC, watch)
  • Plausibility outliers (storage recorded as RAM, unit slips, impossible values)
  • Duplicate or near-duplicate records (merging while keeping the better record)
  • Placeholder dates (YYYY-01-01) and placeholder SoC links
  • Removing values that turn out to be estimates (e.g. DX12 benchmarks on pre-DX12 GPUs)

Rules

  • Only change a value when a source unambiguously gives the right one for that exact SKU or variant. If sources conflict, document the conflict instead of picking a side.
  • Add the evidence URL to source_urls; keep verified as it is.
  • Each fix PR lists its changed fields and the unresolved items in the body.

Status (2026-09-27)

Area Done Open
Storage recorded as RAM #257, #279, #281, #293 (~100 records) 5 devices with no source stating RAM
SoC release dates #287, #289 (182 records) Kirin tables have no dates; coarse same-year dates
GPU cross-check #260, #290 49 conflicts reviewed; no supportable fix
Intel CPU cross-check #263, #266 (150 records) 45 ambiguous slugs
Duplicates #280, #282 (17 GPU pairs) Near-duplicates in other categories not yet scanned
Placeholder SoC links #271 (53 tablet "0" values) 7,077 phone references to 51 generic SoC slugs; needs an external per-device source

Reporting

One comment per batch: what was fixed, with counts and examples, and what stays unresolved.

Activity

  1. Seungpyo1007 commented on Sep 28, 2026

    @Seungpyo1007
    MemberAuthor

    tablet/watch near-duplicate scan

    Summary: Scanned 3,459 tablet records across 63 brands and 447 watch records across 13 brands using normalized slugs/names and spec cross-checks (release date, SoC, display size/resolution, RAM/storage). Found 192 confirmed duplicate pairs/groups and 38 possible pairs needing human/source verification, plus identified several distinct multi-year hardware refreshes that should not be merged.


    1. Watch Findings

    Confirmed Duplicates (30 pairs / groups)

    1. Identical GSMArena scrape duplicates (3 pairs): Scraped twice from cigarplug/scrape-gsma under different row IDs with 100% identical specs:

      • huawei/2019/watch-gt-2/: huawei-watch-gt-2-scrapegsma-103 vs huawei-watch-gt-2-scrapegsma-986
      • huawei/2020/watch-gt-2e/: huawei-watch-gt-2e-scrapegsma-77 vs huawei-watch-gt-2e-scrapegsma-960
      • lg/2015/watch-urbane-lte/: lg-watch-urbane-lte-scrapegsma-1082 vs lg-watch-urbane-lte-scrapegsma-199
    2. Cross-directory base model duplicates (26 pairs): Same underlying device imported into sibling directories due to brand-prefixed directory names (one from sarangzambare/smartdevices_pricing, one from msainani/gsmarena-mobile-devices):

      • Apple (17 pairs):
        • 2014: apple-watch-38mm-1st-gen vs watch-38mm-1st-gen, apple-watch-42mm-1st-gen vs watch-42mm-1st-gen, apple-watch-edition-38mm-1st-gen vs watch-edition-38mm-1st-gen, apple-watch-edition-42mm-1st-gen vs watch-edition-42mm-1st-gen, apple-watch-sport-38mm-1st-gen vs watch-sport-38mm-1st-gen, apple-watch-sport-42mm-1st-gen vs watch-sport-42mm-1st-gen
        • 2016: apple-watch-series-1-aluminum-38mm vs watch-series-1-aluminum-38mm, 42mm vs 42mm, apple-watch-series-2-38mm vs watch-series-2-38mm, 42mm vs 42mm, apple-watch-series-2-aluminum-38mm vs watch-series-2-aluminum-38mm, 42mm vs 42mm, apple-watch-edition-series-2-38mm vs watch-edition-series-2-38mm, 42mm vs 42mm
        • 2017: apple-watch-series-3 vs watch-series-3, apple-watch-series-3-aluminum vs watch-series-3-aluminum, apple-watch-edition-series-3 vs watch-edition-series-3
      • Asus (1 pair): 2016 asus-zenwatch-3-wi503q vs zenwatch-3-wi503q
      • Huawei (3 pairs): 2018 huawei-watch-2-2018 vs watch-2-2018, huawei-watch-gt vs watch-gt, huawei-watch-magic vs watch-magic
      • LG (2 pairs): 2017 lg-watch-style vs watch-style, 2018 lg-watch-w7 vs watch-w7
      • Samsung (3 pairs): 2018 samsung-galaxy-watch vs galaxy-watch, 2016 samsung-gear-s3-classic vs gear-s3-classic, 2016 samsung-gear-s3-frontier vs gear-s3-frontier
    3. Intra-directory brand prefix duplicate (1 pair):

      • intex/2015/irist-smartwatch/: intex-irist-smartwatch vs irist-smartwatch

    Possible Duplicates / Needs Human Check (5 pairs)

    • lg/watch-urbane-2nd-edition-lte: 2015 vs 2016 year split across sources (lg-watch-urbane-2nd-edition-lte-0-75gb-4gb-4g-lte vs lg-watch-urbane-2nd-edition-lte-aitoolbuzz-5427-0-75gb-4gb-4g-lte).
    • samsung/galaxy-watch: 2018 42mm bluetooth record vs base record with slightly differing listed screen diameter (1.2" vs 1.3").
    • samsung/gear-s2-classic: 2015 Wi-Fi vs 2016 3G release year split.
    • samsung/gear-s3-classic: 2016 Wi-Fi vs 2017 LTE release year split.
    • samsung/gear-s3-classic-lte: duplicate across gear-s3-classic-lte and samsung-gear-s3-classic-lte directories.

    2. Tablet Findings

    Confirmed Duplicates (162 pairs / groups)

    1. Identical GSMArena scrape duplicates (15 pairs):

      • hp/2013/slate8-pro/: hp-slate8-pro-scrapegsma-52 vs hp-slate8-pro-scrapegsma-935
      • huawei/2013/mediapad-7-youth/: huawei-mediapad-7-youth-scrapegsma-1002 vs ...-119
      • huawei/2019/matepad-pro/: huawei-matepad-pro-scrapegsma-93 vs ...-976
      • huawei/2019/mediapad-m6-10-8/: huawei-mediapad-m6-10-8-scrapegsma-107 vs ...-990
      • huawei/2019/mediapad-m6-8-4/: huawei-mediapad-m6-8-4-scrapegsma-109 vs ...-992
      • huawei/2019/mediapad-m6-turbo-8-4/: huawei-mediapad-m6-turbo-8-4-scrapegsma-108 vs ...-991
      • huawei/2020/matepad/: huawei-matepad-scrapegsma-76 vs ...-959
      • huawei/2020/matepad-pro-5g/: huawei-matepad-pro-5g-scrapegsma-83 vs ...-966
      • huawei/2020/matepad-t8/: huawei-matepad-t8-scrapegsma-68 vs ...-951
      • lenovo/2016/yoga-tab-3-10/: lenovo-yoga-tab-3-10-scrapegsma-1054 vs ...-171
      • Additional pairs in Lenovo, LG, and Samsung tablets.
    2. Cross-directory base model duplicates (50 pairs):

      • Acer (6 pairs): 2018 acer-chromebook-tab-10 vs chromebook-tab-10, 2015 acer-iconia-tab-10-a3-a30 vs iconia-tab-10-a3-a30, 2014 acer-iconia-tab-7-a1-713 vs iconia-tab-7-a1-713, a1-713hd, a3-a20, a3-a20fhd
      • Apple (7 pairs): 2017 apple-ipad-9-7-2017 vs ipad-9-7-2017, apple-ipad-pro-10-5-2017 vs ipad-pro-10-5-2017, 2018 apple-ipad-pro-11 vs ipad-pro-11, 2015 apple-ipad-pro-12-9-2015 vs ipad-pro-12-9-2015, 2017 apple-ipad-pro-12-9-2017 vs ipad-pro-12-9-2017, 2018 apple-ipad-pro-12-9-2018 vs ipad-pro-12-9-2018, 2016 apple-ipad-pro-9-7-2016 vs ipad-pro-9-7-2016
      • HP (3 pairs): 2015 hp-pro-slate-10-ee-g1 vs pro-slate-10-ee-g1, hp-pro-slate-8 vs pro-slate-8, 2013 hp-slate10-hd vs slate10-hd
      • Huawei (2 pairs): 2018 huawei-mediapad-m5-lite vs mediapad-m5-lite, huawei-mediapad-t5 vs mediapad-t5
      • Lenovo (3 pairs): 2017 lenovo-tab-7 vs tab-7, lenovo-tab-7-essential vs tab-7-essential, 2016 lenovo-yoga-tab-3-plus vs yoga-tab-3-plus
      • Prestigio (4 pairs): 2013 prestigio-multipad-10-1-ultimate vs multipad-10-1-ultimate, multipad-4-quantum-10-1, 9-7, 2014 multipad-4-quantum-9-7-colombia
      • Samsung (4 pairs): 2017 samsung-galaxy-tab-a-8-0-2017 vs galaxy-tab-a-8-0-2017, 2018 samsung-galaxy-tab-a-10-5 vs galaxy-tab-a-10-5, 2018 samsung-galaxy-tab-s4-10-5 vs galaxy-tab-s4-10-5, 2017 galaxy-tab-active-2 vs galaxy-tab-active2 vs samsung-galaxy-tab-active-2 (triplet)
      • Vodafone (1 pair): 2017 vodafone-smart-tab-n8 vs smart-tab-n8
      • Other brands: Amazon Fire (2015/2017 Fire 7, HD 8, HD 10), Allview Viva, etc.
    3. Intra-directory brand prefix duplicates (97 pairs):

      • Same directory containing <brand>-<model>.json (from arwinneil) and <model>.json (from msainani) with identical release year, display size, SoC, and memory.
      • Examples: acer-iconia-tab-a100 vs iconia-tab-a100, apple-ipad-air vs ipad-air, apple-ipad-mini-2 vs ipad-mini-2, apple-ipad-air-2 vs ipad-air-2, amazon-kindle-fire vs kindle-fire, asus-google-nexus-7 vs google-nexus-7, hp-slate7-extreme vs slate7-extreme.

    Possible Duplicates / Needs Human Check (33 pairs)

    • RAM tier mismatch on same device (20 pairs): Same device model where one source recorded the base RAM tier and another recorded the higher tier:
      • Huawei: huawei-mediapad-m2-8-0 (2GB vs 3GB), x2 (2GB vs 3GB), m2-10-0 (2GB vs 3GB), t2-10-0-pro (2GB vs 3GB), t2-7-0 (1GB vs 2GB), m3-lite-10 (3GB vs 4GB), t3-10 (2GB vs 3GB), etc.
      • Lenovo: lenovo-tab-4-10-plus (3GB vs 4GB), tab-4-8-plus (3GB vs 4GB)
      • Samsung: samsung-galaxy-tab-3-8-0 (1.5GB vs 1GB), tab-4-7-0 (1.5GB vs 1GB), tab-active (1.5GB vs 1GB), tab-a-8-0 2015 (1.5GB vs 2GB), tab-e-9-6 (1.5GB vs 1GB), tab-e-8-0 (1.5GB vs 1GB), tab-j (1.5GB vs 1GB)
    • 1-year source date discrepancies (13 pairs): Same model specs but release dates span year boundary:
      • alcatel/one-touch-evo-8hd (2013 vs 2014, Rockchip RK3066)
      • alcatel/pixi-3-3g (2014 vs 2015, MediaTek MT8312)
      • blu/touch-book-m7 vs touchbook-m7 (2016 vs 2017, Intel Atom x3)
      • htc/shift (2007 vs 2008, Intel Stealey 800MHz)
      • huawei/mediapad-7-vogue (HiSilicon K3V2 Hi3620 vs Huawei K3V2, 7.01" vs 7.0")
      • microsoft/surface-duo: dual directories microsoft-surface-duo/ and surface-duo/ for global/US 128GB/256GB SKUs

    Confirmed False Positives (Do Not Merge)

    • Legitimate annual revisions / refreshes with different silicon:
      • Samsung Galaxy Tab 4 10.1 & 8.0: 2014 (Snapdragon 400) vs 2015 refresh (Snapdragon 410 64-bit).
      • Samsung Galaxy Tab S6 Lite: 2020 vs 2022 vs 2024 editions (Exynos 9611 vs Snapdragon 720G vs Exynos 1280).
      • Wi-Fi only vs LTE hardware models having distinct chassis weights and antennas.

    Total Candidate Near-Duplicates: 192 confirmed duplicate pairs/groups, 38 possible pairs needing human verification.

  2. added 2 commits that reference this issue on Sep 28, 2026
    65d7401
    3d6289c
  3. Seungpyo1007 commented on Sep 28, 2026

    @Seungpyo1007
    MemberAuthor

    Intel CPU ambiguous slugs (#323)

    Resolved 30 of the 45 slugs #266 skipped. The change is the field the matching Wikipedia rows agree on, or the row selected by socket, segment, clock, or TDP. Fifteen stay unresolved: package rows that still disagree and the record cannot tell them apart, month phrases that are not a day ("Mid 2008", "Q2 2009"), and slugs with no record (celeron-d-320, celeron-d-325, celeron-d-335, core-m3, core-ultra-x9, mobile-pentium-mmx-166).

    Examples: Core 2 Duo T-series release dates moved back to the 2006–2008 Merom/Penryn months both package rows list; E8335 base clock 2.66 → 2.93 GHz; Celeron 2.00 GHz desktop date 2009-02-01 → 2002-09-18; Xeon Gold 5215 date 2019-12-01 → 2019-04-02.

  4. Seungpyo1007 commented on Sep 28, 2026

    @Seungpyo1007
    MemberAuthor

    2026-09-29 batch: tablet/watch duplicate merge + placeholder-data root cause

    PR #324 merged: 183 duplicate records removed across 182 verified tablet/watch groups (18 identical GSMArena re-imports, 70 cross-directory model duplicates, 95 intra-directory brand-prefix duplicates). Ambiguous cases (RAM-tier variants, year splits, LTE/Wi-Fi distinctions, Surface Duo, MediaPad Vogue, 3 genuine Prestigio revisions, Watch Magic's unsourced OS) were deliberately left unmerged.

    Root cause found along the way: a whole import batch had stamped placeholder values (battery_mah=1500, weight_g=150.0) on 43 watch and 217 tablet records regardless of actual device — not real conflicting data. 170 fields were repaired using same-device sibling evidence; 362 with no confident replacement are now null rather than fake. Also caught a comma-parsing bug: iPad Pro 12.9 (2015) battery was stored as 307 mAh instead of 10,307.

    This required a schema change: TechEngine#109 makes mobile battery_mah/weight_g nullable (previously non-nullable, which is very likely why the placeholder bug existed in the first place - some import had to put a number to satisfy the schema).

    Also this batch: #323 resolved 30 of the 45 ambiguous Intel CPU slugs from #266 by matching each record's socket/clock/segment/TDP against the correct Wikipedia row; 15 remain unresolved.

  5. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Scanned the smartphone category (93,407 records) for near-duplicates, using the
    same method as the tablet/watch pass (#324). Opened #329.

    Merged: 50 confirmed brand-prefix duplicate pairs. These are the cases where a
    device exists under both a bare slug (nord-n100) and a brand-prefixed twin
    (oneplus-nord-n100) and the core specs (release_date / soc / ram / battery /
    weight / os) are identical. Kept the shorter canonical slug, unioned source_urls,
    carried the richer side's fields onto the keeper, deleted the redundant file — same
    convention as the Intel Atom CPU dedup. One real conflict inside an apparent
    duplicate (OnePlus 5T Star Wars main camera 20 MP vs 16 MP) was resolved to 16 MP
    per GSMArena/OnePlus (the 20 MP is the secondary sensor), not blind-merged.

    Bigger finding — a systemic placeholder-value bug, reported not mass-edited.
    487 distinct device names across 624 files share the exact triple
    battery_mah=1500, weight_g=150.0, ram_gb=1, spanning 1994–2020 and even including
    watches/tablets, mostly from the gsmarena-mobile-devices Kaggle import. Same
    fabricated-defaults signature (1500/150) #324 flagged. More triples with the
    150.0 g / 1 GB + generic YYYY-01-01 tell-tale: (2000,150,1)×182,
    (1800,150,1)×92, (1000,150,1)×83, (2500,150,1)×80.

    This is also why 717 of the 767 brand-prefix pairs read as "conflicts" rather than
    duplicates: the bare (Kaggle) side carries these placeholder defaults while the twin
    has real specs. They're very likely the same devices, but merging them correctly
    means repairing the fabricated side against a real source per record — inventing
    values for hundreds of devices, which violates the "never invent values" rule. That
    warrants a dedicated repair pass; I left those 717 pairs untouched and documented
    rather than guessing.

    Both gates pass on the merged tree: app.validate and integrity_check.py --strict
    (0 hard anomalies). Dump refreshed in the same PR.

  6. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Worked through the 717 brand-prefix conflict pairs that #329 left documented but
    unresolved (the pairs that disagreed on soc / release_date / ram / battery / weight /
    os). Opened #330.

    Approach: re-detected the pairs on current develop, then classified every conflict
    field against evidence rather than guessing. The soc disagreement (572 pairs) is
    almost always the same physical chip under a different slug spelling
    (qualcomm-msm7227-snapdragon-s1 vs qualcomm-msm7227), and the other conflicts on
    those pairs are the fabricated bulk-import placeholders (weight_g=150.0,
    battery_mah=1500, generic YYYY-01-01).

    Exact counts (all 717 accounted for):

    • resolved-and-merged: 364 — same-chip pairs sourced Kaggle-GSMArena (bare) vs
      phonedb.net (twin). Kept the GSMArena record (real specs; spot-verified Acer DX650
      = 133 g and CloudMobile S500 = 122 g against GSMArena/imei.org, vs the phonedb
      twins' 150.0 g placeholder), unioned source_urls, deleted the twin. No value
      invented — in every one of these the keeper already held the real value and only
      the deleted twin carried the placeholder. verified was not flipped (the
      phonedb scalars proved unreliable), and release_date was not changed
      (phonedb dates are unreliable too — it lists DX650 as 2009-03 while GSMArena/imei
      say 2009-06 — so importing them would be guessing).

    • confirmed-different-devices-not-duplicates: 31 — pairs whose two sides name
      genuinely different silicon, i.e. different SKUs/devices that merely share a base
      name. Left separate and dropped from the duplicate classification. Examples:
      Huawei P50 Pro (Kirin 9000 and Snapdragon 888 — GSMArena confirms both
      chipset variants shipped), Honor Magic V2/V3, HTC One M9 (SD810) vs One M9+
      (Helio X10), OnePlus Nord CE 2 (Dimensity 900) vs Nord CE 2 Lite (SD695), Sony
      Xperia Z (S4 Pro) vs Z1 (SD800), Motorola razr generations, Oppo Find X8 Pro
      (Dimensity 9400), ZTE Grand Memo (SD800 vs SD600).

    • still-unresolved-no-source: 322 — left untouched rather than guessed:

      • 37 same-chip pairs sourced against vendor/Wikipedia pages instead of phonedb
        (different provenance, want individual checks);
      • 157 with a genuine ram/weight/battery question needing a per-record source;
      • 104 where one side's soc is a vague descriptor (dual-core 1.0 GHz, N/A,
        <brand> mobile platform) — same device, but the date/scalars still need a source;
      • 23 same-silicon-spelling pairs that are mergeable in a follow-up pass;
      • 1 odd pair (blu selfie vs vivo-selfie).
        Plus the release_date field on the 364 merged records, deliberately left as-is.

    Merging the remaining 322 correctly needs per-record web sourcing of hundreds of
    individual devices; per the "never invent a value" rule I've left them documented
    rather than guessed. Both gates pass on #330 (app.validate,
    integrity_check.py --strict → 0 hard anomalies); dump refreshed in the same PR.

  7. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Brand near-duplicate scan, 2026-09-29

    Picking up the tracker's open "near-duplicates in other categories not yet scanned" item, I hand-reviewed the whole brand category — the same duplicate-hunt CPU/tablet/watch/smartphone/laptop/monitor/pda got today. Scope: 207 brand records under data/brand/<country>/<slug>.json, on develop 9ce1979.

    Result: the brand category is already clean — no confirmed duplicates, no merge, no PR. Reporting here rather than forcing a change.

    What I checked

    Brand records are manufacturer entities (name, slug, country, categories, founded_year, website, logo, descriptions), not product specs, so I looked for the three failure modes that apply to entities:

    1. Same company under two slugs (e.g. hp vs hewlett-packard, casing/punctuation variants). Automated check: 0 normalized-name collisions and 0 normalized-slug collisions across all 207 records (normalizing to lowercase alphanumerics). None of the classic synonym pairs are double-listed — hp, google, lg (LG Electronics), samsung, at-t, t-mobile each appear exactly once.
    2. Sub-brand/product-line mis-split from a parent. 11 records carry the sub-brand tag or are sub-brand-like (alcatel, infinix, iqoo, itel, nubia, poco, redmi, tecno, xolo, redmagic, alienware). Each is a legitimately separate entity with correct parent attribution in its description (TCL, Transsion, vivo, ZTE, Xiaomi, Lava, Dell). These are the "Beats-under-Apple" case — kept distinct, not merged.
    3. Placeholder/near-empty stubs duplicating a fuller record. 0 found — no sparse stub shadows a fuller record for the same company.

    Related-but-distinct entities (deliberately NOT merged)

    The only pairs that look superficially like duplicates are joint-venture / brand-licensing lineages, and each side is a genuinely separate entity with its own device fan-out:

    • BenQ (TW, 2001, monitors/projectors) vs BenQ-Siemens (DE, the 2005-2007 JV mobile brand) — distinct.
    • Fujitsu (JP, 1935, cpu-designer) vs Fujitsu Siemens (DE, the 1999-2009 JV that made Pocket LOOX PDAs) vs Siemens (DE, 1847) — three distinct entities.
    • Sony vs Sony Ericsson (the 2001-2012 JV) — distinct.
    • HMD Global vs Nokia (HMD licenses the Nokia brand; separate company) — distinct.
    • HONOR vs Huawei (HONOR independent since its 2020 spin-off) — distinct.

    Near-name prefixes that are unrelated companies: blackberry/blackview, infineon/infinix, microchip/micromax/microsoft, transmeta/transsion. All distinct.

    This is consistent with the recent develop history — the "364 resolved brand-prefix conflicts" work already normalized brand slugs, so the category was in good shape going in.

    Verification

    No data files were changed. Confirmed the current tree is green:

    • python -m app.validate → Data validation passed (exit 0), including brand path/slug convention and the brand foreign-key set used by device records.
    • python integrity_check.py <data> --strict → no hard anomalies (exit 0); the structural section reports no duplicate brand slugs/names, no slug/filename mismatches, and no verified-without-source brands.

    No FK repoints were needed (nothing merged), and no dump refresh is warranted since no data changed.

  8. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Software near-duplicate scan, 2026-09-29

    Continuing the "near-duplicates in other categories not yet scanned" item after the brand pass, I scanned the software category. Scope: 42,493 software records under data/software/<2char>/<slug>.json, on develop ea3f31e.

    Result: no true duplicates — the software category is clean. No merge, no PR. Reporting here, same standard as brand.

    Method (adapted to how this dataset is keyed)

    Software records here are Wikidata-derived, and every one of the 42,493 records carries a Wikidata QID in source_urls (0 without). The QID is the entity identity, and slugs suffixed -qNNNN are deliberate disambiguators for records that share a display name. So "same name, different slug" is expected and is not itself a duplicate — a real duplicate is the same Wikidata entity imported twice, an unkeyed name-and-developer collision, or a content clone. I checked all three:

    1. Same Wikidata QID cited by more than one record — the decisive entity-level check. 0 shared-QID clusters. No entity is imported twice.
    2. Same normalized name + same developer set with no distinguishing QID — 0 clusters.
    3. Byte-identical record bodies (ignoring slug) — 0 clusters. No content clones.

    Name collisions are genuinely distinct entities

    There are 993 normalized-name collisions covering 2,165 records, and all 993 are fully QID-disambiguated (0 ambiguous). Spot-checking the largest:

    • "Mail" (6): Apple Mail (distinct watchOS / iOS / macOS Wikidata items), Microsoft's Windows Mail, a Linux mail client, etc. — different QIDs and developers.
    • "Calculator" (4): Apple's, Google's, and Sabec's calculators — distinct products.
    • "Arc" (4): The Browser Company's Arc browser vs Paul Graham's Arc Lisp dialect vs others.
    • "GEOS" (4): the GeoWorks GEOS vs the Unix-like and other GEOS entities.
    • "Apollo", "Opus", "R", "Spark", "Snap!", "Flow", "PRISM", "Visi On" — same pattern: each is a set of unrelated products that happen to share a name, separated by distinct QIDs.

    These are exactly the "don't merge real distinct entities that happen to be related/similarly named" case.

    The "stub-looking" QID-suffixed records confirm the pattern rather than break it: e.g. zune (Q229851) vs zune-q1970794 (Q1970794), zoa-space (Q118225284) vs zoa-space-q118225291 (Q118225291), and zsnes vs zsnes-2 (ZSNES 2, Q116868010, a distinct 2021 project) all cite different Wikidata items — they are separate entities, not stubs shadowing a fuller record.

    Verification

    No data files changed. Confirmed the current tree is green:

    • python -m app.validate → Data validation passed (exit 0), incl. software required-field / slug-convention / filename-match checks.
    • python integrity_check.py <data> --strict → no hard anomalies (exit 0); the structural section reports no duplicate software slugs, no slug/filename mismatches, and no verified-without-source software records.

    No FK repoints were needed (nothing merged) and no dump refresh is warranted since no data changed.

  9. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Laptop near-duplicate scan is done — #332 (against develop).

    Ran the same method as the tablet/watch (#324) and smartphone (#329) passes on all 1,995 data/laptop/ records: slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date, cpu ref/name, ram_gb, storage_gb, display size/resolution, gpu, weight, os).

    Per-pattern results:

    • Pattern 1 (same scrape imported twice under different row-id slugs): 29 groups merged. Each pair was byte-identical except slug and variant.source_row, all within Ammok/laptop_price_prediction. Kept the lower source_row, deleted the redundant file; source_urls identical so the union was a no-op and no values were changed. Breakdown: acer 1, dell 13, hp 3, lenovo 8, msi 2, toshiba 2. 1995 → 1966 records.
    • Pattern 2 (brand-prefix path variance, e.g. dell-xps-13 vs xps-13): none. All records live under a consistent brand-prefixed base dir.
    • Pattern 3 (cross-dataset dup under different slug conventions): none. No device name appears in both bulk datasets; the datasets cover disjoint models. Identical-coarse-spec collisions between different SKUs (e.g. Acer Aspire ES1-572 vs Extensa EX2540) were correctly left unmerged since their names differ.

    Configuration variants (same model, different CPU/RAM/storage) were left untouched — distinct real SKUs, not duplicates.

    Placeholder / fabricated-defaults check: clean. No equivalent of the phone battery_mah=1500 / weight_g=150.0 stamping. ram_gb, storage_gb, weight_g, and display.size_inch all show organic distributions with distinct-name diversity proportional to frequency (e.g. ram_gb=16 across 735 distinct models, storage_gb=512 across 739). No single value is stamped across unrelated devices.

    app.validate and integrity_check.py --strict both pass on the final state (only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only). Dump refreshed as the PR's last commit, scoped to the laptop collection.

    That closes out the "near-duplicates in other categories not yet scanned" item for laptop — no ambiguous groups were left pending.

  10. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Monitor near-duplicate scan is done — #333 (against develop).

    Continuing the "near-duplicates in other categories not yet scanned" item after the tablet/watch (#324), smartphone (#329) and laptop (#332) passes. Scope: all 882 data/monitor/ records, every one sourced from a single Kaggle Amazon monitor dataset. Same method: slug/name normalization (punctuation- and brand-prefix-agnostic) plus a spec cross-check (release_date, size_inch, resolution, aspect_ratio, refresh_hz, panel_type, ppi, and the model number in name).

    Result: 651 confirmed near-duplicates merged. 882 → 231.

    Per-pattern results:

    • Pattern 1 (same scrape imported twice under different row-id slugs): 650 files merged across 14 base models. Each cluster was byte-identical except slug and variant.source_row. Kept the lowest source_row, deleted the redundant files; source_urls were identical so the union was a no-op and no values changed. Largest clusters: asus-va329he −83, aopen-aopen-by-acer −68, acer-cb272 −62, acer-sh242y −59, acer-xb273k −59, acer-xz342cu −46, acer-kb252q −44, acer-sb240y −43, acer-sb272 −38, acer-xz320qu −30, acer-hdr10 −72, acer-sb270 −26, acer-kc242y −13, acer-xb273u −6.
    • Pattern 2 (brand-prefix path variance, e.g. dell-u2723qe vs u2723qe): none. Every record already lives under a consistent brand-prefixed base dir.
    • Pattern 3 (cross-source dup under a different title convention): 1 merged. asus-va24ehe rows 60 & 404 — the same VA24EHE (23.8", 1080p, 75 Hz, IPS, $99) imported with two Amazon title strings. Kept the cleaner title and adopted the more complete adaptive_sync = "FreeSync" from the other side (a real, documented VA24EHE spec, cited in the PR).

    Configuration variants (same panel, different size/resolution/refresh/SKU) were left untouched — distinct real products, not duplicates. Documented and intentionally left unmerged: acer-eda343cur (100 vs 180 Hz), acer-sb240y (G0bi vs G0wi), acer-sb242y (EBI vs Hbi), acer-sb272 (new vs Renewed), aopen-aopen-by-acer (32SA2QK 4K vs 27E1 FHD), gigabyte-m27q (M27Q vs M27Q PRO), hp-hp-series-monitor (Series 5 24" vs 524sa), lenovo-lenovo-thinkvision-full (M15 vs T22i-30), lg-lg-digital-signage (49" vs 86"), lg-lg-ultragear-gaming (QHD 240 Hz vs 5120×1440 165 Hz), lilliput-fa1014-np (1280×800 non-touch vs 1080p touch), samsung-hdr2000 (Neo G7 vs Neo G8), and the acer-hdr10 residual (its hdr10 slug is an importer fallback shared by a few genuinely different Nitros — left as-is rather than guess).

    Placeholder / fabricated-defaults check: clean. No equivalent of the phone-batch stamping — refresh_hz, ppi, size_inch, msrp_usd all show organic distributions with distinct-name diversity proportional to frequency; no single spec value is stamped across unrelated devices. The one systematic artifact is the placeholder date release_date = "2023-01-01" on all 882 records — a uniform dataset-wide import default, not a spec forced onto unrelated devices, so out of scope for this merge but worth flagging for the "placeholder dates (YYYY-01-01)" item: this whole Amazon-monitor import has no real release dates.

    app.validate and integrity_check.py --strict both pass on the final state (only the pre-existing advisory CPU/GPU ratio outliers print, which the gate treats as review-only). Dump refreshed as the PR's last commit, scoped to the monitor collection — the 651 orphaned detail pages were pruned, the monitors list rebuilt (231), and the manifest count updated; unrelated pre-existing serialization drift in other collections was kept out to keep the diff monitor-scoped.

    That closes out the "near-duplicates in other categories not yet scanned" item for monitor — no ambiguous groups were left pending beyond the documented distinct-SKU cases above.

  11. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Continued the smartphone brand-prefix dedup. #329 (50) and #330 (364) left a
    residual of 353; #334 resolves 170 more with source-verified fixes and un-pairs 3
    genuinely different devices.

    resolved-and-merged: 170

    • 37 vendor/Wikipedia-sourced same-chip pairs — same device, same chip (spelling
      variance); adopted the twin's reliable Wikipedia/vendor release_date over the
      bare YYYY-01-01 placeholder (verified e.g. Huawei P30 Pro 2019-03-26 = GSMArena).
    • 7 same-model pairs whose sides named different chips, resolved against a cited
      source: Oppo Find X8 Pro → Dimensity 9400 (oppo.com), Motorola Edge 50 Ultra →
      Snapdragon 8s Gen 3 (motorola.com), Honor Magic V3 → Snapdragon 8 Gen 3
      (phonearena), plus OnePlus Nord CE 2 Lite / CE 2, Sony Xperia Z, Honor Magic V2
      (bare already correct; GSMArena/vendor cited).
    • 126 same-chip-spelling / vague-soc-vs-specific pairs (Kaggle-GSMArena vs phonedb):
      kept the GSMArena record, adopted the specific soc where the keeper's was vague,
      rejected the phonedb stamped placeholders, deleted the twin.

    confirmed-different-devices-not-duplicates (un-paired): 3 — HTC one-m9 holds
    the One M9+ (Helio X10) vs One M9 (SD810); Motorola razr the 2023 (SD 8+ Gen 1)
    vs 2019 (SD710); Sony xperia-z3 the Z3+ (SD810) vs Z3 (SD801). Left as separate
    records.

    still-unresolved-no-source: 180 — 21 same-model pairs whose correct chip I
    couldn't source this round (includes the Huawei P50 Pro, a genuine dual-chipset
    model — Kirin 9000 and Snapdragon 888 both shipped), 157 pairs needing a per-record
    ram/weight/battery source, and 2 odd/other pairs. Left documented rather than
    guessed.

    Running total across #329/#330/#334: 584 of the original 767 brand-prefix pairs
    resolved (50 + 364 + 170), plus the genuinely-different devices un-paired. Both
    gates pass on #334 (app.validate, integrity_check.py --strict); dump refreshed
    in the same PR. Refs #296 (leaving the tracker open for the residual and the
    other categories' work).

  12. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Scanned the pda category (140 records) for near-duplicates by hand — small
    enough to review every match individually. Opened #335.

    Merged: 28 confirmed duplicate records → 21 keepers (140 → 112). Same three
    patterns as the tablet/watch (#324) and smartphone (#329/#330) passes:

    • Identical scrape imported twice (different row-id slug): 2 pairs — Motorola
      Accompli 388 (scrapegsma-344/-1227) and Mitac MIO A701 (-278/-1161),
      byte-identical except source_row. Kept the lower row-id.
    • Brand-prefix twins in sibling dirs: 4 pairs — treo-680 / treo-750v /
      treo-750 / treo-500v each had a phonedb-sourced palm-treo-… twin. Kept the
      bare GSMArena record (same convention as the fix(data): resolve 364 smartphone brand-prefix conflicts #330 merge), upgraded the name to the
      brand-inclusive form ("Palm Treo 680" …), unioned source_urls, deleted the twin.
      Treo 750 vs 750v and 500 vs 500v were kept separate — different SKUs.
    • Cross-source clean + scrape-gsma in same base dir: 22 records across 19 groups —
      the 16 HP iPAQ pairs, i-mate pda2/pda2k/pdal, and the Palm Treo Pro 3-way (bare +
      scrape-gsma + phonedb). Kept the clean record and repaired its ram_gb from the
      deleted twin's cited memory_internal (the clean side carried the fabricated
      ram_gb=1; the twin had the real 64/128 MB). scrape-gsma URL unioned so the RAM is
      sourced. No value invented.

    Left unmerged and documented (not guessed): the GSM webOS palm-pre-scrapegsma-7065
    and palm-pre-plus-scrapegsma-7063 overlap the region-split castle family
    (palm-pre[-plus]-palm-castle-c CDMA and palm-pre[-plus]-gsm-{eu,na}-palm-castle-g)
    but do not map 1:1. The EU/NA regional variant pairs (pixi-gsm, pixi-plus-gsm,
    pre-2-gsm, pre[-plus]-gsm-castle-g) are legitimate separate SKUs and were left alone.

    Systemic finding (reported, not mass-edited): the same fabricated-defaults
    signature the earlier passes flagged is present in pda too — nearly every clean
    record carries ram_gb=1 (real values 16–256 MB) and many Palm records carry
    os="Feature phone" for Palm OS / webOS devices. I only repaired ram_gb inside the
    confirmed merges (where the deleted twin supplied a cited value); the remaining
    placeholder records need a per-record source and were left untouched rather than
    mass-edited.

    Both gates pass on the merged tree (app.validate; integrity_check.py --strict —
    0 hard anomalies). Dump refreshed in the same PR (pdas 140 → 112, no other
    collection touched).

    Using Refs #296 rather than Closes: this clears the pda slice of the
    "near-duplicates in other categories not yet scanned" item, but that item stays open
    — smartphone still has 322 documented-unresolved pairs (per #330), and the remaining
    categories (laptop, monitor, CPU beyond the Atom pass, etc.) haven't been scanned for
    near-duplicates yet.

  13. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Follow-up to #332: ran a dedicated placeholder-value / fabricated-defaults scan across all 1,966 post-dedup data/laptop/ records (not just duplicate pairs), looking for the phone-import pattern where an import batch stamps an identical fake default (e.g. battery_mah=1500, weight_g=150.0) on unrelated devices regardless of their real specs.

    Result: no fabricated-default bug found in laptop data. Nothing to fix beyond what the dedup pass already caught. No files changed, so no PR.

    What I checked and why each common-looking value is real, not fabricated:

    • Per-field value frequency counted by distinct device (brand + base_model_slug, so config variants of one model don't inflate a value). For weight_g, ram_gb, storage_gb, os_version, display.size_inch/resolution/ppi, and combos (size+resolution, ram+storage, weight+size, full 5-tuple). Every high-frequency value is a genuinely market-dominant config (16GB RAM, 512GB SSD, 15.6"/1920x1080, Windows "10") and the distinct-device counts decay smoothly — there is no isolated spike of an implausible value stamped across unrelated models.
    • ppi is 100% internally consistent with each record's own size_inch + resolution — 0 mismatches >8px across 1,966 records. A fabricated display-spec batch would leave ppi inconsistent with the stated panel; it doesn't.
    • Field presence per source dataset is complementary, not faked: abhinavflac supplies msrp+ppi but no weight/os_version (0%); Ammok supplies weight+os_version but no msrp. Missing fields are simply absent in each source, not stamped with a default.
    • Non-round shared weights (the most placeholder-looking case) trace to real causes on spot-check: 1910 g is entirely the HP 15 (2017) chassis reused across configs; 1370 g is the real MacBook Pro 13" (2016-2019) / ThinkPad Yoga weight; 1360/1250/1200 g are genuine ultrabook weights shared by real ~13" machines. None is an implausible value on unrelated devices.
    • No physically-impossible values: nothing outside sane ranges; the 14 heaviest-for-size items are all real gaming/workstation laptops (Alienware 15/17, ROG G703, ZBook 17, Legion Y720, IdeaPad Y910) at their true weights.
    • Laptops have no battery_mah/battery_wh field at all, so the phone bug's exact vector doesn't exist here.

    validate.py / integrity_check.py --strict were not re-run since no records were modified (both already pass on this merged state per #332).

  14. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Follow-up to the monitor dedup (#333): I ran a standalone placeholder-value scan across all 231 post-dedup data/monitor/ records (216 distinct devices), specifically hunting for the fabricated-default pattern the tablet/watch and smartphone passes found (an import stamping an identical fake value — e.g. battery_mah=1500, weight_g=150.0 — onto unrelated devices), including records that have no duplicate twin to reveal it.

    Result: no fabricated-default bug in monitors. No data changed, no fix PR. Reporting plainly, as agreed.

    Method

    For every device-varying field I counted how many distinct devices share each exact value, then applied a decisive correlation test: a real common value is consistent with each device's own name/derivable specs, whereas a stamped default appears regardless of the device. I checked each field's most-repeated value against the model name (and, for ppi, against size+resolution).

    Why the high-frequency values are real, not stamped

    • features.response_time_ms = 1 (65 devices) — the biggest "suspicious" cluster. 64/65 are ≥100 Hz and/or say "gaming"/"1ms" in the title; 0 low-refresh office monitors falsely carry 1 ms, and 0 names state a different response time than the field. 1 ms is the genuine marketed spec for these gaming panels.
    • refresh_hz — of 125 records that state a refresh rate in the name, 0 disagree with the field. resolution — 0 mismatches across 109 named records. size_inch — only 4 apparent mismatches, all regex artifacts (e.g. "34.18 inch" parsed as 18, a "22-to-24-inch" privacy-screen range, rounding of 24.5→25).
    • msrp_usd = 100 (19 devices) — all genuinely budget/portable/office 1080p panels (22–27" IPS, ZenScreen portables, touch monitors); the field has 110 distinct prices overall and neighbours at 80/99/110, i.e. an organic distribution, not one price pasted everywhere.
    • rating — a textbook Amazon distribution: a smooth bell 4.2→4.6 (peak 4.6 = 51) with a tail down to 1.0 and a small 5.0 spike. Not a default.
    • panel_type = IPS, aspect_ratio = 16:9, resolution = 1920x1080, size_inch = 27" are simply the genuinely dominant real monitor specs, and they co-vary correctly with the names.
    • ppi — 3 records are internally inconsistent with their size+resolution (Dell Inspiron 24 5400 AiO, MSI MAG274UPF, Lilliput FA1014), but each has a unique wrong value, i.e. individual scrape noise, not the repeated-default fingerprint this pass targets (general per-field accuracy is out of scope here).

    The one genuinely-repeated placeholder — and why it's left as-is

    release_date = "2023-01-01" sits on all 216 devices. It is a real placeholder, but it's category-wide (not a fake value stamped on a subset of unrelated devices while others vary), it's a required field that validate.py will reject if nulled or removed, and there's no per-device source to derive true dates — so nulling it is impossible and inventing dates is off-limits. This is already the tracked "placeholder dates (YYYY-01-01)" item on this issue and needs an external per-device source, so I'm leaving it rather than touching it under this task.

    Verification

    No files changed. Confirmed the current tree is green:

    • python -m app.validate → Data validation passed.
    • python integrity_check.py <data> --strict → no hard anomalies.

    So the monitor category has no hidden fabricated-default values beyond the category-wide placeholder date already on the tracker.

  15. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Opened #337 against develop — a first source-citation pass over
    data/laptop/ and data/monitor/.

    Scope (checked live): laptop 1995 total / 1890 unverified; monitor 882
    total / 882 unverified. Every unverified record already carried a
    bulk-dataset link (huggingface Ammok, abhinavflac, Kaggle Amazon-monitor),
    so the gap was uncited, not unsourced.

    This PR (45 records touched):

    • Newly cited-and-verified (no value change): 35
    • Value-corrected-with-citation: 10 — MacBook Air weight x2, Surface
      Laptop weight x3, Pixelbook weight x3, LG 24GN60R size x1, Samsung
      Odyssey G9 49" DQHD resolution x1 (all corrected to match the cited
      vendor/Wikipedia spec, never invented).
    • Left uncited, no confirmable source this pass: 2727 (1866 laptop +
      861 monitor) — untouched, keep existing provenance.

    Each verified record was confirmed field-by-field against a manufacturer
    spec page or Wikipedia infobox (Apple, Microsoft, Google, ASUS, Dell,
    Gigabyte, MSI, Samsung, LG, BenQ, ViewSonic).

    python -m app.validate passes and integrity_check.py --strict reports
    no hard anomalies. The static dump was regenerated (python -m app.dump
    from TechEngine against the worktree) as the PR's final commit.

    The large remaining uncited set is a good candidate for follow-up batches
    (the abhinavflac records carry exact model SKUs that are individually
    verifiable against vendor pages).

  16. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Generic SoC placeholder sourcing — first pass (PR #338)

    Rescanned current develop for placeholder soc references. Three genuine
    placeholder SoC records hold essentially all of them:

    • data/soc/arm/2014/0.json ("0") — 3,610 device refs
    • data/soc/qualcomm/2011/qualcomm-snapdragon.json (bare family) — 1,762
    • data/soc/arm/2008/n-a.json ("N/A") — 273

    Total ~5,645 refs across ~2,793 base models. All three are verified: false
    Kaggle/phonedb bulk-import defaults with "Unknown mobile GPU"/"N/A" fields.

    Fixed: 45 records (Google Pixel). Pixel phones use one global SoC per model
    (no regional Exynos/SD split), so the chip is unambiguous and Wikipedia-documented.
    Re-pointed to existing verified records: Pixel(2016)→snapdragon-821,
    Pixel 3→snapdragon-845, Pixel 3a/3a XL→snapdragon-670,
    Pixel 4/4 XL→snapdragon-855, Pixel 4a→snapdragon-730g,
    Pixel 4a 5G/5/5a 5G→snapdragon-765g (30 off "0", 15 off qualcomm-snapdragon).
    Each gained the canonical Wikipedia device-page citation in source_urls; no
    values invented; assignments corroborated by sibling Pixel records already on the
    same chips.

    Left unresolved (documented, not guessed): ~5,600. The device records carry no
    embedded chipset signal — only name/RAM/battery/display, overwhelmingly from the
    aitoolbuzz/phonedb Kaggle imports. Repairing them means per-record web sourcing
    of thousands of mostly obscure/regional devices, i.e. inventing at scale — the same
    line #324/#329/#330 drew. Notable trap avoided: Samsung Galaxy S/Note region
    variants (SM-N960F Exynos vs SM-N960U Snapdragon, etc.) can't be blanket-mapped
    and need per-model-number sourcing.

    Both gates pass (app.validate ✅; integrity_check.py --strict → 0 hard
    anomalies); dump refreshed as the PR's last commit. Follow-up passes can pick off
    further single-global-SoC brands the same way (fully sourced, no invention).

    Refs #296

  17. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Follow-up to #332 (laptop): ran the same distinct-device placeholder-value scan on data/tablet/ (3,305 records) and data/watch/ (418 records), checking whether the fabricated-default pattern (battery_mah=1500, weight_g=150.0) or the RAM-in-storage bug survives outside what #324 and #301 already caught.

    Result: both categories are clean. No fabricated-default pattern remains, nothing to fix — no PR.

    What I checked:

    • The exact phone-bug values are gone: battery_mah=1500 and weight_g=150 have zero occurrences in either tablet or watch. fix(data): merge 183 confirmed tablet/watch near-duplicate records #324 fully eliminated them.
    • Distinct-device value frequency (keyed by brand + base_model_slug) for battery_mah, weight_g, ram_gb, storage_options_gb, display.size_inch/resolution/ppi, and combos (battery+weight, size+res+ppi). Every high-frequency value is a genuinely common real spec (4000/5000/6000 mAh tablet batteries, 7"/8"/10.1" sizes, 1GB/2GB/4GB RAM) and — critically — each is spread across 8–11 independent source datasets (aitoolbuzz, madajaswanth, jakubkhalponiak, cigarplug/scrape-gsma, arwinneil, beridzeg45, sady36, lileejass…). The fabricated-default fingerprint is a single value concentrated in one import batch stamped on unrelated devices; that does not appear here — the distributions decay smoothly and are cross-source.
    • ppi internal consistency with size + resolution: tablets 0 / 2,092 mismatches. Watches showed 11 apparent mismatches, but all are round-display models (255 of 413 watch resolutions are square W×W) where GSMArena publishes a diameter-based ppi rather than the naive pixel-diagonal. Spot-checked the Samsung Gear S3 (1.3", 360×360, stored ppi=278) against GSMArena, which lists exactly "360 x 360 pixels (~278 ppi density)" — so the stored values are correctly sourced, not fabricated. My rectangular formula was simply the wrong model for round panels.
    • fix(data): correct storage, battery and display fields from each record's own text #301 RAM-in-storage residue: 0 cases of ram_gb exceeding the max storage_options_gb — no swap residue.
    • Physical plausibility: no impossible values. The handful of small-for-tablet flags are real phone/tablet hybrids (Asus PadFone family) and Archos PMP-style "Internet Tablets"; the one battery flag (iPad Pro 12.9 2017 = 891 mAh, likely a source typo for ~10,891 mAh) is a unique value, not a repeated default, so it's outside the placeholder-pattern scope of this task.
    • Field presence per dataset is complementary, not fabricated (e.g. madajaswanth/jakubkhalponiak/lileejass supply battery+weight+ram but no display size/ppi; the gsmarena-derived sets supply full display). Missing ≠ stamped.

    validate.py / integrity_check.py --strict were not re-run since no records were modified.

    That covers tablet + watch for the "fabricated defaults in other categories" item alongside the laptop pass in #332.

  18. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Applied the same placeholder-value methodology from the monitor pass to data/gpu/ — all 2,083 records, hunting for the fabricated-default fingerprint (one value stamped across many unrelated cards whose real specs differ). Checked base_clock_mhz, boost_clock_mhz, memory_gb, tdp_w, release_date, plus memory_bus_bit, memory_type, memory_bandwidth_gbps, core counts, pcie_version, architecture, msrp_usd.

    Result: no fabricated-default bug in GPUs. No data changed, no fix PR. Reporting plainly.

    Why the repeated values are real, not stamped

    This is a curated TechPowerUp/Wikipedia-sourced set (verified: true), and the distinct-card frequency counts plus correlation-against-card-class checks all come back consistent:

    • Clocks — base_clock_mhz has 435 distinct values, boost_clock_mhz 478. The top repeats (500/600/650 MHz, ~80 cards each) are genuine round clocks that early-2000s GPUs actually ran at. Only 5 post-2016 cards carry a 500–650 base, and all are legitimate: Quadro/RTX Max-Q mobile parts (real 600 MHz base, 1215–1350 MHz boost) and Jetson Orin Nano SoC modules (fixed 625 MHz). boost == base on 53/735 post-2016 cards — every one is an OEM/workstation/datacenter part (FirePro, Radeon Instinct/Pro, Tesla, Radeon R5/R7 OEM) that genuinely ships without a separate boost clock, matching TechPowerUp.
    • memory_gb — the canonical power-of-two sizes (1/2/0.5/4/8/16 GB). The biggest cluster, 1.0 GB (319 cards), spans 2005–2017 and peaks 2010–2013 — the actual 1 GB-VRAM era — with zero cards released ≥2020. Period-consistent, not pasted onto modern cards.
    • tdp_w — standard tiers (50/75/150/250 W = slot power, 6-pin, 6+8-pin, flagship). Modern ≥16 GB cards show varied high TDPs (300/250/350/700/600/400 W), i.e. no flat default.
    • memory_bus_bit / memory_type / pcie_version / architecture — real taxonomies with natural distributions (128/256/64-bit; GDDR5/GDDR3/GDDR6; PCIe 2.0/3.0 x16; TeraScale/Tesla/Kepler…).
    • release_date — 955 distinct dates. The largest cluster, 2021-04-12 (30 cards), is a real NVIDIA professional/datacenter launch day (RTX A2000/A4000/A5000, A10/A16/A30, T600, Quadro T1200). Only 57/2,083 records use YYYY-01-01, and they are spread across 1996–2026 on old/obscure/OEM/engineering and future-unannounced cards (e.g. 2026-01-01 RTX 5070/5080 SUPER) — the honest "only the year is known" convention, not a single stamped placeholder like the monitor set's uniform 2023-01-01. (release_date is a required field, so it couldn't be nulled regardless, and there's no per-card source to invent exact days.)
    • Identical full-spec fingerprints — the largest are 4–9 cards, and each is a genuine rebadge/form-factor family that really shares specs (Radeon HD 5610/6510/6530 rebrands; Tesla K40c/d/m; Radeon X800 XL AGP/PCIe; Voodoo family). Real shared values, not fabrication.

    Verification

    No files changed. Current tree is green:

    • python -m app.validate → Data validation passed.
    • python integrity_check.py <data> --strict → no hard anomalies.

    Note on the parallel GPU work

    Another pass is doing near-duplicate/cross-check on data/gpu/. My scan was strictly the fabricated-placeholder angle and found nothing to fix, so there's no overlap with their dedup edits and no data conflict from my side (I changed nothing). The GPU category has no hidden fabricated-default values.

  19. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Follow-up backfill run (targeting a higher --limit 2000 after #339 merged) is blocked by an active GSMArena rate limit — no records were upgraded this run.

    Details:

    • Rebased a fresh branch onto latest develop and ran the tool from the develop-pinned TechEngine commit (147d27d).
    • Started with a small dry-run to check the limit status before committing to a large batch. On the first fresh HTTP request GSMArena returned 429 with Retry-After: 36000s (10 hours). The tool stopped cleanly before sending any further requests (1 request total, exit 1) and wrote nothing.
    • This is the same 10-hour cooldown tripped during the fix(data): backfill 37 confirmed GSMArena URLs via verify tool #339 apply run (~3 hours ago); it has not yet expired.

    Per the standing rules I did not retry/spin, and since this was a no-op (working tree clean, zero writes) I'm reporting here rather than opening a PR. The tool and gate are unchanged and working — the earlier dry-run before #339 was clean — so this is purely a cooldown window, not a tooling regression.

    Next step: wait out the Retry-After window (roughly ~7 more hours from now) and re-run the --limit 2000 backfill then. The resume cache already carries the 37 confirmations from #339, so a later run resumes from where this left off without re-fetching them.

  20. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Ran a within-record self-consistency pass over data/pda/ (140 records), same pattern as #301 - values derived only from each record's own fields, no external sources.

    Found and fixed 1 real issue (PR #340):

    • data/pda/palm/2018/palm/palm.json: storage_options_gb was [3, 32], but the 3 is the RAM value (
      am_gb: 3 / �ariant.memory.ram_gb: 3) leaked into the storage list. The record's own �ariant.memory.storage_gb: 32 confirms storage is 32 GB only -> corrected to [32].

    Reviewed and intentionally left (ambiguous):

    • 6 records (i-mate PDAL, Mitac A701 x2 / A501 / A502 / A702):
  21. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Scanned the website category (one of the last untouched in the accuracy tracker) for near-duplicates. Opened #341 against \develop.

    Result: merged 153 confirmed near-duplicate groups and removed 154 redundant files (one 3-way group); the \websites\ collection drops 40,084 -> 39,930.

    Candidates were grouped by normalized \homepage_url\ + display name, with records sharing the exact same normalized URL as the strong signal (195 such groups reviewed). Confirmed duplicates fell into these patterns:

    • QID-suffixed clones of an otherwise identical record (<slug>-q);
    • bare-domain stubs vs the proper record (e.g. \3dmgame-com\ -> \3dm, \�mazon-com\ -> \�mazon);
    • protocol / www / trailing-slash / punctuation variants;
    • same-entity name variants incl. acronym/expansion and translated open-data-portal names (e.g. ANNO / AustriaN Newspapers Online, Gosuslugi / State Services).

    Each merge kept the fuller/verified/better-sourced side, unioned \source_urls, OR'd \�erified, and filled only missing optional fields — no invented values.

    Left as distinct (the known trap): records that only share an index/landing URL but are different real entities — museum exhibition pages under one archive host, imageboard boards, registration services differing only by URL fragment (#ul/#ip), and borderline company-vs-website / subsite / regional-version pairs (itv.com vs ITVX, Daily Mail vs MailOnline, two municipalities sharing one open-data portal URL, etc.).

    Verification: \�pp.validate\ passes; \integrity_check.py --strict\ reports no hard anomalies (no dup slugs, no slug/filename mismatch, no verified-without-source). Static dump refreshed as the PR's last commit.

    Remaining open sub-item: the parallel software scan.

  22. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Ran a fresh within-dataset self-consistency pass over \data/tablet/\ and \data/watch/\ (follow-up to #301, since more data has moved through since — e.g. the tablet/watch dedup in #324 and the mobile-source reclassification).

    Found and fixed 261 records (259 tablet + 2 watch) where the device's RAM size had leaked into \storage_options_gb. In every case the polluting value equals the record's own \�ariant.memory.ram_gb\ (and top-level
    am_gb) and is strictly smaller than every real storage tier in the same list — those tiers independently confirmed by the record's own \�ariant.memory.storage_gb. A 1-2 GB entry can't be a storage capacity for these devices, so it's the RAM size, and the fix removes it. Example: \iconia-tab-a100\ had \storage_options_gb: [1, 16]\ with
    am_gb: 1\ → corrected to [16].

    Every correction is derived only from fields already in the same record — no external sources, no invention. Genuinely ambiguous cases (multiple RAM tiers in the list, or a low value that could plausibly be a real storage size) were deliberately left untouched.

    Verification:

    • \python -m app.validate\ → passed
    • \integrity_check.py --strict\ → exit 0, no hard anomalies, structural section clean, no tablet/watch flags
    • Mandatory dump refresh committed last, scoped 1:1 to the 261 tablet/watch detail files (pre-existing unrelated dump drift excluded).

    PR: #342

  23. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    GPU near-duplicate + spec-accuracy pass done — #343 (against develop).

    Scanned all 2,076 data/gpu/ records with the same method as the CPU / tablet
    / watch / smartphone passes. Found 8 confirmed cross-source duplicate pairs
    (same physical product imported twice under different slug conventions, one
    side Kaggle-sourced, one TPU/vendor-sourced). Merged each into the more
    complete / correctly-sourced record, source-verified every disagreeing field,
    unioned source_urls, deleted the redundant file: 2,076 → 2,068.

    Notable spec conflicts resolved against authoritative sources:

    • MI325X: kept 256 GB HBM3e (AMD official), dropped the 288 GB Kaggle value.
    • RTX 5060 Ti 16 GB: kept base 2407 MHz (NVIDIA 2.41 GHz), dropped 2250 MHz.
    • V100 PCIe: kept TDP 250 W; V100 SXM2: kept TDP 300 W and corrected
      FP32 16.35 → 15.7 TFLOPS (NVIDIA datasheet).
    • RTX 3060 8 GB: kept the correct 2022-10-12 launch record.

    No configuration variants were merged. Close families that only share a name
    prefix — Tesla K40 d/m/s/t, GRID vGPU profiles, PCI/AGP and MXM/PCIe bus
    variants, OEM SKUs — are distinct real products and were left untouched, per
    the #327 guidance.

    Both gates pass: app.validate ✅ and integrity_check.py --strict ✅
    (structural section clean). Dump refreshed via python -m app.dump, scoped to
    v1/gpus/** + manifest.

  24. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Followed up the pda dedup pass (#335) with a placeholder-value scan across all
    112 post-dedup pda records
    — looking for the fabricated-default fingerprint (one
    fake-precise value repeated across many unrelated devices) independent of whether a
    record had a duplicate twin. Opened #344.

    Clusters found:

    • ram_gb = 1 on 83 of 112 records — fabricated (no PDA 2000-2011 had 1 GB;
      Dell Axim X5 = 32/64 MB, Palm IIIc = 8 MB, verified against Wikipedia/PCMag/
      TidBITS). The Kaggle/phonedb bulk-import placeholder.
    • battery_mah = 1500 on 33 (mostly the classic Palm line) — fabricated but
      unfixable, see below.
    • ppi = 114 (26), the 3.5"/240x320 combo (26), and weight_g = 190 (9) are
      real shared values, not fabricated: 240x320 QVGA on a 3.5" panel genuinely is
      ~114 ppi, and 190 g is real chassis-sharing (iPAQ h63xx family + i-mate HTC
      rebrands). Left untouched.

    Fixed: 18 ram_gb=1 → real sourced values (ram_gb is a required field, so
    repaired to a cited value, not nulled; no value invented):

    • 13 Dell Axim — Wikipedia Dell Axim spec table (low-end "Basic" = 32 MB; every
      other model incl. the whole X50/X51 line = 64 MB).
    • 5 Fujitsu-Siemens Pocket Loox N520/N560 = 64 MB (Fujitsu N520 manual + clubic
      N560 spec).

    Left documented rather than guessed or nulled (this is the important part for
    the tracker):

    • The classic Palm line (m100 = 2 MB, m105/m125/m130/m500/m505 = 8 MB, m515 =
      16 MB, IIIc/IIIxe/VIIx = 8 MB, Tungsten/Zire/Treo/Centro …) has real RAM below
      the schema's ram_gb >= 0.016 (16 MB) floor
      , so it is not representable —
      almost certainly why these got the ram=1 placeholder on import. Can't set the
      real value (fails validate) and can't null a required field. This is a schema
      limitation worth noting
      : sub-16 MB RAM devices can't be recorded accurately at
      all. ~50 records are in this bucket.
    • Remaining ram_gb=1 (i-mate PDA phones, HP iPAQ Glisten, Mitac Mio 728, later
      Palm Treo/webOS) need per-device sourcing; left documented.
    • battery_mah=1500: classic Palms ran on AAA/AA cells (no mAh rating) or small
      internal Li-ions — no clean per-device mAh to cite without inventing, and the
      field is required (can't null).

    Both gates pass (app.validate; integrity_check.py --strict — 0 hard anomalies).
    Dump refreshed in the same PR (only the 18 repaired detail pages change).

    Refs #296 — removes a sourced slice of the pda placeholders; the
    unrepresentable/unsourceable remainder stays open and documented above.

  25. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Batch progress on the uncited-website backlog — opened #345 against develop.

    What landed: 7,086 data/website/ records upgraded from Wikidata-only
    (verified: false) to source-verified, by adding an independent, checkable
    citation — the English Wikipedia article linked from each record's Wikidata item
    (enwiki sitelink) — and setting verified: true. No values were fabricated or
    altered.

    Method: for each verified: false, Wikidata-only record, resolve its QID and
    query the Wikidata API. If the item has an English Wikipedia article, that URL is
    a real human-checkable source for the same entity, so it's appended to
    source_urls and the record is verified. Records with no enwiki article are left
    untouched — no citation invented.

    Counts (this batch):

    Outcome Count
    Cited-and-verified (enwiki URL added, verified: true) 7,086
    Value-corrected 0
    Left uncited — no enwiki article 25,346
    Left uncited — launch_date vs Wikidata P571 year mismatch, flagged for manual review 17
    Already verified before this pass 7,635
    Total website records 40,084

    0 corrections on purpose: 17 records had a launch_date disagreeing with the
    Wikidata inception year (P571). Since a site's launch date and its subject's
    inception are different facts (e.g. Netflix 1997 the company vs. 2007 the service),
    overwriting one with the other would inject wrong data, so they were left as-is and
    flagged for manual review.

    Verification: app.validate passes; TechEngine integrity_check.py --strict
    exits 0 with no hard anomalies; static dump refreshed for the affected website
    records (scoped to data/website/ + its site/public/v1/websites dump).

    The remaining uncited records (those without an English Wikipedia article) are
    left for future batches.

  26. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Opened #346 against develop — a first source-citation pass over
    data/tablet/ and data/watch/ (companion to the laptop/monitor pass in
    #337).

    Scope (checked live): tablet 3305 total / 3020 unverified; watch 418
    total / 325 unverified. As with laptops/monitors, every unverified record
    already carried a bulk-dataset link (Kaggle, aitoolbuzz, GSMArena-scrape,
    GitHub), so the gap was uncited, not unsourced.

    This PR (13 records touched):

    • Newly cited-and-verified (no value change): 10 — Galaxy Tab S11 /
      S11 Ultra / S10 Ultra / S10+, Galaxy Fold, Galaxy Z Fold 6, Kindle Fire;
      Pixel Watch 4 41mm / 45mm, Galaxy Watch Ultra.
    • Value-corrected-with-citation: 3 — Surface 2 os "Windows Mobile" ->
      "Windows RT 8.1"; Galaxy Watch 7 storage_options [2,32] -> [32]; Galaxy
      Watch 6 storage_options [2,16] -> [16] (spurious 2 GB entries removed).
    • Left uncited, no confirmable source this pass: 3312 (3012 tablet +
      300 watch) — untouched, keep existing provenance.

    Each verified record was confirmed field-by-field (RAM, battery, weight,
    OS, SoC, storage) against a Wikipedia infobox or manufacturer/GSMArena spec
    page.

    python -m app.validate passes and integrity_check.py --strict reports
    no hard anomalies. The static dump was regenerated (python -m app.dump
    from TechEngine against the worktree, fresh scratch DB) as the PR's final
    commit.

    Large remaining uncited set is a strong follow-up target: ~60 tablet/watch
    records already cite GSMArena/phonedb per-model pages and just need
    field-by-field confirmation once those sources are fetchable (GSMArena was
    rate-limiting during this pass).

  27. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    PDA placeholder ram_gb/os fix, 2026-09-29

    Followed up on the PDA dedup's systemic finding (uniform ram_gb=1 and os="Feature phone" across the category). PR #347 corrects 83 of the 112 data/pda/ records, each value sourced from that device's cited datasheet.

    • os (50 records): "Feature phone" → real platform: "Palm OS" (Palm III/V/VII/m-series/Tungsten/Zire/TX/Z22/Treo≤755p/Centro), "webOS" (Pre/Pixi families), "Windows Mobile" (Dell Axim X5, MiTAC Mio 728). Already-correct Windows Mobile records were left as-is.
    • ram_gb: placeholder 1 → true capacity from phonedb.net's "RAM Capacity" field, converted MiB→GB in the existing convention (64→0.0625, 128→0.125, 256→0.25, 512→0.5, 32→0.032, 16→0.016). variant.memory.ram_gb mirrored.
    • source_urls: added for the 6 kaggle-only records (Wikipedia for Treo 180/270/600/650; imei24 for iPAQ Glisten and i-mate Pocket PC).

    Spot-verified several directly (Palm IIIc 8 MiB/Palm OS 3.5, Palm TX 32 MiB/Palm OS 5.4, Palm Pre 2 512 MiB/webOS). No values invented.

    Left unresolved (documented, not guessed): 12 pre-2003 Palm classics have 2–8 MiB RAM (m100 & Zire = 2 MiB; IIIc/IIIxe/VIIx/m105/m125/m130/m500/m505/i705/Zire 21 = 8 MiB), which is below the ram_gb schema floor (0.016 GB = 16 MiB in app/validate.py). Their os was corrected to Palm OS, but ram_gb was left rather than writing an overstated in-range value. Raising the validate floor is an engine-side change; noted for a follow-up.

    Verified: python -m app.validate passed; integrity_check.py --strict reported no hard anomalies. Dump refreshed as the PR's last commit (83 changed PDA pages only).

  28. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Follow-up self-consistency pass over data/monitor/ (fresh scan of all 882 records).

    Found and fixed: 18 records with a null aspect_ratio that is unambiguously derivable from the record's own resolution field (and consistent with its own name). Filled using the ratio labels this catalog already uses — 16:9 (1920x1080 / 2560x1440 / 3840x2160), 21:9 (3440x1440 / 3840x1600), 32:9 (3840x1080 / 5120x1440), 16:10 (1280x800). PR: #348 (against develop).

    Deliberately left alone:

    • Resolution vs explicit digits in name: 0 mismatches.
    • Refresh rate: 0 mismatches (dual/OC rates like "165Hz/180Hz(OC)" already store the base value).
    • Size: ~19 differences, all marketing "Class"/nominal vs actual-panel size (e.g. 24" Class = 23.8"), which are legitimate and not errors.
    • 9 aspect_ratio records where the record's own resolution and name contradict each other (e.g. 49" "Dual QHD"/DQHD listed as 3440x1440, "27" 1440p" listed as 5120x1440, names asserting 21:9 on a 1920x1080 field). No single value is safely derivable from the record alone, so they were left and noted rather than guessed.

    Verification: python -m app.validate passes; integrity_check.py --strict reports no hard anomalies. Mandatory dump refresh included as the PR's last commit — only the 18 affected monitor detail files changed.

  29. Seungpyo1007 commented on Sep 29, 2026

    @Seungpyo1007
    MemberAuthor

    Focused follow-up pass for the "these are actually different devices, not
    duplicates" pattern (distinct from the "needs a source" residual, which I've left
    alone). Opened #349.

    Found 7 brand-prefix pairs that are genuinely different products sharing a base
    name — the 3 reported earlier plus 4 new ones — and un-paired each by giving the
    mis-slugged variant a distinct slug/filename/name (no merges, nothing invented):

    • HTC U11+ (u11 → u11-plus) vs U11 — distinct fall-2017 model (androidcentral).
    • HTC One M9+ (one-m9 → one-m9-plus) vs One M9 — Helio X10 vs SD810.
    • Huawei P40 Pro+ (p40-pro → p40-pro-plus) vs P40 Pro (pocket-lint/CNET).
    • Sony Xperia Z3+ (xperia-z3 → xperia-z3-plus) vs Xperia Z3 — the Z3+/Z4 is a
      distinct 2015 SD810 model with its own Wikipedia page.
    • Motorola Razr 2023 (razr → razr-2023) vs Razr 2019.
    • Fairphone 3+ (→ fairphone-3-plus-dual-sim-lte-fp3plus) vs Fairphone 3.
    • Gigabyte GSmart GS202+ (→ gsmart-gs202-plus) vs GS202.

    In each pair the base-model record is left untouched; only the variant's slug/name/
    filename change so the two stop colliding as a false pair. Residual brand-prefix
    pairs drop 183 → 176.

    Counts: confirmed-different-devices-not-duplicates un-paired 7; merged 0
    (not this pass); still-needs-source residual (21 same-model wrong-soc + 157
    ram/weight/battery) intentionally untouched — those need external sources.

    Both gates pass on #349 (app.validate, integrity_check.py --strict, slug==
    filename holds); dump refreshed in the same PR. I believe this exhausts the readily
    identifiable "genuinely different device" pattern in the smartphone brand-prefix
    set — the remainder is the source-needed residual. Refs #296.

  30. added 20 commits that reference this issue on Sep 29, 2026
  31. Seungpyo1007 commented on Oct 6, 2026

    @Seungpyo1007
    MemberAuthor

    Google Play identifiers merged (#359): 31,903 smartphone/tablet/watch records now carry model_numbers / codenames from Google Play supported_devices.csv. Ids only, no spec value changes. ~21.8k lower-confidence rows are held for review.

  32. Seungpyo1007 commented on Oct 6, 2026

    @Seungpyo1007
    MemberAuthor

    Google Play ids round 2 merged (#361): 174 more records, written per path with SoC and sibling-id checks (blind audit 38/40; the 2 misses are fixed by the ≥4-char codename rule). The new-device hold pools failed two blind fresh-sample audits (81–84%) and stay in review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't workingdataDataset changesenhancementNew feature or requesttrackingLong-running tracker issue (stays open)

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions