An open methodology for measuring the real quality of third-party contact data — using labels curated from your own inbox and CRM.
Author: Harrison Tang · License: CC BY 4.0 — reuse freely with attribution.
Every B2B data buyer faces the same two questions: which vendor's data is actually good for my use case? and am I getting the quality I'm paying for? Today, almost everyone answers them blind.
Vendors self-report accuracy — "95% accurate, human-verified" — on samples they select themselves. Independent tests routinely find that providers claiming 90–98% accuracy deliver 70–85% on real lists. The "bake-off" content you find online is mostly published by vendors ranking themselves first. Review sites measure sentiment, not data. And the standard buyer defense — request 200 sample records, spot-check 20 against LinkedIn, call 10 phone numbers — has no statistical power, no per-field resolution, and no protection against a cherry-picked sample.
There is a market failure hiding in this, and it is also my candidly self-interested reason for publishing. When every vendor claims the best, most accurate data and no buyer can verify any of it, the claims cancel each other out: quality becomes unobservable, so purchases get decided by marketing and price, and genuinely better data cannot command its premium. This is the classic market for lemons — unverifiable quality punishes exactly the vendors who invest in it. I build data products for a living, so my motivation here is not altruism: a buyer side equipped with rigorous, statistical verification is what lets sophisticated data vendors shine on merit. Measurement helps honest vendors as much as it helps buyers.
Meanwhile, B2B contact data decays at a commonly cited ~2%/month (with measured estimates ranging from ~22% to as much as 70% per year depending on the field), so even an honest benchmark goes stale. And because data brokers extensively buy from and resell each other — a fact documented as far back as the FTC's 2014 data broker report — buying from two vendors often means buying the same records through two storefronts, without knowing it.
The fix is a very old idea applied to a new domain: trust, but verify. You don't need to take any vendor's claim on faith, because you already own the one asset no vendor can fake: ground truth about the people you actually know. Your inbox and your CRM contain thousands of implicitly verified facts — the email address that got a reply last Tuesday is live; the title in a fresh email signature is current; the phone number with a logged connected call works. Curate those facts into a labeled truth set, match vendor data against it with proper entity resolution, and you can compute real, per-field, per-segment quality metrics for any vendor — including detecting when two vendors are likely selling you the same upstream data.
Every component of this methodology is established science — weak supervision, probabilistic record linkage, truth discovery, binomial confidence intervals. What's new is the combination, applied to an industry that has never been audited from the buyer's side. This document specifies the methodology precisely enough that you (or your AI assistant) can build a working evaluator from it. At the end there is a build spec you can paste into an AI agent verbatim.
Start from one premise: no single truth set rules them all. The truth about a person is high-dimensional — who they are, where they work, how to reach them, all changing over time — while any one dataset (your inbox, your CRM, any vendor) is a low-dimensional observation of it, sampled from one vantage point at one moment. No labeled set sees the full picture; the only way to approximate truth is to piece together many partial, lower-dimensional views. So: treat each of your first-party sources (inbox, CRM, email verifier) as a noisy labeling function that votes on contact facts; aggregate the votes with weak supervision to produce a probabilistic truth set — never letting any vendor being graded contribute to the answer key. Pull each vendor's records for the same people, including explicit misses. Entity-resolve vendor records to truth-set entities before scoring anything, because a mismatched person masquerades as an accuracy error. Then compute per-field × per-segment fill rate, precision, recall, and F1 with confidence intervals, and run cross-vendor overlap and correlated-error analysis to see which vendors are likely reselling the same source. Fill rate tells you what a vendor has; precision tells you whether it's right; recall tells you what they're missing; overlap tells you whether a second subscription adds anything at all.
Why multiple sources. This is the operational consequence of the premise above: each source observes a different low-dimensional projection of the same person, and no single one is trustworthy alone. CRMs are stale and often silently populated by the very vendors you want to grade. Inboxes verify emails superbly but say little about phone numbers. Email verifiers validate deliverability but know nothing about titles. Call logs know phone-correctness and nothing else. The design principle: combine several individually unreliable, differently-angled sources into one label set that sees more of the picture than any of them.
The weak supervision frame. This is a solved problem — don't hand-roll a scoring heuristic. Data programming (Ratner et al., NeurIPS 2016; Snorkel, VLDB 2018) models each source as a labeling function (LF) that votes a value or abstains, then learns each LF's accuracy and the correlation structure between LFs from agreement/conflict patterns alone — no ground truth needed. The output is a probabilistic label per field per person. Dawid–Skene (1979) is the classical ancestor and a fine lightweight substitute.
Example labeling functions, roughly ordered by trustworthiness:
| Labeling function | Field(s) | Signal strength |
|---|---|---|
| Email received a human reply (recent) | work email | Near-certain positive |
| Email hard-bounced | work email | Near-certain negative |
| Third-party verifier says deliverable | work email | Strong positive |
| Phone had a logged call with "connected" disposition | phone | Strong positive |
| Title parsed from a recent email signature | title | Moderate |
| CRM field, human-entered, recently updated | any | Moderate |
| Name/company inferred from sender + corporate domain | name, company | Moderate-strong |
| CRM field of unknown provenance | any | Weak |
An LLM is the practical enabler here: it can parse signatures, classify reply-vs-bounce threads, and normalize titles at scale — extraction tasks that needed custom models a decade ago (signature-line extraction hit F1 ≈ 0.97 with CRFs back in 2004; LLMs do it zero-shot).
Three hard rules that keep the evaluation honest:
- No graded vendor ever contributes to the answer key. If a vendor's data leaks into your labels (e.g., a CRM title auto-stamped by that vendor's enrichment), grading it against those labels is circular — the vendor gets credit for agreeing with itself. Track field provenance in your CRM and exclude vendor-sourced fields from labeling.
- Verification services refine labels; they are never contestants and never the sole label source. An email verifier is an LF on deliverability, not an answer oracle for anything else.
- Timestamp every observation (
observed_at). A 2019 signature title is likely wrong today. Weight labels by recency and let fresh outcome signals (new replies, new bounces) upgrade or downgrade old labels.
One critical subtlety: declare your correlations. Weak supervision's standard label models assume LFs are conditionally independent unless told otherwise. Your CRM contact and your inbox are often not independent — the CRM record may have been created from that same email. Unmodeled correlation makes the label model overconfident. Either declare known dependencies (Snorkel supports this) or collapse correlated sources into a single LF.
Sampling. You do not need your whole CRM. A stratified sample — stratified by the segments you actually buy data for (industry × geography × seniority × company size) — of a few hundred entities with ≥30 per stratum gives stable metrics. Cap contacts per company (3–4) so one account doesn't dominate a stratum.
For each vendor, look up the same entities that are in your truth set and record what the vendor returns — including explicit misses. A vendor returning nothing for a person is the signal that recall measures; if you only keep hits, recall is uncomputable and every vendor looks better than it is. Never let the vendor choose the sample: sample selection is where vendor-run benchmarks quietly win. If you're evaluating a vendor pre-purchase from a free sample or trial, note that trial data may be curated; measured-on-trial ≠ measured-in-production.
Blind the pull. Query vendors with a blinded version of the truth set: strip out the target fields you're evaluating (email, phone, title, …) and send only matching keys — enough identifying information for the vendor to resolve the person, and nothing more. If your query contains the email you're testing, a vendor (or its API) can simply echo it back, and your precision measurement collapses into measuring the vendor's ability to copy your input. Blinding also minimizes the personal data you ship to a third party.
Choosing matching keys is a real design problem. Name + company alone is usually not discriminating enough — many people share a name, and many companies share a name — so include the company URL domain at minimum, and a LinkedIn URL or location when it isn't itself a field under test. There is an inherent tension here: the strongest join keys (work email, LinkedIn URL) are often the very fields you want graded, and withholding them makes vendor-side matching harder. Accept that trade-off, but account for it: with weak keys, a vendor "miss" may be a match failure rather than absent data, so keep the two cases distinct where the vendor's response lets you tell them apart ("no record found" vs "record found, field empty"), and report both. Your own Stage 3 scoring is unaffected — on your side of the wall you resolve vendor returns against the full, unblinded truth set.
You cannot score "is the vendor's title correct for Jane?" until you're confident the vendor's record is Jane. Entity-resolution errors masquerade as accuracy errors: mismatch the person and the vendor looks wrong when your join was wrong. This is classic probabilistic record linkage (Fellegi & Sunter, 1969) with excellent open-source implementations (Splink, dedupe, recordlinkage).
Practical rules: match on a cascade of keys (exact canonical email → LinkedIn URL → fuzzy name + company domain); set a minimum match-confidence threshold; and exclude ambiguous pairs from scoring but report the exclusion count. If 15% of pairs were too ambiguous to match, that is a finding in itself, and silently scoring them would corrupt every downstream metric. Ideally, re-run the scoring at a couple of thresholds — if a vendor's precision swings with the match threshold, the swing is linkage noise, not data quality.
Matching is field-specific. Exact string equality is the wrong test for almost every field:
| Field | Normalize | Match |
|---|---|---|
| lowercase, canonical form | exact | |
| Phone | E.164 (e.g., phonenumbers lib) |
exact |
| Name | trim, casefold | fuzzy (Jaro-Winkler ≥ ~0.92) |
| Company | casefold + resolve domain | fuzzy + domain equality |
| Title | normalize seniority/function ("VP Sales" ≈ "Vice President, Sales") | taxonomy / LLM-judged equivalence |
Exact-matching titles badly understates every vendor's accuracy; the match function is a hyperparameter, so document whatever you choose and apply it identically to every vendor.
Metrics, per field × per stratum:
- Fill rate (coverage): share of matched entities where the vendor supplied a value.
- Precision: of the values supplied, share correct per the truth set.
- Recall: of the entities where the truth set knows the value, share the vendor supplied correctly.
- F1, plus raw support (n) and TP/FP/FN so others can recompute.
Fill rate and precision must never be conflated — a vendor can fill 95% of phone fields with 60% of them wrong, and most public "match rate" numbers hide exactly this.
Report uncertainty with Wilson score intervals, not the ±z√(p(1−p)/n) Wald formula. Your per-stratum cells will be small (n = 20–50), and Wald intervals fail exactly there: they collapse to zero width at 0% or 100% (5-of-5 correct reports "precision 100%, CI [100%, 100%]") and can exceed [0,1]. Wilson turns 5-of-5 into an honest [57%, 100%] (Wilson 1927; Brown, Cai & DasGupta 2001). Flag any stratum below a minimum support (~30) as low-confidence.
Resist the single composite score. A vendor great at emails and poor at mobiles must show up as exactly that. If you need one number for a decision, weight fields by your use case explicitly — and show the weights.
With two or more vendors measured against the same truth set, three cheap analyses answer a question no vendor will: is my second subscription redundant?
- Coverage overlap: for each field, the Jaccard overlap of which entities each vendor fills. High overlap → the marginal vendor adds little coverage.
- Value agreement (Cohen's κ): agreement on supplied values, corrected for chance. Suspiciously high κ suggests a shared upstream source.
- Shared-error fingerprinting — the decisive test: two vendors independently getting facts right is expected (there is one truth); two vendors making the same mistakes — the same wrong title, the same dead email — is strong evidence of common sourcing. This is the core insight of the truth-discovery literature (Dong, Berti-Équille & Srivastava, VLDB 2009): shared false values, not shared values, indicate dependence. Compute, among entities where both vendors are wrong per your truth set, how often they are wrong identically.
Interpretation caution: correlated errors establish shared data lineage, not a business relationship. Vendor A buying from Vendor B, both licensing from the same aggregator, or both scraping the same public profile all produce the same signature and are largely indistinguishable without temporal data. Report findings as "consistent with a shared upstream source."
A practical corollary of independence analysis: agreement between vendors is only weak evidence of correctness. If your two vendors agree on a phone number, that's confirmation only if they're independent — which is precisely what this stage measures. And once you have per-segment metrics per vendor, an obvious consumption of the results is to reorder any multi-vendor waterfall by measured quality per segment rather than by price or habit.
Being honest about limitations is what separates measurement from marketing.
1. Your truth set measures "known-contact accuracy," not market coverage. Inbox-derived entities are people you already correspond with — systematically different from the cold prospects you buy vendors for. A vendor strong in geographies or industries you never email will be underrated. The IR-evaluation literature shows biased judgment pools can be unfair to specific systems in exactly this way (Büttcher et al., SIGIR 2007). Mitigations: stratify, label results honestly as known-contact accuracy, and where possible add outcome-based checks on cold data (send/bounce/connect rates on newly purchased records — live outcomes are immune to selection bias, and email is the one field with a cheap near-oracle in deliverability verification).
2. The truth set is itself an imperfect reference. People change jobs; signatures go stale. The diagnostic-testing literature on imperfect gold standards shows even small reference errors meaningfully bias accuracy estimates. Treat vendor–truth disagreement as a discrepancy to sample and inspect, not automatically a vendor error — in one classic audit of commercial web sources, a large share of apparent conflicts were staleness and semantic ambiguity rather than error (Li et al., "Truth Finding on the Deep Web," VLDB 2013). Recency-weight your labels; retire old ones.
3. Fields differ enormously in verifiability. This method verifies email cleanly (replies, bounces, deliverability oracles), company moderately, title weakly (taxonomy problems, staleness), and phone-correctness barely at all without call outcomes. Expect and report per-field confidence in the methodology itself, not just in the data.
4. Small strata lie. A "healthcare / EU / C-level" cell with n = 12 will produce dramatic-looking differences that are pure noise. This is why the Wilson intervals and low-support flags are not optional decoration.
5. Titles and firmographics have no oracle. Where no verifier exists, your best evidence is cross-source agreement discounted for the source-dependence you measured in Stage 5 — never raw agreement.
6. One user's truth set is small. A single person's inbox and CRM typically yield well under 500 high-confidence labeled entities, and after stratification each cell is smaller still. At that scale you can reliably detect large quality gaps (90% vs 60% precision) but not close calls (88% vs 84% — the Wilson intervals will overlap), and rare segments may be unmeasurable entirely. The mitigations are all forms of pooling: have co-workers contribute labels from their own inboxes and CRM views (same company, same ICP, so the pooled set stays relevant — dedupe entities across contributors, record who contributed what, and get internal consent before pooling colleague-derived contact data); or rely on a third party that aggregates many users' labeled sets into a larger, cross-organization benchmark — which buys statistical power and segment coverage at the cost of introducing exactly the trust, privacy, and independence questions this document says to ask of any data intermediary. Pooled or not, report your n per stratum and let the intervals speak.
Everything below is a self-contained specification. Paste it into a capable AI agent (e.g., "build me an agent skill implementing this spec") along with access to your inbox/CRM exports and vendor CSVs. A competent agent can implement it as a set of scripts or an agent skill in an afternoon. Suggested stack: Python, pandas,
splink(or deterministic cascade matching),snorkel(or simple Dawid–Skene EM, or weighted vote as a fallback),rapidfuzz,phonenumbers,statsmodels(Wilson intervals), optional email-verification API.
SPEC: Data-Vendor Evaluator ("trust but verify")
CONFIG (declare up front):
TARGET_FIELDS = fields being evaluated, e.g. { work_email, phone, title }
MATCH_KEYS = fields used to identify the person in vendor queries,
e.g. { full_name, company, company_domain, linkedin_url }
RULE: vendor queries contain MATCH_KEYS only — never TARGET_FIELDS (blinding).
If a field is a target (e.g. linkedin_url), it cannot also be a match key.
CANONICAL RECORD (one per person per source):
entity_hint: { full_name, work_email, phone, title, company, company_domain, linkedin_url }
each field is { value | null, source_tag, observed_at } # null = explicit miss; never drop nulls
strata: { industry, geo, seniority, company_size }
source_tag examples: gmail:reply, gmail:signature, crm:human_entered, crm:unknown,
verifier:<name>, vendor:<name>
outcome events (evidence, not fields): reply(email,+), bounce(email,-), call_connected(phone,+)
STEP 1 — TRUTH SET
1a. Pull correspondence contacts (inbox) + active CRM contacts. EXCLUDE any CRM field
whose provenance is a vendor being graded (circularity rule).
1b. Extract per-field observations with source_tag + observed_at
(LLM parses signatures, reply/bounce classification, title normalization).
1c. Aggregate per field with weak supervision:
preferred: Snorkel LabelModel; acceptable: Dawid-Skene EM; fallback: weighted vote
with weights ~ [reply/bounce 1.0, verifier 0.9, connected-call 0.9, recent signature 0.6,
human CRM 0.5, unknown CRM 0.2], recency-decayed (half-life ~12 months).
Declare/merge correlated sources (CRM populated from email = NOT independent).
1d. Keep fields with aggregate confidence >= 0.8 as labels; store confidence + latest observed_at.
1e. Stratified sample: >= 30 entities/stratum, 200-500 total, <= 4 per company.
1f. Optional pooling (raises statistical power): merge labeled sets contributed by
co-workers. Dedupe entities across contributors (same ER cascade as Step 3),
tag each label with its contributor, re-run Step 1c on the merged label matrix,
and obtain internal consent before pooling colleague-derived contact data.
1g. Emit TWO views of the truth set:
- FULL view (all fields + labels) -> used only in Steps 3-5, never sent out
- BLINDED view (MATCH_KEYS + strata) -> the only thing used to query vendors
STEP 2 — VENDOR PULL (blinded)
For each vendor V and each truth entity: query V using the BLINDED view only.
Store returned values AND explicit nulls per field, with retrieval timestamp.
Distinguish and record separately where the vendor's response allows:
- no_match (vendor found no record for the keys) -> possible match failure
- field_null (record found, field empty) -> true coverage gap
Report the no_match rate alongside recall; with weak keys a "miss" may be the
vendor failing to resolve the person, not lacking the data.
Never let V choose the sample. Never include TARGET_FIELDS in any query.
STEP 3 — ENTITY RESOLUTION (before any scoring; uses the FULL truth-set view)
Match cascade: canonical-email exact -> linkedin_url exact ->
(name Jaro-Winkler >= 0.92 AND company_domain equal).
Probabilistic option: splink with these comparisons. min_match_confidence = 0.85.
Pairs below threshold: EXCLUDE from scoring, COUNT and REPORT as "ambiguous_excluded".
Sensitivity check: recompute headline metrics at 0.80/0.90 thresholds.
STEP 4 — SCORING (per field x per stratum, per vendor)
normalize: email lowercase-canonical; phone E.164; name/company casefold;
title -> normalized (seniority, function) via taxonomy or LLM-equivalence judge
match: email/phone exact; name JW >= 0.92; company fuzzy+domain; title taxonomy-equivalent
metrics: fill_rate = filled / matched
precision = correct / filled (only where truth set has a label)
recall = correct / truth_known
f1, support, tp, fp, fn
intervals: Wilson score, 95%, on every proportion; flag support < 30 as low_support
output: NO composite score by default; optional weighted composite with printed weights
STEP 5 — OVERLAP ANALYSIS (each vendor pair, per field)
coverage_overlap = Jaccard(filled_A, filled_B)
agreement_kappa = Cohen's kappa on supplied values
shared_error_rate = P(identical wrong value | both wrong per truth set)
Interpret shared_error_rate >> 0 as "consistent with shared upstream source" (not proof of resale).
REPORT (markdown):
1. Methodology summary + truth-set provenance mix + sample sizes per stratum
2. Per-vendor: per-field table (fill, precision [CI], recall [CI], F1, n) overall + per stratum
3. ER audit: match rate, ambiguous_excluded, no_match vs field_null rates,
threshold sensitivity
4. Vendor-pair overlap matrix + kappa + shared-error findings
5. Caveats: known-contact bias, low-support strata, fields with weak verifiability,
overall sample size / statistical power (state the smallest detectable gap)
6. Discrepancy sample: 20 random vendor-vs-truth disagreements for manual review
PRIVACY & LEGAL GUARDRAILS (encode into the skill):
- Contact data is personal data: minimize the sample, document lawful basis (GDPR/CCPA),
check your DPA before sending contacts to any vendor API.
- Many vendor ToS restrict DISCLOSING benchmark results (DeWitt-style clauses).
Keep results internal to your procurement decision; warn the user before any export/share.
- Never publish measured results for named vendors without checking your subscription terms.
A skill or script built from this spec is genuinely useful — and I encourage you to build one — but know what you're signing up for at scale. A one-off evaluation on 300 entities is an afternoon. Doing it well, continuously, is a system: labels decay (so curation must re-run and re-weight perpetually), outcome signals need ongoing capture, entity resolution and title taxonomies need real investment, per-segment coverage needs far more labeled entities than one team's inbox contains, and every re-measurement should be versioned and reproducible. None of that changes the methodology — it changes the engineering. Start with the free version; you'll learn more about your data vendors in a week than in years of taking their word for it.
This methodology stands on established work; the contribution is the combination and the domain.
- Weak supervision: Ratner et al., Data Programming, NeurIPS 2016; Ratner et al., Snorkel, VLDB 2018 (arXiv:1711.10160); Dawid & Skene, 1979.
- Truth discovery & source dependence: Dong, Berti-Équille & Srivastava, Integrating Conflicting Data: The Role of Source Dependence, VLDB 2009; Li et al., Truth Finding on the Deep Web: Is the Problem Solved?, VLDB 2013 (the closest methodological precedent — an empirical audit of commercial data sources, finding rampant inconsistency and copying); Li et al., A Survey on Truth Discovery, SIGKDD Expl. 2016.
- Record linkage / entity resolution: Fellegi & Sunter, JASA 1969; Splink (MoJ); Wu et al., Ground Truth Inference for Weakly Supervised Entity Matching, arXiv:2211.06975.
- Evaluation with imperfect/biased ground truth: Buckley & Voorhees, SIGIR 2004; Büttcher et al., SIGIR 2007; the partial-verification-bias literature in diagnostic testing.
- Interval estimation: Wilson, JASA 1927; Brown, Cai & DasGupta, Statistical Science 2001.
- Data quality dimensions: Wang & Strong, JMIS 1996.
- Auditing data brokers: Neumann, Tucker & Whitfield, How Effective Is Third-Party Consumer Profiling?, Marketing Science 2019; FTC, Data Brokers: A Call for Transparency and Accountability, 2014.
If you build something from this, I'd love to hear about it. Attribution per CC BY 4.0 appreciated.