RELIANCE: Curating and Evaluating Reproductive Health Information on Social Media, accepted at the KDD 2026 Datasets and Benchmarks track.
🌐 Project page: https://realize-lab.github.io/RELIANCE/ · 📄 Paper: https://arxiv.org/abs/2606.18285
RELIANCE (Reproductive hEaLth Information Access in oNline Content Environments) is an expert clinician-annotated dataset and benchmark for evaluating whether large language models (LLMs) can fact-check inaccurate and harmful reproductive health misinformation on social media. It contains 409 clinician-annotated sentences or paragraphs from 336 real-world TikTok videos across 56 clinician-reviewed peripartum queries covering pregnancy, postpartum, and fertility, each labeled separately for medical inaccuracy and harmfulness by expert clinicians in obstetrics, gynecology, and internal medicine. We use these annotations to benchmark LLMs (including Gemini, GPT-4o, Qwen, Llama Guard, and omni-moderation) on claim-level verification and video-level detection of health misinformation.
data.json consists of the RELIANCE dataset
TikTik was the semi-manual tool used to collect the data