Supplementary Materials for "EduGuardBench"

This document provides an overview of the supplementary materials for the paper, "EduGuardBench: A Holistic Benchmark for Evaluating the Pedagogical Fidelity and Adversarial Safety of LLMs as Simulated Teachers".

The repository contains the full paper appendix, datasets, code, and a directory for results, allowing for the full replication of our experiments.

Directory Structure

The repository is organized as follows:

/
├── README.md                           # This overview file
├── Appendix for EduGuardBench.pdf                      # Full appendix of the paper in PDF format
│
├── Dataset/
│   ├── SATAs.xlsx                      # Dataset for the Teaching Harm experiment
│   └── adversarial_prompts.xlsx      # Dataset for the Adversarial Safety experiment
│
├── code/
│   ├── Teaching_Harm_Evaluation/
│   │   └── run_teaching_harm_evaluation.py
│   │
│   ├── Adversarial_Safety_Experiment/
│   │   └── run_adversarial_safety_experiment.py
│   │
│   ├── Analysis_and_Plotting/
│   │   └── run_analysis_and_generate_figures.ipynb
│   │
│   └── utils/
│       ├── api_clients.py
│       └── helpers.py
│
└── Results/
    ├── Teaching_Harm_Results/          # Directory for .xlsx outputs from the Teaching Harm script
    └── Adversarial_Safety_Results/     # Directory for .xlsx outputs from the Adversarial Safety script

File Descriptions

Root Directory

• README.md: This file, providing a guide to the repository contents and structure.
• Appendix for EduGuardBench.pdf: The full appendix of the paper in PDF format. It provides extensive details that supplement the main paper, including in-depth descriptions of the benchmark scenarios, ethical flaw taxonomies, dataset design, the data generation process, full supplementary results, and the detailed annotation guidelines provided to the external expert team.

Dataset/

This directory contains the two core datasets used in our experiments.

• SATAs.xlsx: The dataset for the Teaching Harm evaluation. It contains the Select All That Apply (SATA) questions used to assess pedagogical fidelity and potential teaching harms.
• adversarial_prompts.xlsx: The dataset for the Adversarial Safety evaluation. It contains the persona-based adversarial prompts designed to assess safety alignment, particularly concerning academic misconduct.

code/

This directory contains all Python scripts and notebooks required to run the experiments and subsequent analysis.

code/utils/

This subdirectory contains shared helper functions.

• api_clients.py: A helper script that manages and standardizes API calls to various Large Language Models.
• helpers.py: A helper script containing common data I/O functions for Excel files and other formats.

code/Teaching_Harm_Evaluation/

This subdirectory contains the script for the Teaching Harm Assessment.

• run_teaching_harm_evaluation.py: The main script for this component. It reads the SATAs.xlsx dataset and a file of a model's pre-generated responses, then calculates and saves the key evaluation metrics (RFS, Accuracy, etc.) to a comprehensive Excel file.

code/Adversarial_Safety_Experiment/

This subdirectory contains the script for the Adversarial Safety Assessment.

• run_adversarial_safety_experiment.py: The main script for this component. This end-to-end script performs the following steps:
  1. Reads prompts from adversarial_prompts.xlsx.
  2. Queries a target model to obtain responses.
  3. Uses an LLM-as-a-Judge to evaluate the safety of each response.
  4. Saves a single, comprehensive Excel file containing the original prompt, model response, and the final safety/refusal quality labels.

code/Analysis_and_Plotting/

This subdirectory contains the script for final analysis and visualization.

• run_analysis_and_generate_figures.ipynb: A Jupyter Notebook that loads the final .xlsx result files from the Results/ directory. It is used to perform the statistical analyses and generate the figures presented in the paper.

Results/

This directory is intended to store the final, comprehensive outputs generated by the scripts.

• Teaching_Harm_Results/: This folder will contain the final .xlsx evaluation outputs from the Teaching Harm script. Each file (one per model) will include the original question data, the model's answer, and all calculated evaluation metrics.
• Adversarial_Safety_Results/: This folder will contain the final .xlsx evaluation outputs from the Adversarial Safety script. Each file (one per model) will include the original prompt, the model's raw response, the judge's final label, and the refusal type.