Replication package for the paper:
An empirical study on test fixture usage by coding agents on open source software João Almeida, Andre Hora
FixtureDB is a between-group study of test fixtures extracted from agent-enabled GitHub repositories. It is the companion code for a master's degree thesis in Software Engineering. The collection pipeline detects agent commits, extracts fixtures, and compares agent-authored and human-authored test code within the same repositories. It also includes a separate human-only dataset collected from pre-agent repositories for inter-repository baseline comparison.
Some of our agent-commit detection heuristics (co-authored-by trailer parsing, config-file patterns for individual agents) were cross-checked and improved against agent-mining, a community-maintained, empirically-verified catalog of AI coding agent detection heuristics maintained by the LaBRI research group. We're grateful for their work being openly available.
The repository contains three main datasets. The fixture collections will be regenerated during the next collection cycle.
-
fixtures-from-agents (Dataset A) — Agent-authored test fixtures extracted from commits identified as agent-generated. This is the agent corpus for the within-repository comparison. The directory also includes stratified repository sample CSVs (e.g.
dataset_c_sample.csv) for Dataset C. -
fixtures-from-humans (Dataset B) — Human-authored test fixtures extracted from the same repositories as Dataset A. This is the matched human control sample for the within-repository comparison.
-
pre-agent-baseline (Dataset C) — Human-authored test fixtures collected from pre-2022 software repositories that are independent from the agent-enabled corpus. This dataset serves as an inter-repository baseline. The repository sample files are stored under
fixtures-from-agents/asdataset_c_*.csv.
FixtureDB covers Python, Java, JavaScript, and TypeScript. For each fixture it extracts structural, semantic, and usage metrics through tree-sitter AST analysis, Lizard complexity measurement, and framework-specific pattern matching.
| Metric | Description |
|---|---|
loc |
Non-blank lines of code in the fixture body |
cyclomatic_complexity |
McCabe cyclomatic complexity of the fixture |
max_nesting_depth |
Maximum block nesting depth in the fixture body |
num_parameters |
Number of fixture parameters |
num_objects_instantiated |
Estimated object creations inside the fixture |
num_external_calls |
Estimated I/O or external library calls inside the fixture |
fixture_type |
Detected pattern (e.g. pytest_decorator, unittest_setUp) |
scope |
Execution scope (per_test, per_class, per_module, global) |
framework |
Detected testing framework (pytest, unittest, junit, jest, mocha, etc.) |
has_teardown_pair |
Whether the fixture has a teardown or cleanup counterpart |
fixture_dependencies |
Other fixtures or setup functions this fixture depends on |
mock_usages |
Mock framework usages associated with the fixture |
| Topic | Document |
|---|---|
| Overview and methodology | What is FixtureDB? |
| Installation and setup | Setup & Requirements |
| Repository layout | Repository Structure |
| Running the pipeline | Reproducing Results |
| Database schema | Database Schema |
| Agent detection | Agent Detection |
| Fixture detection | Fixture Detection |
| Metric definitions | Metrics Reference |
| Fixture patterns | Fixture Patterns Reference |
| CSV exports | CSV User Guide |
| Analysis examples | Analysis Guide |
| Limitations | Limitations & Threats to Validity |
| Tests | Test Suite & Validation |
See the full documentation index for the complete set of guides.