PALM: Path-Aware LLM-based Test Generation

PALM is a test generation technique that extracts path conditions from Java programs and uses LLMs to generate tests that cover each execution path.

Project Structure

PALM/
├── frontend/                  # Vue 3 interactive UI
├── backend/                   # Flask backend server
│   ├── app.py
│   ├── requirements.txt
│   ├── Dockerfile
│   └── PALM/                  # Core PALM library (Flask-facing)
│       ├── testMaker/         # LLM test generator
│       ├── pathExtractor/     # Scala/SBT path extractor
│       ├── benchmarks/        # HumanEval Java benchmark suite
│       └── sample/            # Sample JSON outputs
├── testMaker/                 # CLI-facing test generator (for evaluation)
├── pathExtractor/             # Scala/SBT path extractor (for evaluation)
├── subjects/                  # 133 Java benchmark programs (HumanEval)
├── experiments/               # Experiment results (30 runs × 3 models × 2 methods)
├── mutationTester/            # Mutation testing infrastructure
├── docker-compose.yml
└── README.md

---

Usage 1: Interactive UI (Docker)

Try PALM interactively with any Java program via the web interface.

Requirements

• Docker

Setup

In the backend/ folder, create a .env file:

OPENAI_API_KEY=your_key_here

Start

docker compose up --build

Then open http://localhost:5173

Stop

docker compose down

---

Usage 2: CLI Evaluation (Reproduce Experiments)

Run the full evaluation pipeline to reproduce our experimental results.

Requirements

• Python 3.8+
• Java 11+
• SBT (Scala Build Tool)
• An OpenAI API key

Setup

export OPENAI_API_KEY=your_key_here

Install Python dependencies:

pip install openai

Run

Execute 5 evaluation runs (matching our paper's experimental setup):

python -m testMaker.eval.eval 1 2 3 4 5

Each run generates tests for all benchmarks in subjects/ using both the PALM and Direct strategies across three models (gpt-4o-mini, o3-mini, o4-mini). Results are saved to experiments/.

Evaluate Existing Results

To compute metrics from pre-existing experiment results in experiments/:

python -m testMaker.eval.show

---

Experiments

Pre-computed results for 30 experiment runs are in experiments/. Each directory follows the naming convention {method}-{model}-{run_id}:

• direct-*: Direct LLM test generation (baseline)
• palm-normal-*: PALM path-aware test generation

Models evaluated: gpt-4o-mini, o3-mini, o4-mini

Each experiment directory contains:

• _output.json / _output2.json: Per-benchmark test generation outcomes
• _pc.log / _pc2.log: Aggregate coverage metrics (path, branch, statement coverage)
• generated-tests/: Generated test files
• verify-tests/: Compilation and execution verification results

---

Benchmark Suite

The subjects/ directory contains 133 Java implementations of HumanEval problems (correct versions for path extraction, buggy versions in benchmarks/ for mutation testing).