🤖 QA Automation Agent

An AI agent that reads your requirements, tests your staging app, and alerts your team — automatically.

Stop manually testing before every deploy. This agent reads your Notion docs, drives a real browser through your user flows, spots bugs using Claude, and pings your Slack with a one-click approve button that ships to prod.

View Demo · Quick Start · Architecture · Contributing

---

Live Deployment

• API base URL: https://<your-domain>
• Health check: https://<your-domain>/health

---

What it does

Stage 1 — Pre-deploy QA (runs on every PR merge to staging)

1. Reads your feature requirements from a Notion page
2. Claude extracts structured acceptance criteria from your prose
3. Claude generates a human-readable browser action plan
4. Playwright executes every step, taking screenshots along the way
5. Claude evaluates the results against your requirements
6. If a bug is found → Slack alert with the action plan, screenshot, severity label, and one-click Approve & Deploy / Dismiss buttons

Stage 2 — Post-deploy validation (runs after prod deploy)

1. Pulls the last hour of Amplitude event streams from production
2. Claude reads the session sequences against the same Notion requirements
3. If real users are hitting unexpected paths → alerts the developer who pushed

Notion ──► requirementsParser ──► planGenerator ──► Playwright ──► resultEvaluator ──► Slack
                                                                                          │
                                                                              Approve ────┤
                                                                                          ▼
                                                                               GitHub Actions
                                                                                          │
                                                                                          ▼
                                                                    Amplitude ──► Claude ──► Slack

---

Demo

Stage 1 fires when a PR is merged to staging. The agent navigates the staging URL, fills the form, clicks submit, and catches the bug:

🔴 Bug Detected — demo-contact-form
Severity: critical  |  Confidence: high

What went wrong:
After clicking Submit, the page redirected to /error instead of showing
the success message "Thank you! We'll be in touch." as required.

Action plan Claude executed:
1. Navigate to https://staging.example.com/contact → Contact form visible
2. Fill "Name" field with "Test User" → Name field populated
3. Fill "Email" field with "test@example.com" → Email field populated
4. Fill "Message" field with "Hello" → Message field populated
5. Click "Submit" button → Success message appears

Triggered by: [View Notion requirement →]

[ ✅ Approve & Deploy to Prod ]   [ ❌ Dismiss ]

---

Architecture

qa-agent/
├── agent/
│   ├── claudeClient.js          # Shared Anthropic API wrapper
│   ├── requirementsParser.js    # Claude Call 1 — Notion prose → structured JSON
│   ├── planGenerator.js         # Claude Call 2 — requirements → action plan
│   ├── resultEvaluator.js       # Claude Call 3 — screenshots + page state → bug report
│   ├── stage1.js                # Stage 1 orchestrator
│   └── stage2.js                # Stage 2 orchestrator
├── browser/
│   └── playwrightRunner.js      # Headless browser — executes action plans
├── integrations/
│   ├── notionClient.js          # Fetches page content
│   ├── amplitudeClient.js       # Pulls session event streams
│   ├── slackClient.js           # Sends alerts + interactive buttons
│   └── githubClient.js          # Dispatches workflow runs
├── db/
│   ├── schema.sql               # feature_runs table
│   └── queries.js               # Raw pg queries
├── server/
│   └── index.js                 # Express: /slack/interact + /run-check
├── config/
│   └── team.config.json         # GitHub username → Slack user ID map
├── scripts/
│   └── seed-demo.js             # Seeds a demo run + instructions to add a bug
└── .github/workflows/
    ├── stage1-qa-check.yml      # Triggers on PR merge to staging
    └── stage2-prod-verify.yml   # Triggers via workflow_dispatch after prod deploy

Three focused Claude calls, each independently logged to Postgres. If one fails, the others don't re-run. Every input and output is stored — full audit trail per run.

---

Quick Start

Prerequisites

• Node.js 20+
• A Railway account (free tier — for Postgres + Express hosting)
• A Notion workspace with API access
• An Amplitude account (free tier works)
• A Slack workspace where you can create apps

1. Clone and install

git clone https://github.com/your-username/qa-agent.git
cd qa-agent
npm install
npx playwright install chromium

2. Set up environment variables

cp .env.example .env

Open .env and fill in every value. See the Environment Variables section for where to get each one.

3. Set up the database

In Railway: create a new Postgres service → copy the connection string into DATABASE_URL. Then:

psql $DATABASE_URL -f db/schema.sql

Expected output: CREATE TABLE

4. Configure your team

Edit config/team.config.json:

{
  "userMap": {
    "your-github-username": "YOUR_SLACK_USER_ID"
  }
}

Find your Slack user ID: Slack → your profile → three-dot menu → Copy member ID.

5. Create your Notion requirements page

Create a Notion page with this structure (prose is fine — no database needed):

Feature: Contact Form

Users should be able to submit a contact form with their name, email, and message.

Acceptance Criteria:
- The form has fields for Name, Email, and Message
- Clicking Submit sends the form
- A success message "Thank you! We'll be in touch." appears after submission
- The form fields are cleared after successful submission

Relevant URL: https://your-staging-url.com/contact

Copy the page ID from the Notion URL (the 32-character string after the last -). Add it to .env as NOTION_PAGE_ID.

6. Set up the Slack app

1. Go to api.slack.com/apps → Create New App → From scratch
2. Incoming Webhooks → Activate → Add to workspace → copy URL → SLACK_WEBHOOK_URL
3. Interactivity & Shortcuts → Enable → set Request URL to https://<your-domain>/slack/interact
4. OAuth & Permissions → Bot scopes: chat:write, files:write → Install to workspace → copy Bot Token → SLACK_BOT_TOKEN
5. Basic Information → copy Signing Secret → SLACK_SIGNING_SECRET

7. Run tests

Run these in order. Each group depends on the previous passing.

# Infrastructure
npm test tests/db.test.js
npm test tests/claudeClient.test.js

# Integrations
npm test tests/notionClient.test.js
npm test tests/amplitudeClient.test.js

# Agent modules
npm test tests/requirementsParser.test.js
npm test tests/planGenerator.test.js
npm test tests/playwrightRunner.test.js
npm test tests/resultEvaluator.test.js

# Slack + GitHub
npm test tests/slackClient.test.js      # check Slack channel for the message
npm test tests/getSlackUser.test.js
npm test tests/githubClient.test.js

# End-to-end
npm test tests/stage1.test.js           # full pipeline — Slack alert should fire
npm test tests/stage2.test.js           # requires stage1 run to exist first

8. Deploy the Express server

Push to GitHub → Railway → New Project → Deploy from GitHub repo → add all .env values as Railway environment variables. Railway auto-detects npm start.

Set your Slack app's Interactivity Request URL to https://<your-domain>/slack/interact.

---

Running the demo

Option A — Manual trigger (fastest for demos)

npm start
curl -X POST "http://localhost:3000/run-check?feature=<your-notion-page-id>"

Option B — Automated via PR

Open a PR against the staging branch. Paste your Notion page ID as the PR body. Merge it — Stage 1 fires automatically.

Introduce a deliberate bug for the demo

In your staging app, make the form submit handler redirect to /error instead of showing the success message. Stage 1 will catch it, fire the Slack alert, and show the Approve/Dismiss buttons.

Seed demo data

npm run seed

---

15-Minute Handover Checklist

Use this to onboard the QA agent into a new team/project quickly.

1. Fork or clone this repository into the target GitHub account.
2. Create and fill .env from .env.example.
3. Set up Postgres and run schema:
   • psql $DATABASE_URL -f db/schema.sql
4. Configure Slack app:
   • webhook URL
   • bot token
   • signing secret
   • interactivity URL: https://<your-domain>/slack/interact
5. Configure Notion:
   • create integration
   • share requirements page with integration
   • set NOTION_PAGE_ID
6. Configure Amplitude keys for prod session verification.
7. Configure GitHub token and repo envs (GITHUB_OWNER, GITHUB_REPO).
8. Deploy backend service (Railway or equivalent persistent Node host).
9. Set custom domain and verify health:
   • https://<your-domain>/health
10. Add GitHub Actions secrets and enable workflows.
11. Update config/team.config.json:
    • map GitHub usernames to Slack member IDs.
12. Run validation tests in order:
    • npm test tests/db.test.js
    • npm test tests/stage1.test.js
    • npm test tests/stage2.test.js

After this checklist, teams can trigger Stage 1 from PR merges or manual API calls and use Slack Approve/Dismiss for controlled promotion to Stage 2 verification.

---

Environment Variables

Variable	Where to get it
ANTHROPIC_API_KEY	console.anthropic.com → API Keys
NOTION_API_KEY	notion.so/my-integrations → New integration
NOTION_PAGE_ID	From your Notion page URL
AMPLITUDE_API_KEY	Amplitude → Settings → Projects → API Key
AMPLITUDE_SECRET_KEY	Amplitude → Settings → Projects → Secret Key
AMPLITUDE_EXPORT_DELAY_HOURS	Optional. Defaults to 4 (handles Export API lag)
AMPLITUDE_EXPORT_WINDOW_HOURS	Optional. Defaults to 24 (query window size in hours)
SLACK_WEBHOOK_URL	Slack app → Incoming Webhooks
SLACK_SIGNING_SECRET	Slack app → Basic Information
SLACK_BOT_TOKEN	Slack app → OAuth & Permissions
AI_PROVIDER	anthropic, gemini, or openai
ANTHROPIC_MODEL	Optional Anthropic model override
GEMINI_API_KEY	Required only if AI_PROVIDER=gemini
GEMINI_MODEL	Optional Gemini model override
OPENAI_API_KEY	Required only if AI_PROVIDER=openai
OPENAI_MODEL	Optional OpenAI model override
OPENAI_BASE_URL	Optional OpenAI-compatible base URL (for example, ChatMock)
GITHUB_TOKEN	GitHub → Settings → Developer Settings → Fine-grained tokens (needs Actions: write)
GITHUB_OWNER	Your GitHub username
GITHUB_REPO	Your repo name
DATABASE_URL	Railway Postgres → connection string
STAGING_BASE_URL	e.g. https://your-app.vercel.app
PORT	Default 3000

Add all of these as GitHub Actions secrets too (Settings → Secrets → Actions). Use GH_TOKEN for your personal access token — GITHUB_TOKEN is reserved by GitHub Actions.

---

How Claude is used

Three separate, stateless API calls — each with a focused system prompt:

Call	Input	Output
requirementsParser	Raw Notion page content	[{ requirement, acceptanceCriteria, relevantUrl }]
planGenerator	Structured requirements	[{ stepNumber, action, expectedOutcome }]
resultEvaluator	Action plan + page states	{ passed, severity, confidence, findings[], summary }

Each call's input and output is logged to Postgres independently. If evaluation fails, you don't re-run the browser session. Every run is a full audit trail.

Default provider: anthropic (claude-sonnet-4-6). You can switch to Gemini/OpenAI via env.

---

Contributing

Contributions are welcome. A few areas where help would be great:

• Better Playwright action parsing — the current step executor uses regex heuristics. A proper NLP-to-selector mapper would make it more reliable across diverse UIs
• Screenshot diffing — attach screenshots to the Slack alert as image attachments
• Multi-page requirement support — currently one Notion page per feature run; supporting linked pages would help larger teams
• Structured Notion DB support — the requirements fetcher is abstracted behind getPageContent() — a DB adapter would be a clean swap

To contribute:

git checkout -b feat/your-feature
# make changes
npm test
git commit -m "feat: description"
# open a PR

Please run the full test suite before opening a PR.

---

License

MIT — use it, fork it, build on it.

---

Built with Claude API · Playwright · Notion API · Amplitude · Slack API