E2E Tests and Golden Paths Analysis

.github/workflows/ci.yml

@@ -0,0 +1,78 @@
+name: CI
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  unit-and-integration-tests:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Build and start containers
+      run: |
+        cd mock-project
+        docker compose up --build -d
+
+    - name: Wait for API
+      run: sleep 10
+
+    - name: Run unit tests
+      run: |
+        cd mock-project
+        docker compose exec api pytest unit_tests/ -v
+
+    - name: Run integration tests
+      run: |
+        cd mock-project
+        docker compose exec api pytest integration_tests/ -v
+
+    - name: Stop containers
+      run: |
+        cd mock-project
+        docker compose down
+
+  e2e-tests:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Setup Node.js
+      uses: actions/setup-node@v4
+      with:
+        node-version: '20'
+
+    - name: Install dependencies
+      run: |
+        cd mock-project
+        npm ci
+
+    - name: Install Playwright browsers
+      run: |
+        cd mock-project
+        npx playwright install chromium
+
+    - name: Start containers
+      run: |
+        cd mock-project
+        docker compose up --build -d
+
+    - name: Wait for API to be ready
+      run: |
+        sleep 10
+        curl --retry 5 --retry-delay 2 --retry-connrefused http://localhost:5000/health || true
+
+    - name: Run Playwright tests
+      run: |
+        cd mock-project
+        npx playwright test
+
+    - name: Stop containers
+      run: |
+        cd mock-project
+        docker compose down

.gitignore

@@ -205,3 +205,6 @@ cython_debug/
 marimo/_static/
 marimo/_lsp/
 __marimo__/
+node_modules/
+mock-project/node_modules/
+*.swp

assignment-4-e2e-test-report.md

@@ -0,0 +1,34 @@
+# E2E Test Report - Assignment 4
+
+## Test Environment
+
+- **Local:** Docker Compose on Ubuntu
+- **CI:** GitHub Actions (Ubuntu latest)
+- **Browser:** Chromium (headless)
+- **Test Framework:** Playwright 1.60.0
+
+## Test Results
+
+| Test | Status | Duration |
+|------|--------|----------|
+| homepage loads and shows title | PASS | 278ms |
+| health endpoint returns ok | PASS | 48ms |
+| user can submit answers and complete the test | PASS | 13.4s |
+| results page shows cone scores | PASS | 13.5s |
+| reset button clears results and restarts test | PASS | 13.7s |
+
+**Total:** 5 passed, 0 failed (41.8s)
+
+## AI Generation
+
+The E2E tests were initially generated by AI and then manually refined to match the Ishihara-style diagnostic application. The AI provided a good structure but assumed a simpler API-based application. I had to rewrite the tests to work with the form-based submission and dynamic plate generation.
+
+## Limitations
+
+- The tests use "0" as the default answer for all plates. This does not test the user's actual color vision.
+- The tests do not verify the accuracy of the diagnosis, only that the flow completes.
+- Full diagnostic accuracy requires manual validation against clinical tests like Enchroma.
+
+## CI/CD Integration
+
+The E2E tests run automatically on every push and pull request via GitHub Actions. The workflow installs Playwright browsers, starts the Docker container, runs the tests, and stops the container.

assignment-4-golden-paths.md

@@ -0,0 +1,31 @@
+# Golden Paths Analysis
+
+## What users pay for
+
+Users pay for an accurate color blindness diagnosis. The most valuable scenario is a user completing the full test and receiving a reliable result that matches clinical tests.
+
+## Golden Path 1: Complete diagnostic test
+
+A user opens the web application, answers all test plates by typing the numbers they see (or 0 if they see nothing), and receives a diagnosis with cone percentage scores.
+
+**Why this is valuable:** This is the core value proposition. Without an accurate diagnosis, the test has no value.
+
+## Golden Path 2: Retaking the test
+
+A user completes the test, receives a result, and clicks "Take Test Again" to start a fresh session with randomized plates.
+
+**Why this is valuable:** Users may want to verify their results or test again after the variance disclaimer. This also demonstrates that the test is not deterministic.
+
+## Golden Path 3: Health check for operations
+
+A DevOps engineer calls the `/health` endpoint to verify the service is running before routing traffic.
+
+**Why this is valuable:** For enterprise deployment, reliability monitoring is essential. Users cannot take the test if the service is down.
+
+## E2E tests covering these paths
+
+| Golden Path | E2E Test |
+|-------------|----------|
+| Complete diagnostic test | `user can submit answers and complete the test` |
+| Retaking the test | `reset button clears results and restarts test` |
+| Health check | `health endpoint returns ok` |

assignment-4-manual-testing.md

@@ -0,0 +1,44 @@
+# Manual Testing Required Beyond Automation
+
+## What cannot be automated
+
+1. **Diagnostic accuracy** – An automated test cannot verify that the diagnosis matches a clinical test like Enchroma. Only a human with known color blindness can validate this.
+
+2. **Perceptual difficulty calibration** – The test needs to be challenging but not impossible. Automated tests only check that the flow completes, not that the colors are properly calibrated.
+
+3. **User experience** – An automated test cannot judge whether the instructions are clear, the interface is intuitive, or the results are easy to understand.
+
+4. **Realistic user behavior** – Automated tests follow a script (typing 0 for every plate). Real users may hesitate, change answers, or behave unpredictably.
+
+## Manual test cases to run
+
+1. **Diagnostic accuracy** – Have a user with known color blindness (Protan, Deutan, or Tritan) take the test. Compare the results to their Enchroma or clinical diagnosis.
+
+2. **Calibration check** – A person with normal vision should get "Normal Color Vision" consistently. If they get a false positive, the test is too hard.
+
+3. **Variance check** – Take the test 3 times. Scores should vary within the +/- 13% disclaimer. Large swings indicate instability.
+
+4. **Usability** – Ask a first-time user to take the test without instructions. Where do they get stuck? Is the "enter 0 if you see no number" instruction clear?
+
+5. **Cross-browser testing** – Test on Chrome, Firefox, and Safari. The Canvas rendering should be consistent.
+
+6. **Mobile testing** – The test is designed for desktop. On mobile, the numbers may be too small. Document this limitation.
+
+## Manual test results template
+
+| Test Case | Result | Notes |
+|-----------|--------|-------|
+| Protan user diagnosis | Pending | Need test subject |
+| Deutan user diagnosis | Pending | Need test subject |
+| Tritan user diagnosis | Pending | Need test subject |
+| Normal vision (3x) | Pending | Should get Normal each time |
+| First-time usability | Pending | Observe hesitation points |
+| Chrome | Pending | |
+| Firefox | Pending | |
+| Safari | Pending | |
+
+## Recommendations for improvement
+
+- Recruit colorblind users for validation
+- Add a calibration mode with known control plates
+- Implement a confidence score based on response consistency

mock-project/Dockerfile

@@ -0,0 +1,6 @@
+FROM python:3.11-slim
+WORKDIR /app
+COPY backend/requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+COPY backend/ .
+CMD ["python", "app.py"]

mock-project/README.md

@@ -0,0 +1,126 @@
+# Color Vision Diagnostic Test
+
+A web-based Ishihara-style color vision test that diagnoses Protan (red deficiency), Deutan (green deficiency), and Tritan (blue deficiency) color blindness.
+
+## Prerequisites
+
+- Docker installed on your machine
+- Git (to clone the repository)
+- Node.js and npm (for E2E tests only)
+
+## Getting Started
+
+Clone the repository and navigate to the mock-project folder:
+
+```bash
+git clone https://github.com/pitekopaga/testing.git
+cd testing/mock-project
+```
+
+## Run the Web UI
+
+Start the application:
+
+```bash
+docker compose up --build -d
+```
+
+Open your browser and go to: `http://localhost:5000`
+
+## How to Take the Test
+
+1. A circle of colored dots will appear with a hidden number
+2. Type the number you see in the input box
+3. If you see no number, type **0**
+4. Click Submit
+5. Repeat for all plates
+6. After the final plate, you will see your results with cone percentage scores
+
+## Functional Requirements
+
+| ID | Requirement | Status |
+|----|-------------|--------|
+| FR-1 | Present Ishihara-style dot patterns with hidden numbers | Implemented |
+| FR-2 | Accept user input as a number (0-99) | Implemented |
+| FR-3 | Accept "0" when user sees no number | Implemented |
+| FR-4 | Track answers across all test plates | Implemented |
+| FR-5 | Calculate cone response percentages for red, green, and blue | Implemented |
+| FR-6 | Diagnose Protan, Deutan, or Tritan color blindness based on lowest cone score | Implemented |
+| FR-7 | Provide diagnosis with description | Implemented |
+| FR-8 | Allow user to reset and retake the test | Implemented |
+
+## Non-Functional Requirements
+
+| ID | Requirement | Status |
+|----|-------------|--------|
+| NFR-1 | Test completes within 2 minutes for typical users | Implemented |
+| NFR-2 | Plates are generated client-side using Canvas API | Implemented |
+| NFR-3 | Application runs in Docker container | Implemented |
+| NFR-4 | No user data is stored permanently (session-only) | Implemented |
+| NFR-5 | Test provides results with +/- 13% variance disclaimer | Implemented |
+| NFR-6 | Instructions are displayed on every test page | Implemented |
+| NFR-7 | Progress indicator shows current plate number and total | Implemented |
+| NFR-8 | Results page shows individual cone scores with progress bars | Implemented |
+
+## Run Automated Tests
+
+### Unit Tests
+
+```bash
+docker compose exec api pytest unit_tests/ -v
+```
+
+### Integration Tests
+
+```bash
+docker compose exec api pytest integration_tests/ -v
+```
+
+### E2E Tests (Playwright)
+
+```bash
+npm install
+npx playwright install chromium
+npx playwright test
+```
+
+## Test the API manually with curl
+
+Health check:
+
+```bash
+curl http://localhost:5000/health
+```
+
+Expected output: `{"status":"ok"}`
+
+## Stop the Application
+
+```bash
+docker compose down
+```
+
+## Clearing Browser Data (if needed)
+
+If the test behaves unexpectedly, clear your browser data for localhost:
+
+**Chrome:**
+1. Click the lock icon next to the address bar
+2. Click "Cookies and site data"
+3. Click "Manage cookies and site data"
+4. Click the trash icon next to `localhost`
+5. Refresh the page
+
+**Alternative:** Open an incognito/private browsing window.
+
+## Troubleshooting
+
+**Port 5000 is already in use:** Stop the process using port 5000, or change the port mapping in `docker-compose.yml`.
+
+**The test gives unexpected results:** Clear your browser data as described above.
+
+**Playwright tests fail:** Run `npx playwright install chromium` to ensure browsers are installed.
+
+## License
+
+MIT