Real-Time Event-Driven Analytics Platform

Real-Time Event-Driven Analytics Platform

Applications send us events. We turn them into real-time analytics.

PulseFlow is a production-grade, full-stack analytics pipeline: client apps POST events via REST, a RabbitMQ broker decouples ingestion from processing, PostgreSQL stores durable records, Redis caches hot stats, and a React dashboard streams live updates over WebSocket — all orchestrated with Docker Compose in a single command.

Quick Start · Architecture · Event Lifecycle · API Reference

At a Glance

Metric	Value
Response Time	`< 100ms` on `POST /events` (publish-only, zero sync DB writes)
Throughput	Handles burst ingestion via async RabbitMQ queue
Event Types	12 supported: `LOGIN`, `PURCHASE`, `PAGE_VIEW`, `SEARCH`, etc.
Real-Time Latency	Sub-second dashboard updates via STOMP WebSocket
Failure Resilience	Dead-letter queue + exponential backoff + `failed_events` table
Auth	Dual: JWT (HS256) for admins, API key for machine clients
Tests	16 test files — unit + integration with Testcontainers
Infrastructure	5-service Docker Compose with health-checked dependencies
One-Command Launch	`docker compose up -d --build`

Why PulseFlow?

Most backend portfolio projects are CRUD applications. PulseFlow is a system — it demonstrates the architectural patterns that power analytics platforms at scale:

Pattern	Implementation
Event-Driven Architecture	Decoupled producers/consumers via RabbitMQ topic exchange
CQRS (Command Query Responsibility Segregation)	Write path (events module) separated from read path (analytics module)
Async Non-Blocking Ingestion	`POST /events` returns `202 Accepted` instantly — no synchronous DB write on the hot path
Single Writer Principle	One consumer owns all writes to Postgres/Redis — eliminates race conditions
Cache-Aside Pattern	Redis as disposable accelerator; Postgres is always the source of truth
Dead-Letter Queue	Failed messages retried with exponential backoff, then routed to DLQ + `failed_events`
Graceful Degradation	WebSocket disconnects? Auto-fallback to 5s REST polling — dashboard never goes blank
Dual Authentication	JWT for human admins, API key for machine-to-machine ingestion
Repository + Specification	Dynamic query building with JPA Specification API for filterable event listing
Observability	Health checks, queue depth monitoring, Redis status, audit logging

Architecture

System Pipeline Architecture

System Overview

flowchart TB
    subgraph Clients["Client Layer"]
        ClientApp["Client Application<br/><i>Any app sending events</i>"]
        Dashboard["React Dashboard<br/><i>Real-time analytics UI</i>"]
    end

    subgraph Ingestion["Ingestion Layer"]
        API["Spring Boot REST API<br/><i>Validates + publishes only</i>"]
        RateLimit["Rate Limiter<br/><i>Redis fixed-window</i>"]
    end

    subgraph Broker["Message Broker"]
        RMQ["RabbitMQ<br/><i>Topic exchange + DLQ</i>"]
    end

    subgraph Processing["Processing Layer"]
        Consumer["Analytics Consumer<br/><i>Idempotent, sole writer</i>"]
        DLQ["DLQ Consumer<br/><i>Failure tracking</i>"]
    end

    subgraph Storage["Storage Layer"]
        PG[("PostgreSQL 16<br/><i>Source of truth<br/>5 tables</i>")]
        Redis[("Redis 7<br/><i>Hot cache<br/>Namespaced keys</i>")]
    end

    subgraph Delivery["Delivery Layer"]
        WS["WebSocket Broadcaster<br/><i>STOMP over SockJS</i>"]
        AnalyticsAPI["Analytics REST API<br/><i>5 endpoints</i>"]
    end

    ClientApp -->|"POST /events<br/>X-API-Key"| RateLimit
    RateLimit --> API
    API -->|"publish<br/>EventMessage"| RMQ
    RMQ -->|"events.queue"| Consumer
    RMQ -->|"events.dlq"| DLQ
    Consumer -->|"persist event"| PG
    Consumer -->|"update counters"| Redis
    Consumer -->|"broadcast"| WS
    DLQ -->|"mark FAILED"| PG
    AnalyticsAPI -->|"read"| PG
    AnalyticsAPI -->|"read cached"| Redis
    Dashboard -->|"REST + JWT"| AnalyticsAPI
    Dashboard -->|"subscribe live"| WS

    style Clients fill:#1a1a2e,stroke:#5EEAA0,color:#E8F5E9
    style Ingestion fill:#16213e,stroke:#F2B84B,color:#E8F5E9
    style Broker fill:#0f3460,stroke:#FF6600,color:#E8F5E9
    style Processing fill:#16213e,stroke:#5EEAA0,color:#E8F5E9
    style Storage fill:#1a1a2e,stroke:#4169E1,color:#E8F5E9
    style Delivery fill:#16213e,stroke:#61DAFB,color:#E8F5E9

Event Lifecycle

The complete journey of an event from ingestion to dashboard display:

sequenceDiagram
    participant C as Client App
    participant R as Rate Limiter
    participant A as REST API
    participant Q as RabbitMQ
    participant Con as Consumer
    participant PG as PostgreSQL
    participant RD as Redis
    participant WS as WebSocket
    participant D as Dashboard

    C->>R: POST /events (X-API-Key)
    R->>A: Allow / Reject (429)
    A->>A: Validate payload
    A->>PG: INSERT event (status=PENDING)
    A->>Q: Publish EventMessage
    A-->>C: 202 Accepted (< 100ms)

    Q->>Con: Deliver message
    Con->>Con: Idempotency check
    Con->>RD: Update counters (INCR, SADD, ZINCRBY)
    Con->>PG: UPDATE status=PROCESSED
    Con->>WS: Broadcast to /topic/events + /topic/stats
    WS->>D: Live update (sub-second)

    Note over Con,Q: On failure: retry → exponential backoff → DLQ

Failure Handling Flow

flowchart LR
    A["Message<br/>Consumed"] --> B{"Processing<br/>Succeeds?"}
    B -->|"Yes"| C["Mark PROCESSED<br/>Update Redis<br/>Broadcast WS"]
    B -->|"No"| D{"Retries<br/>Remaining?"}
    D -->|"Yes"| E["Nack +<br/>Backoff Delay"]
    E --> A
    D -->|"No"| F["Route to DLQ"]
    F --> G["DlqConsumer:<br/>Mark FAILED"]
    G --> H["Insert into<br/>failed_events"]
    G --> I["Log to<br/>audit_logs"]

    style A fill:#16213e,stroke:#5EEAA0,color:#E8F5E9
    style C fill:#1a4a2e,stroke:#5EEAA0,color:#E8F5E9
    style F fill:#4a1a1a,stroke:#FF6600,color:#E8F5E9
    style H fill:#4a1a1a,stroke:#FF6600,color:#E8F5E9

Database Schema

erDiagram
    users {
        bigint id PK
        varchar username UK
        varchar email UK
        varchar password
        varchar role
        timestamp created_at
    }

    events {
        bigint id PK
        varchar event_type
        bigint user_id
        varchar source
        jsonb metadata
        timestamp received_at
        timestamp processed_at
        varchar status
    }

    failed_events {
        bigint id PK
        bigint event_id FK
        text reason
        int retry_count
        timestamp created_at
    }

    audit_logs {
        bigint id PK
        bigint admin_id FK
        varchar action
        text details
        timestamp created_at
    }

    users ||--o{ audit_logs : "generates"
    events ||--o{ failed_events : "fails into"

Key Design Decisions

Decision	Rationale
Publish-only ingestion	`POST /events` never touches Postgres or Redis synchronously — guarantees sub-100ms latency regardless of downstream load
Single writer principle	The RabbitMQ consumer is the sole writer for event data — centralizes retry/failure logic and eliminates race conditions
Redis is disposable	Every Redis key is namespaced, TTL'd, and reconstructible from Postgres — cache failures never cause data loss
Transactional queue publish	`EventMessage` is published only after the DB transaction commits — prevents phantom messages in the queue
WebSocket + polling fallback	Dashboard never goes blank — auto-switches to 5s REST polling if WebSocket disconnects
Dead-letter queue	Failed messages retried with exponential backoff, then routed to DLQ + `failed_events` — zero silent data loss
DTOs at boundaries	Java records for all request/response DTOs — JPA entities never leak to the API layer
Specification API	Dynamic query building for filterable/paginated event listing — no manual JPQL string concatenation

Tech Stack

Layer	Technology	Why
Language	Java 21 LTS	Virtual threads, modern LTS, pattern matching
Framework	Spring Boot 3.5	Web, Security, Data JPA, AMQP, WebSocket, Validation
Database	PostgreSQL 16	JSONB metadata, GIN indexing, full ACID compliance
Cache	Redis 7	Counters, sorted sets, HyperLogLog, fixed-window rate limiting
Message Broker	RabbitMQ 3.x	Topic exchange, DLQ, management plugin, dead-letter routing
Frontend	React 19 + Vite 8	Fast HMR, modern SPA with STOMP.js WebSocket client
Charting	Recharts 3.9	Declarative line/bar charts with React integration
Auth	Spring Security + JWT	HS256 tokens, BCrypt passwords, role-based access
API Docs	springdoc-openapi 2.8	Auto-generated Swagger UI
Testing	JUnit 5 + Testcontainers	Real Postgres, Redis, RabbitMQ in integration tests
Infrastructure	Docker Compose	5 services, health checks, one-command launch
Reverse Proxy	Nginx 1.27	Static serving + API/WebSocket proxying for frontend

Project Structure

pulseflow/
├── backend/                              # Spring Boot (Java 21)
│   ├── src/main/java/com/pulseflow/backend/
│   │   ├── auth/                         # JWT auth, registration, login, BCrypt
│   │   ├── events/                       # Ingestion, validation, RabbitMQ publish, rate limiting
│   │   ├── queue/                        # RabbitMQ topology: exchange, queue, DLQ, bindings
│   │   ├── analytics/                    # Consumer, aggregation, Redis R/W, analytics APIs
│   │   ├── dashboard/                    # WebSocket config + broadcast service
│   │   ├── monitoring/                   # Health checks, queue/Redis status, audit logs
│   │   ├── config/                       # Security, CORS, Swagger, global exception handler
│   │   └── common/                       # Shared enums, DTOs, constants
│   ├── src/test/                         # 16 test files (unit + Testcontainers integration)
│   ├── Dockerfile                        # Multi-stage: Temurin JDK 21 → JRE 21
│   └── pom.xml
│
├── frontend/                             # React SPA (Vite)
│   ├── src/
│   │   ├── components/
│   │   │   ├── LoginPage.jsx             # JWT authentication UI
│   │   │   ├── HeroStrip.jsx             # Live counters + canvas waveform animation
│   │   │   ├── DailyTrendChart.jsx       # 7-day event volume line chart
│   │   │   ├── TopEventsChart.jsx        # Event type distribution bars
│   │   │   ├── RecentEvents.jsx          # Live tail feed (15 latest events)
│   │   │   ├── EventsLog.jsx             # Paginated, filterable event browser
│   │   │   └── ConnectionStatus.jsx      # WebSocket reconnect indicator
│   │   └── services/
│   │       ├── api.js                    # REST client (JWT, all endpoints)
│   │       └── websocket.js              # STOMP/SockJS + auto-reconnection
│   ├── nginx.conf                        # Reverse proxy for /api + /ws
│   └── Dockerfile                        # Multi-stage: Node 22 → Nginx 1.27
│
├── docs/
│   ├── PRD.md                            # Product Requirements Document
│   ├── api.md                            # Full API specification
│   └── AGENT.md                          # Development guidelines
│
├── scratch/
│   └── send_events.py                    # Load testing / demo event generator
│
├── docker-compose.yml                    # 5 services with health checks
├── .env.example                          # Environment variable template
└── README.md

Quick Start

Prerequisites

Docker & Docker Compose
Ports 3000, 5432, 5672, 6379, 8082, 15672 available

One-Command Launch

git clone https://github.com/Abdul-Rafy2005/pulseflow.git
cd pulseflow
cp .env.example .env
docker compose up -d --build

Access the Stack

Service	URL	Credentials
Dashboard	`http://localhost:3000`	Register via UI
Backend API	`http://localhost:8082`	JWT / API Key
Swagger UI	`http://localhost:8082/swagger-ui.html`	—
RabbitMQ UI	`http://localhost:15672`	`guest` / `guest`

Send Test Events

# Register an admin account
curl -X POST http://localhost:8082/auth/register \
  -H "Content-Type: application/json" \
  -d '{"username":"admin","email":"admin@example.com","password":"password123"}'

# Fire 50 sample events (requires Python + requests)
python scratch/send_events.py 50 0.5

Local Development

1. Start Infrastructure Only

docker compose up -d postgres redis rabbitmq

2. Run the Backend

cd backend
./mvnw spring-boot:run

3. Run the Frontend

cd frontend
npm install
npm run dev

Dashboard dev server runs at http://localhost:5173 with API proxy to backend.

API Reference

Full interactive docs at /swagger-ui.html. See docs/api.md for complete specification.

Authentication

Method	Endpoint	Auth	Description
`POST`	`/auth/register`	None	Create an admin account
`POST`	`/auth/login`	None	Authenticate and receive JWT
`GET`	`/auth/profile`	JWT	Get current user profile

Event Ingestion

Method	Endpoint	Auth	Description
`POST`	`/events`	API Key	Ingest event → `202 Accepted` immediately
`GET`	`/events`	JWT	Paginated & filterable event list
`GET`	`/events/{id}`	JWT	Single event detail

Supported Event Types: LOGIN · LOGOUT · REGISTER · SEARCH · PAGE_VIEW · BUTTON_CLICK · PURCHASE · VIDEO_PLAY · LIKE · COMMENT · SHARE · DOWNLOAD

Analytics

Method	Endpoint	Auth	Description
`GET`	`/analytics/summary`	JWT	Today's aggregated totals (Redis-first)
`GET`	`/analytics/daily`	JWT	Time-series data for the last N days
`GET`	`/analytics/top-events`	JWT	Ranked event types by frequency
`GET`	`/analytics/top-users`	JWT	Most active users ranked
`GET`	`/analytics/realtime`	JWT	Live snapshot for dashboard initialization

System Monitoring

Method	Endpoint	Auth	Description
`GET`	`/health`	None	Liveness / readiness probe
`GET`	`/queue/status`	JWT	Queue depth, consumer count, DLQ size
`GET`	`/redis/status`	JWT	Connection health, key count, memory usage

WebSocket

Endpoint	Protocol	Topics
`/ws`	STOMP over SockJS	`/topic/events` · `/topic/stats`

Testing

cd backend
./mvnw test

Unit tests — service layer logic, validation, DTOs, rate limiting, auth, monitoring
Integration tests — full ingest → queue → consume → persist pipeline using Testcontainers (real Postgres, Redis, RabbitMQ containers)

Deployment

Docker Compose (Production-Ready)

docker compose up -d --build

Production Checklist

Set strong, unique values for JWT_SECRET and EVENTS_API_KEY
Restrict RabbitMQ management UI (port 15672) access
Enable TLS/HTTPS termination
Configure log aggregation for structured JSON logs
Set up database backups for PostgreSQL

Roadmap

Multi-tenant support — isolated analytics per client organization
Anomaly detection consumer — flag unusual event patterns
Alerting system — Slack/email notifications on queue backlog
Event replay endpoint — manually replay DLQ events
Grafana integration — metrics dashboards and alerting
Kubernetes manifests — horizontal scaling for production

What I Learned Building This

Challenge	Takeaway
Async without data loss	Transactional outbox pattern (publish after commit) prevents phantom queue messages
Idempotent consumers	Checking event status before processing prevents duplicate side effects
Cache invalidation	TTL-based expiry + Postgres as source of truth = no stale data headaches
Real-time + fallback	WebSocket is great until it isn't — polling fallback keeps the UI alive
Failure is not optional	DLQ + retry + audit logging means no event is ever silently dropped
Docker Compose orchestration	Health checks + `depends_on: condition: service_healthy` = reliable startup order

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'feat: add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

License

This project is open source under the MIT License.

Built by Abdul Rafy

If this project helped you or you found it interesting, please consider giving it a ⭐

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
.idea		.idea
backend		backend
docs		docs
frontend		frontend
scratch		scratch
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
architecture.png		architecture.png
docker-compose.yml		docker-compose.yml

Folders and files

Latest commit

History

Repository files navigation

Real-Time Event-Driven Analytics Platform

At a Glance

Why PulseFlow?

Architecture

System Pipeline Architecture

System Overview

Event Lifecycle

Failure Handling Flow

Database Schema

Key Design Decisions

Tech Stack

Project Structure

Quick Start

Prerequisites

One-Command Launch

Access the Stack

Send Test Events

Local Development

1. Start Infrastructure Only

2. Run the Backend

3. Run the Frontend

API Reference

Authentication

Event Ingestion

Analytics

System Monitoring

WebSocket

Testing

Deployment

Docker Compose (Production-Ready)

Production Checklist

Roadmap

What I Learned Building This

Contributing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages