diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000..e976efb --- /dev/null +++ b/SETUP.md @@ -0,0 +1,341 @@ +# EKA Setup Guide + +Step-by-step guide to get EKA running on any machine with full functionality — AI chat, document ingestion, knowledge graph, and OAuth2 login. + +## Prerequisites + +| Requirement | Version | Check | +|---|---|---| +| Java JDK | 21+ | `java -version` | +| Docker | Latest | `docker --version` | +| Docker Compose | v2+ | `docker compose version` | +| Git | Latest | `git --version` | + +## Quick Start (5 minutes) + +Gets you a running backend with mock AI responses and seeded demo data — zero API keys needed. + +```bash +# 1. Clone +git clone && cd eka-backend + +# 2. Start infrastructure +docker compose up -d + +# 3. Build +./gradlew build + +# 4. Run with local profile (mock AI, seeded data) +./gradlew :eka-app:bootRun --args='--spring.profiles.active=local' +``` + +The app boots at `http://localhost:8080`. The `local` profile seeds demo data and prints a dev JWT token to the console for testing. + +```bash +# Health check +curl http://localhost:8080/actuator/health +``` + +## Full Setup (all features) + +### Step 1: Clone & prepare + +```bash +git clone +cd eka-backend +cp .env.example .env +``` + +### Step 2: Generate JWT keys + +The `dev` profile generates ephemeral keys at startup, but for persistence across restarts or for the `default` (production) profile, generate RS256 key pair: + +```bash +openssl genrsa -out priv.pem 2048 +openssl rsa -in priv.pem -pubout > pub.pem +``` + +Copy the contents into `JWT_PRIVATE_KEY` and `JWT_PUBLIC_KEY` in `.env` (as a single-line PEM). + +### Step 3: Set up API keys + +Edit `.env` with your credentials: + +```bash +# Required for chat / embeddings +OPENAI_API_KEY=sk-... +ANTHROPIC_API_KEY=sk-ant-... + +# Required for hybrid search reranking (optional, improves results) +COHERE_API_KEY=... + +# Alternative embedding provider (optional) +VOYAGE_API_KEY=... +``` + +> **No API keys?** Use the `dev` profile — starts with mock LLM/embedding providers so you can explore the UI without any external service. + +### Step 4: Configure OAuth2 (for login) + +Create OAuth apps at [Google Cloud Console](https://console.cloud.google.com) and [GitHub Developer Settings](https://github.com/settings/developers). Set the redirect URI to: + +``` +http://localhost:8080/login/oauth2/code/google +http://localhost:8080/login/oauth2/code/github +``` + +Add to `.env`: + +```bash +GOOGLE_CLIENT_ID=xxx.apps.googleusercontent.com +GOOGLE_CLIENT_SECRET=... +GITHUB_CLIENT_ID=... +GITHUB_CLIENT_SECRET=... +``` + +> **No OAuth2?** The `local` profile sets `eka.oauth2.token-delivery=redirect` and prints a dev JWT at startup — you can test all APIs with that token. + +### Step 5: Start infrastructure + +```bash +docker compose up -d +``` + +This starts: + +| Service | Image | Port | Purpose | +|---|---|---|---| +| PostgreSQL 16 + pgvector | `pgvector/pgvector:pg16` | 5432 | Primary DB + vector storage | +| Redis 7 | `redis:7-alpine` | 6379 | Cache, sessions, rate limiting | +| Kafka 7.6 | `confluentinc/cp-kafka:7.6.0` | 9092 | Ingestion pipeline | +| Neo4j 5 | `neo4j:5-community` | 7687 / 7474 | Knowledge graph (optional) | + +Verify everything is healthy: + +```bash +docker compose ps +``` + +### Step 6: Build + +```bash +./gradlew build +``` + +This compiles all 10 modules, runs unit tests, OWASP dependency check, and JaCoCo coverage verification (≥30%). + +> **Build fails on OWASP?** Check `config/dependency-check-suppressions.xml`. If it's a false positive CVE, add a suppression entry. You can also skip: `./gradlew build -x dependencyCheckAnalyze`. + +### Step 7: Run + +Choose a profile based on what you need: + +```bash +# Profile: local — mock AI, seeded data, no API keys needed +./gradlew :eka-app:bootRun --args='--spring.profiles.active=local' + +# Profile: dev — debug logging, no rate limiting, mock AI fallbacks +./gradlew :eka-app:bootRun --args='--spring.profiles.active=dev' + +# Profile: default — production config, full AI, rate limiting on +./gradlew :eka-app:bootRun +``` + +The app starts on `http://localhost:8080`. + +### Step 8: Start the frontend (separate terminal) + +```bash +# Clone and run the frontend +cd ../eka-frontend +npm ci +npm run dev +``` + +The frontend starts on `http://localhost:5173` and proxies API calls to the backend. + +> Without the frontend, you can interact via `curl`, `httpie`, or any REST client. A dev JWT is printed to the backend console when using the `local` profile. + +## Profiles Explained + +| Profile | JWT | AI/LLM | Kafka | Rate Limiting | Use Case | +|---|---|---|---|---|---| +| `default` | From env vars | Real providers | Active | On | Production | +| `dev` | Ephemeral (auto-generated) | Mock fallbacks | Active | Off | Development | +| `local` | Ephemeral (auto-generated) | Disabled | Disabled | Off | Quick eval / demo | + +The `local` profile also seeds demo data via `MockDataInitializer`: +- 1 admin user (`demo@eka.dev`) +- 3 sources (GitHub repo, Swagger spec, Confluence space) +- 5 documents with vector embeddings +- 1 conversation with 4 messages + +## Verifying Each Feature + +### Health + +```bash +curl http://localhost:8080/actuator/health +# → {"status":"UP"} +``` + +### Chat (SSE streaming) + +```bash +curl -N http://localhost:8080/api/chat/stream \ + -H "Content-Type: application/json" \ + -d '{"message":"What services does this system have?","conversationId":"demo"}' +``` + +### Auth (JWT) + +```bash +# Use the dev JWT printed in the console (local profile), or login: +curl -X POST http://localhost:8080/api/auth/login \ + -H "Content-Type: application/json" \ + -d '{"email":"demo@eka.dev","password":"demo"}' +``` + +### Ingestion + +```bash +curl -X POST http://localhost:8080/api/ingestion/trigger \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"sourceId":"demo-github","sourceType":"GITHUB"}' +``` + +### Search + +```bash +curl "http://localhost:8080/api/search?q=payment+gateway&topK=5" \ + -H "Authorization: Bearer " +``` + +## Environment Variables Reference + +### Required (no default — app fails without these for full functionality) + +| Variable | Config Property | Description | +|---|---|---| +| `DB_PASSWORD` | `spring.datasource.password` | PostgreSQL password | +| `NEO4J_PASSWORD` | `spring.neo4j.authentication.password` | Neo4j password | +| `OPENAI_API_KEY` | `spring.ai.openai.api-key` | OpenAI key (chat + embeddings) | +| `ANTHROPIC_API_KEY` | `spring.ai.anthropic.api-key` | Anthropic key (primary LLM) | +| `JWT_PRIVATE_KEY` | `eka.jwt.private-key` | RS256 private key (PEM, single-line) | +| `JWT_PUBLIC_KEY` | `eka.jwt.public-key` | RS256 public key (PEM, single-line) | +| `GOOGLE_CLIENT_ID` | `spring.security.oauth2.client.registration.google.client-id` | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | `spring.security.oauth2.client.registration.google.client-secret` | Google OAuth client secret | +| `GITHUB_CLIENT_ID` | `spring.security.oauth2.client.registration.github.client-id` | GitHub OAuth client ID | +| `GITHUB_CLIENT_SECRET` | `spring.security.oauth2.client.registration.github.client-secret` | GitHub OAuth client secret | + +> The `dev` and `local` profiles generate ephemeral JWT keys — you can skip `JWT_PRIVATE_KEY`/`JWT_PUBLIC_KEY` during development. The `local` profile doesn't need API keys or OAuth2 credentials. + +### Optional (sensible defaults provided) + +| Variable | Default | Description | +|---|---|---| +| `DB_HOST` | `localhost` | PostgreSQL host | +| `DB_USER` | `eka` | PostgreSQL user | +| `REDIS_HOST` | `localhost` | Redis host | +| `KAFKA_BOOTSTRAP` | `localhost:9092` | Kafka bootstrap servers | +| `NEO4J_URI` | `bolt://localhost:7687` | Neo4j connection URI | +| `NEO4J_USERNAME` | `neo4j` | Neo4j user | +| `OLLAMA_BASE_URL` | `http://localhost:11434` | Local Ollama endpoint | +| `COHERE_API_KEY` | _(empty)_ | Cohere API key (reranking) | +| `VOYAGE_API_KEY` | _(empty)_ | Voyage AI embedding key | +| `TRACING_SAMPLE_RATE` | `0.1` | OpenTelemetry trace sample rate | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` | OTLP collector endpoint | +| `OAUTH2_FRONTEND_URL` | `http://localhost:5173` | Frontend URL for OAuth redirect | +| `CORS_ALLOWED_ORIGINS` | `http://localhost:5173,http://localhost:3000` | Allowed CORS origins | + +## Connectors (Document Ingestion) + +EKA can ingest documents from 8 source types. Each requires its own config stored as JSONB in the `sources.config` column: + +| Connector | Source Type | Config Fields | +|---|---|---| +| GitHub | `GITHUB` | `token`, `branch`, `allowedExtensions` | +| GitLab | `GITLAB` | `token`, `projectId` | +| Confluence | `CONFLUENCE` | `token`, `spaceKey`, `baseUrl` | +| Jira | `JIRA` | `token`, `jql`, `baseUrl` | +| Swagger/OpenAPI | `SWAGGER` | URL of the OpenAPI spec | +| PDF | `PDF` | URL or file path | +| Markdown | `MARKDOWN` | URL or file path | +| Web pages | `WEB` | `maxDepth=2`, `maxPages=50` | + +## Database Migrations + +Flyway runs 9 migrations automatically on startup: + +| Migration | Description | +|---|---| +| `V1` | Initial schema — pgvector extension, users, sources, documents, chunks (with vector column), conversations, messages, feedback | +| `V2` | Prompt templates table + 2 default templates | +| `V3` | Failed jobs table (DLQ) | +| `V4` | Document metadata columns (repository, api_name) | +| `V5` | Reranker signals table (Cohere fine-tuning feedback) | +| `V6` | Source teams for multi-tenancy | +| `V7` | Team name on chunks and documents | +| `V8` | Performance indexes | +| `V9` | Password hash column (email+password auth) | + +> `ddl-auto: validate` — Hibernate validates entities against the Flyway-managed schema but never modifies it. All schema changes go through Flyway. + +## Troubleshooting + +| Symptom | Likely Cause | Fix | +|---|---|---| +| `Failed to bind properties under 'spring.datasource.password'` | `DB_PASSWORD` not set | Add to `.env` or use `local` profile | +| `Connection refused: localhost:5432` | PostgreSQL not running | `docker compose up -d postgres` | +| `No bean named 'kafkaListenerContainerFactory'` | Kafka not available | Start Kafka: `docker compose up -d kafka` or use `local` profile | +| Chat returns "I'm a mock LLM" | No API keys configured | Set `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` | +| `401 Unauthorized` on API calls | Missing or expired JWT | Use `local` profile and copy the dev JWT from console | +| Build fails with `CVSS >= 7` | OWASP found a vulnerability | Check `config/dependency-check-suppressions.xml` or skip with `-x dependencyCheckAnalyze` | +| `Could not find spring-ai-bom:1.0.0-M6` | Milestone repo not configured | Ensure `mavenCentral()` and `repo.spring.io/milestone` are in repositories | + +## Architecture Overview + +``` +┌──────────────────────────────────────────────────────────┐ +│ React UI (Vite — port 5173) │ +│ Chat │ Search │ Admin │ Source Viewer │ +└──────────────────────────┬───────────────────────────────┘ + │ REST / SSE + ┌───────────▼──────────────────────┐ + │ eka-backend (port 8080) │ + │ Spring Boot 3.4 + WebFlux │ + │ │ + │ ┌─────────────────────────────┐ │ + │ │ eka-web (controllers/SSE) │ │ + │ ├─────────────────────────────┤ │ + │ │ eka-chat │ │ + │ │ eka-retrieval │ │ + │ │ eka-ingestion │ │ + │ │ eka-embedding │ │ + │ │ eka-auth │ │ + │ │ eka-graph │ │ + │ ├─────────────────────────────┤ │ + │ │ eka-common (models/ports) │ │ + │ └─────────────────────────────┘ │ + └──────────┬──────────┬─────────────┘ + │ │ + ┌───────────▼──┐ ┌────▼──────────┐ + │ PostgreSQL │ │ Kafka │ + │ + pgvector │ │ (ingestion) │ + └──────────────┘ └───────────────┘ + ┌───────────┐ ┌────────────┐ + │ Redis │ │ Neo4j │ + │ (cache) │ │ (knowledge │ + └───────────┘ │ graph) │ + └────────────┘ +``` + +## Next Steps + +- [Architecture deep-dive](docs/01-architecture-overview.md) +- [Authentication module](docs/02-auth-module.md) +- [Database schema](docs/03-database-schema.md) +- [Ingestion pipeline](docs/04-ingestion-pipeline.md) +- [Chat & RAG](docs/05-chat-module.md) +- [Deployment config](docs/10-deployment-config.md)