Skip to content

docs: comprehensive README update with security, testing, and TeamCode info #385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ElioNeto merged 1 commit into from
May 26, 2026
+175 −63
Merged

docs: comprehensive README update with security, testing, and TeamCode info #385

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
238 changes: 175 additions & 63 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<p align="center">
<img src="docs/assets/logo.png" width="180" alt="ApexStore Logo">
<p align="center">
<img src="docs/assets/logo.png" width="180" alt="ApexStore Logo">
</p>

<h1 align="center">ApexStore</h1>
Expand Down Expand Up @@ -27,14 +27,17 @@ ApexStore is a modern, Rust-based storage engine designed for write-heavy worklo

Built from the ground up using **SOLID principles**, it provides a production-grade storage solution that is easy to reason about, test, and maintain, while delivering the performance expected from a systems-level language.

> **🚀 Used in production by [TeamCode](https://github.com/ElioNeto/teamcode)** — an autonomous AI coding agent platform that relies on ApexStore for reliable, low-latency key-value storage.

## ⚖️ Why ApexStore?

While industry giants like RocksDB or LevelDB focus on extreme complexity, ApexStore offers:

- **Educational Clarity**: A clean, modular implementation of LSM-Tree that serves as a blueprint for high-performance systems.
- **Strict SOLID Compliance**: Leveraging Rust's ownership model to enforce clear boundaries between MemTable, WAL, and SSTable layers.
- **Observability First**: Built-in real-time metrics for memory, disk usage, and WAL health.
- **Modern Defaults**: Native LZ4 compression, Bloom Filters, and 35+ tunable parameters via environment variables.
- **Modern Defaults**: Native LZ4 compression, Bloom Filters, encryption-at-rest (AES-GCM), and 45+ tunable parameters via environment variables.
- **Security Hardened**: TLS/HTTPS support, CORS enforcement, rate limiting, per-IP connection limits, audit logging, and CSRF protection.

## 📊 Performance Benchmarks

Expand All @@ -49,40 +52,6 @@ While industry giants like RocksDB or LevelDB focus on extreme complexity, ApexS

<!-- BENCHMARK_RESULTS_END -->



































### 📋 YCSB Mixed Workload — `mixed_bench`

*Measured on **Intel Core i5-9300H @ 2.40GHz**, 16 GB DDR4 2667 MHz, HDD SATA 1TB (v2.1.39) — `cargo bench --bench mixed_bench -- --sample-size 10`*
Expand Down Expand Up @@ -121,17 +90,39 @@ While industry giants like RocksDB or LevelDB focus on extreme complexity, ApexS
## ✨ Key Features

### 🛠️ Storage Engine
- **MemTable**: In-memory BTreeMap with configurable size limits.
- **Write-Ahead Log (WAL)**: ACID-compliant durability with configurable sync modes.
- **SSTable V2**: Block-based storage with Sparse Indexing and LZ4 Compression.
- **MemTable**: Concurrent `DashMap`-backed in-memory store — lock-free reads and writes to different keys.
- **Write-Ahead Log (WAL)**: ACID-compliant durability with configurable sync interval and group commit support.
- **SSTable V2**: Block-based storage with Sparse Indexing, LZ4 Compression, and AES-GCM encryption.
- **Bloom Filters**: Drastically reduces unnecessary disk I/O for read operations.
- **Crash Recovery**: Automatic WAL replay on startup to ensure zero data loss.
- **Crash Recovery**: Automatic WAL replay on startup + SSTable auto-repair (truncated files detected and quarantined).
- **Encryption at Rest**: AES-256-GCM encryption enabled by default with configurable keys.
- **Range Deletion**: Efficient range tombstone support with compaction-aware filtering.

### 🔐 Security
- **TLS/HTTPS**: Built-in rustls-based HTTPS with configurable certificates and ports.
- **Authentication**: Bearer token-based auth enabled by default.
- **CORS**: Configurable cross-origin resource sharing with restrictive defaults.
- **Rate Limiting**: Sharded per-endpoint rate limiting with per-IP connection limits.
- **Audit Logging**: Structured audit events with principal tracking for every API operation.
- **CSRF Protection**: Content-Type guard middleware for state-changing requests.
- **Secrets Management**: Constant-time token comparison to prevent timing attacks.
- **File Permissions**: Data files created with 0600/0700 permissions (owner-only access).

### 🔌 Access Patterns
- **Interactive CLI**: REPL interface for development and debugging.
- **REST API**: Full HTTP API with JSON payloads for microservices.
- **Batch Operations**: Efficient bulk inserts and updates.
- **Search Capabilities**: Prefix and substring search (Optimized iterators coming in v2.0).
- **Interactive CLI**: REPL interface with token management (`token create`, `token list`, `token revoke`).
- **REST API**: Full HTTP API with JSON payloads, batch operations, and paginated scans.
- **Admin Dashboard**: Real-time web dashboard with live metrics and auto-refresh via fetch().
- **WebSocket Sync**: Real-time bidirectional sync with authentication.
- **GraphQL API**: Playground with production guard (disabled when auth enabled).
- **Change Data Capture (CDC)**: HTTP webhook delivery with configurable retry, auth, and timeout.

### 🔬 Testing Infrastructure
- **Unit Tests**: 550+ unit tests covering all engine operations.
- **Property-Based Tests**: `proptest` for engine invariants (put/get/delete roundtrip, multi-key independence).
- **Fuzz Testing**: `cargo-fuzz` targets for WAL frame format and SSTable block decoding.
- **Chaos Testing**: Fault injection tests for I/O failures, corruption handling, and crash recovery.
- **Randomized Testing**: Competitive test with reference `HashMap` model for linearizability verification.
- **Integration Tests**: SSTable roundtrip, CLI pagination, restart recovery, stress simulation.

## 🏗️ Architecture

Expand All @@ -142,44 +133,77 @@ graph TB
subgraph "Interface Layer"
CLI[CLI / REPL]
API[REST API Server]
WS[WebSocket Sync]
end

subgraph "Security Layer"
TLS[TLS/HTTPS]
Auth[Bearer Auth]
RateLimit[Rate Limiter]
Audit[Audit Log]
CORS[CORS Middleware]
end

subgraph "Core Domain"
Engine[LSM Engine]
MemTable[MemTable<br/>BTreeMap]
MemTable[MemTable<br/>DashMap Concurrent]
LogRecord[LogRecord<br/>Data Model]
Compaction[Compaction<br/>Strategy]
end

subgraph "Storage Layer"
WAL[Write-Ahead Log<br/>Durability]
SST[SSTable Manager<br/>V2 Format]
Builder[SSTable Builder<br/>Compression]
Builder[SSTable Builder<br/>LZ4 + AES-GCM]
Quarantine[Quarantine<br/>Auto-Repair]
end

subgraph "Infrastructure"
Codec[Serialization<br/>Bincode]
Metrics[Prometheus Metrics]
Error[Error Handling]
Config[Configuration<br/>Environment]
Degradation[Degradation<br/>Manager]
end

subgraph "Testing"
Proptest[Property Tests]
Fuzz[Fuzz Testing]
Chaos[Chaos Testing]
end

CLI --> Engine
API --> Engine
CLI --> Auth --> Engine
API --> TLS --> Auth --> Engine
WS --> Auth --> Engine
Engine --> WAL
Engine --> MemTable
MemTable -->|Flush| Builder
Builder --> SST
Engine -->|Read| MemTable
Engine -->|Read| SST
SST -->|Corrupt| Quarantine
WAL -.->|Recovery| MemTable

Engine --> Compaction
Engine --> Degradation
Engine --> Config
Engine --> Metrics
SST --> Codec
Builder --> Codec
WAL --> Codec

API --> RateLimit
API --> Audit
API --> CORS

style Engine fill:#f9a,stroke:#333,stroke-width:3px
style WAL fill:#9cf,stroke:#333,stroke-width:2px
style SST fill:#9cf,stroke:#333,stroke-width:2px
style TLS fill:#6c6,stroke:#333,stroke-width:2px
style Quarantine fill:#fc6,stroke:#333,stroke-width:2px
style Proptest fill:#cfc,stroke:#333,stroke-width:1px
style Fuzz fill:#cfc,stroke:#333,stroke-width:1px
style Chaos fill:#cfc,stroke:#333,stroke-width:1px
```

## 🚀 Quick Start
Expand All @@ -201,6 +225,15 @@ cargo run --release
# > stats
```

### Server with TLS
```bash
# Generate self-signed certificates
openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes

# Start HTTPS server
TLS_ENABLED=true TLS_CERT_PATH=cert.pem TLS_KEY_PATH=key.pem cargo run --release --bin server
```

## 🐳 Docker Deployment

Run ApexStore as a standalone API server:
Expand All @@ -214,7 +247,11 @@ docker run -d \
--name apexstore-server \
-p 8080:8080 \
-e MEMTABLE_MAX_SIZE=33554432 \
-e TLS_ENABLED=true \
-e TLS_CERT_PATH=/certs/cert.pem \
-e TLS_KEY_PATH=/certs/key.pem \
-v apexstore-data:/data \
-v ./certs:/certs:ro \
elioneto/apexstore:latest
```

Expand All @@ -224,29 +261,70 @@ docker run -d \
|--------|----------|-------------|
| `POST` | `/keys` | Insert/Update: `{"key": "k1", "value": "v1"}` |
| `GET` | `/keys/{key}` | Retrieve value |
| `GET` | `/stats/all` | Full telemetry (Memory, Disk, WAL) |
| `GET` | `/health/check` | Comprehensive health (uptime, engine mode, memtable stats) |
| `GET` | `/stats` | Engine telemetry (memory, disk, WAL, write/read amplification) |
| `DELETE` | `/keys/{key}` | Delete a key |
| `POST` | `/keys/batch` | Batch insert/update |
| `POST` | `/admin/flush` | Force memtable flush |
| `POST` | `/admin/compact` | Force compaction |

## 🔧 Configuration

ApexStore is configured via environment variables:

| Variable | Default | Description |
|----------|---------|-------------|
| `TLS_ENABLED` | `false` | Enable HTTPS |
| `TLS_CERT_PATH` | — | Path to TLS certificate (PEM) |
| `TLS_KEY_PATH` | — | Path to TLS private key (PEM) |
| `TLS_PORT` | `443` | HTTPS port |
| `AUTH_ENABLED` | `true` | Enable bearer token authentication |
| `CORS_ENABLED` | `false` | Enable CORS middleware |
| `RATE_LIMIT_ENABLED` | `true` | Enable rate limiting |
| `MAX_CONNECTIONS_PER_IP` | `100` | Max concurrent connections per IP |
| `ENCRYPTION_ENABLED` | `true` | Enable data encryption at rest |
| `WAL_SYNC_INTERVAL` | `4` | WAL fsync interval (writes between syncs) |

See [docs/CONFIGURATION.md](docs/CONFIGURATION.md) for a complete list.

## 📁 Project Structure

```
ApexStore/
├── src/
│ ├── core/ # LSM Engine, MemTable, Domain logic
│ ├── storage/ # WAL, SSTable V2, Block Builder
│ ├── infra/ # Codec, Error Handling, Config
│ ├── api/ # Actix-Web Server & Handlers
│ └── cli/ # REPL Implementation
│ ├── core/ # LSM Engine, MemTable (DashMap), Compaction, Domain logic
│ ├── storage/ # WAL, SSTable V2, Block Builder, Encryption, Prefix Compression
│ ├── infra/ # Codec, Error Handling, Config, Metrics, Scrubber, CDC
│ ├── api/ # Actix-Web Server, Auth, Rate Limiter, Audit, Health, CORS
│ └── cli/ # REPL + Token management commands
├── docs/ # Detailed documentation & Architecture
├── tests/ # Integration test suite
├── tests/ # Integration, Chaos, Proptest, Fuzz test suites
├── fuzz/ # cargo-fuzz targets for WAL and SSTable
└── Dockerfile # Multi-stage build
```

## 🧪 Testing & Quality

```bash
cargo test # Run all tests
cargo clippy -- -D warnings # Linting
cargo fmt # Formatting
# All tests (unit + integration + proptest + chaos)
cargo test --all-features

# Property-based tests (engine invariants)
cargo test proptest --all-features

# Chaos tests (fault tolerance)
cargo test chaos_ --all-features

# Fuzz testing (requires nightly)
cargo +nightly fuzz run wal -- -runs=10000
cargo +nightly fuzz run sstable -- -runs=10000

# Linting & formatting
cargo clippy --all-targets --all-features -- -D warnings
cargo fmt --all -- --check

# Security audit
cargo audit
```

## 🚀 CI/CD & Development Workflow
Expand All @@ -261,11 +339,23 @@ graph LR
D --> E[v2.1.X]
```

### PR Validation Pipeline

| Stage | What it checks |
|-------|----------------|
| `Rustfmt` | Code formatting |
| `Clippy` | Lint warnings (deny by default) |
| `Build and Docs` | Compilation + documentation generation |
| `Run Tests` | Full test suite (550+ tests) |
| `Security Audit` | `cargo audit` for dependency vulnerabilities |
| `Benchmarks` | Performance regression gates (Write, Read, Scan, Mixed, Stress) |
| `report-status` | Summary with root cause analysis on failure |

### Development Flow

1. **Create feature branch** from `main`
2. **Open PR** → CI runs `cargo fmt`, `clippy`, `test`, `build`
3. **Merge PR** → Auto-increments version in `Cargo.toml`, creates tag & GitHub release
2. **Open PR** → CI runs all stages above
3. **Merge PR** → Auto-increments version, creates tag & GitHub release

📖 **Read:** [`MIGRATION_GUIDE.md`](MIGRATION_GUIDE.md) for team workflow
📂 **Details:** [`.github/workflows/README.md`](.github/workflows/README.md)
Expand All @@ -276,9 +366,13 @@ graph LR
- [x] REST API & Feature Flags
- [x] Global Block Cache
- [x] Trunk-based CI/CD with auto-release
- [ ] **v2.2**: Storage iterators for range queries
- [ ] **v2.3**: Concurrent read optimization
- [x] **v2.2**: Concurrent read optimization (RwLock engine core)
- [x] **v2.3**: WAL I/O decoupling, DashMap MemTable
- [x] Security audit resolution (40+ issues): TLS, encryption, auth, rate limiting, CSRF, audit
- [x] Testing infrastructure: proptest, fuzz, chaos testing
- [ ] **v3.0**: Leveled/Tiered Compaction Strategies
- [ ] **v3.1**: Distributed replication & consensus
- [ ] **v3.2**: SQL query layer via Apache DataFusion

## 🤝 Contributing

Expand All @@ -295,6 +389,24 @@ Contributions are what make the open-source community an amazing place! Please c

Distributed under the MIT License. See `LICENSE` for more information.

## 💼 Powered By

<table>
<tr>
<td align="center">
<a href="https://github.com/ElioNeto/teamcode">
<img src="docs/assets/teamcode-logo.png" width="48" alt="TeamCode Logo" /><br />
<b>TeamCode</b>
</a>
</td>
<td>
<strong>TeamCode</strong> — an <a href="https://github.com/ElioNeto/teamcode">autonomous AI coding agent platform</a> —
uses ApexStore as its embedded storage engine for reliable, low-latency key-value storage.
Managing task state, context, and agent coordination data at scale.
</td>
</tr>
</table>

## 📧 Contact

**Elio Neto** - [GitHub](https://github.com/ElioNeto) - netoo.elio@hotmail.com
Expand Down
Loading