diff --git a/docs/.agent/last_run.json b/docs/.agent/last_run.json index 9d33f02..df0f41f 100644 --- a/docs/.agent/last_run.json +++ b/docs/.agent/last_run.json @@ -1,5 +1,5 @@ { "last_processed_ref": "v0.1.335", "date": "2025-02-24", - "notes": "Added background jobs recipe and expanded learning path with Module 10." + "notes": "Added Phase 4 (Enterprise Scale) to Learning Path, created Testing recipe, and updated File Uploads recipe." } diff --git a/docs/.agent/run_report_2025-02-24.md b/docs/.agent/run_report_2025-02-24.md index 09197fa..6f921cb 100644 --- a/docs/.agent/run_report_2025-02-24.md +++ b/docs/.agent/run_report_2025-02-24.md @@ -31,3 +31,37 @@ This run focuses on expanding the cookbook and refining the learning path to inc ## 4. Open Questions / TODOs - Investigate adding `rustapi-jobs` as a re-export in `rustapi-rs` for better "batteries-included" experience in future versions. - Consider adding more backend examples (Redis, Postgres) to the cookbook recipe when environment setup allows. + +--- + +# Docs Maintenance Run Report: 2025-02-24 (Run 2) + +## 1. Version Detection +- **Repo Version**: `v0.1.335` (Unchanged) +- **Result**: Continuing with Continuous Improvement phase. + +## 2. Changes Summary +This run focuses on "Enterprise Scale" documentation, testing strategies, and improving existing recipes. + +### New Content +- **Cookbook Recipe**: `docs/cookbook/src/recipes/testing.md` - Comprehensive guide to `rustapi-testing`, `TestClient`, and `MockServer`. +- **Learning Path Phase**: Added "Phase 4: Enterprise Scale" to `docs/cookbook/src/learning/curriculum.md`, covering Observability, Resilience, and High Performance. + +### Updates +- **File Uploads Recipe**: Rewrote `docs/cookbook/src/recipes/file_uploads.md` with a complete, runnable example using `Multipart` streaming and improved security guidance. +- **Cookbook Summary**: Added "Testing & Mocking" to `docs/cookbook/src/SUMMARY.md`. + +## 3. Improvement Details +- **Learning Path**: + - Added Modules 11 (Observability), 12 (Resilience & Security), 13 (High Performance). + - Added "Phase 4 Capstone: The High-Scale Event Platform". +- **Testing Recipe**: + - Detailed usage of `TestClient` for integration tests. + - Example of mocking external services with `MockServer`. +- **File Uploads**: + - Replaced partial snippets with a full `main.rs` style example. + - Clarified streaming vs buffering and added security warnings. + +## 4. Open Questions / TODOs +- **Status Page**: `recipes/status_page.md` exists but might need more visibility in the Learning Path (maybe in Module 11?). +- **Observability**: A dedicated recipe for OpenTelemetry setup would be beneficial (currently covered in crate docs). diff --git a/docs/cookbook/src/SUMMARY.md b/docs/cookbook/src/SUMMARY.md index a68ba42..4e1949f 100644 --- a/docs/cookbook/src/SUMMARY.md +++ b/docs/cookbook/src/SUMMARY.md @@ -33,6 +33,7 @@ - [JWT Authentication](recipes/jwt_auth.md) - [CSRF Protection](recipes/csrf_protection.md) - [Database Integration](recipes/db_integration.md) + - [Testing & Mocking](recipes/testing.md) - [File Uploads](recipes/file_uploads.md) - [Background Jobs](recipes/background_jobs.md) - [Custom Middleware](recipes/custom_middleware.md) diff --git a/docs/cookbook/src/learning/curriculum.md b/docs/cookbook/src/learning/curriculum.md index 283fd88..2d31ba6 100644 --- a/docs/cookbook/src/learning/curriculum.md +++ b/docs/cookbook/src/learning/curriculum.md @@ -170,6 +170,67 @@ This curriculum is designed to take you from a RustAPI beginner to an advanced u --- +## Phase 4: Enterprise Scale + +**Goal:** Build observable, resilient, and high-performance distributed systems. + +### Module 11: Observability +- **Prerequisites:** Phase 3. +- **Reading:** [Observability (Extras)](../crates/rustapi_extras.md#observability), [Structured Logging](../crates/rustapi_extras.md#structured-logging). +- **Task:** + 1. Enable `structured-logging` and `otel` features. + 2. Configure tracing to export spans to Jaeger (or console for dev). + 3. Add custom metrics for "active_users" and "jobs_processed". +- **Expected Output:** Logs are JSON formatted with trace IDs. Metrics endpoint exposes Prometheus data. +- **Pitfalls:** High cardinality in metric labels (e.g., using user IDs as labels). + +#### 🧠 Knowledge Check +1. What is the difference between logging and tracing? +2. How do you correlate logs across microservices? +3. What is the standard format for structured logs in RustAPI? + +### Module 12: Resilience & Security +- **Prerequisites:** Phase 3. +- **Reading:** [Resilience Patterns](../recipes/resilience.md), [Time-Travel Debugging](../recipes/replay.md). +- **Task:** + 1. Wrap an external API call with a `CircuitBreaker`. + 2. Implement `RetryLayer` for transient failures. + 3. (Optional) Use `ReplayLayer` to record and replay a tricky bug scenario. +- **Expected Output:** System degrades gracefully when external service is down. Replay file captures the exact request sequence. +- **Pitfalls:** Infinite retry loops or retrying non-idempotent operations. + +#### 🧠 Knowledge Check +1. What state does a Circuit Breaker have when it stops traffic? +2. Why is jitter important in retry strategies? +3. How does Time-Travel Debugging help with "Heisenbugs"? + +### Module 13: High Performance +- **Prerequisites:** Phase 3. +- **Reading:** [HTTP/3 (QUIC)](../recipes/http3_quic.md), [Performance Tuning](../recipes/high_performance.md). +- **Task:** + 1. Enable `http3` feature and generate self-signed certs. + 2. Serve traffic over QUIC. + 3. Implement response caching for a heavy computation endpoint. +- **Expected Output:** Browser/Client connects via HTTP/3. Repeated requests are served instantly from cache. +- **Pitfalls:** Caching private user data without proper keys. + +#### 🧠 Knowledge Check +1. What transport protocol does HTTP/3 use? +2. How does `simd-json` improve performance? +3. When should you *not* use caching? + +### 🏆 Phase 4 Capstone: "The High-Scale Event Platform" +**Objective:** Architect a system capable of handling thousands of events per second. +**Requirements:** +- **Ingestion:** HTTP/3 endpoint receiving JSON events. +- **Processing:** Push events to a `rustapi-jobs` queue (Redis backend). +- **Storage:** Workers process events and store aggregates in a database. +- **Observability:** Full tracing from ingestion to storage. +- **Resilience:** Circuit breakers on database writes. +- **Testing:** Load test the ingestion endpoint (e.g., with k6 or similar) and observe metrics. + +--- + ## Next Steps * Explore the [Examples Repository](https://github.com/Tuntii/rustapi-rs-examples). diff --git a/docs/cookbook/src/recipes/file_uploads.md b/docs/cookbook/src/recipes/file_uploads.md index 7400277..0cad0ef 100644 --- a/docs/cookbook/src/recipes/file_uploads.md +++ b/docs/cookbook/src/recipes/file_uploads.md @@ -1,9 +1,11 @@ # File Uploads -Handling file uploads efficiently is crucial. RustAPI allows you to stream `Multipart` data, meaning you can handle 1GB uploads without using 1GB of RAM. +Handling file uploads efficiently is crucial for modern applications. RustAPI provides a `Multipart` extractor that allows you to stream uploads, enabling you to handle large files (e.g., 1GB+) without consuming proportional RAM. ## Dependencies +Add `uuid` and `tokio` with `fs` features to your `Cargo.toml`. + ```toml [dependencies] rustapi-rs = "0.1.335" @@ -11,70 +13,121 @@ tokio = { version = "1", features = ["fs", "io-util"] } uuid = { version = "1", features = ["v4"] } ``` -## Streaming Upload Handler +## Streaming Upload Example -This handler reads the incoming stream part-by-part and writes it directly to disk (or S3). +Here is a complete, runnable example of a file upload server that streams files to a `./uploads` directory. ```rust use rustapi_rs::prelude::*; -use rustapi_rs::extract::Multipart; +use rustapi_rs::extract::{Multipart, DefaultBodyLimit}; use tokio::fs::File; use tokio::io::AsyncWriteExt; +use std::path::Path; + +#[tokio::main] +async fn main() -> Result<(), Box> { + // Ensure uploads directory exists + tokio::fs::create_dir_all("./uploads").await?; + + println!("Starting Upload Server at http://127.0.0.1:8080"); + + RustApi::new() + .route("/upload", post(upload_handler)) + // Increase body limit to 1GB (default is usually 2MB) + .layer(DefaultBodyLimit::max(1024 * 1024 * 1024)) + .run("127.0.0.1:8080") + .await +} -async fn upload_file(mut multipart: Multipart) -> Result { - // Iterate over the fields - while let Some(field) = multipart.next_field().await.map_err(|_| ApiError::BadRequest)? { +async fn upload_handler(mut multipart: Multipart) -> Result> { + let mut uploaded_files = Vec::new(); + + // Iterate over the fields in the multipart form + while let Some(mut field) = multipart.next_field().await.map_err(|_| ApiError::bad_request("Invalid multipart"))? { - let name = field.name().unwrap_or("file").to_string(); let file_name = field.file_name().unwrap_or("unknown.bin").to_string(); let content_type = field.content_type().unwrap_or("application/octet-stream").to_string(); - println!("Uploading: {} ({})", file_name, content_type); + // ⚠️ Security: Never trust the user-provided filename directly! + // It could contain paths like "../../../etc/passwd". + // Always generate a safe filename or sanitize inputs. + let safe_filename = format!("{}-{}", uuid::Uuid::new_v4(), file_name); + let path = Path::new("./uploads").join(&safe_filename); - // Security: Create a safe random filename to prevent overwrites or path traversal - let new_filename = format!("{}-{}", uuid::Uuid::new_v4(), file_name); - let path = std::path::Path::new("./uploads").join(new_filename); + println!("Streaming file: {} -> {:?}", file_name, path); // Open destination file - let mut file = File::create(&path).await.map_err(|e| ApiError::InternalServerError(e.to_string()))?; + let mut file = File::create(&path).await.map_err(|e| ApiError::internal(e.to_string()))?; - // Write stream to file chunk by chunk - // In RustAPI/Axum multipart, `field.bytes()` loads the whole field into memory. - // To stream efficiently, we use `field.chunk()`: - - while let Some(chunk) = field.chunk().await.map_err(|_| ApiError::BadRequest)? { - file.write_all(&chunk).await.map_err(|e| ApiError::InternalServerError(e.to_string()))?; + // Stream the field content chunk-by-chunk + // This is memory efficient even for large files. + while let Some(chunk) = field.chunk().await.map_err(|_| ApiError::bad_request("Stream error"))? { + file.write_all(&chunk).await.map_err(|e| ApiError::internal(e.to_string()))?; } + + uploaded_files.push(FileResult { + original_name: file_name, + stored_name: safe_filename, + content_type, + }); } - Ok(StatusCode::CREATED) + Ok(Json(UploadResponse { + message: "Upload successful".into(), + files: uploaded_files, + })) +} + +#[derive(Serialize, Schema)] +struct UploadResponse { + message: String, + files: Vec, +} + +#[derive(Serialize, Schema)] +struct FileResult { + original_name: String, + stored_name: String, + content_type: String, } ``` -## Handling Constraints +## Key Concepts -You should always set limits to prevent DoS attacks. +### 1. Streaming vs Buffering +By default, some frameworks load the entire file into RAM. RustAPI's `Multipart` allows you to process the stream incrementally using `field.chunk()`. +- **Buffering**: `field.bytes().await` (Load all into RAM - simple but dangerous for large files) +- **Streaming**: `field.chunk().await` (Load small chunks - scalable) -```rust -use rustapi_rs::extract::DefaultBodyLimit; +### 2. Body Limits +The default request body limit is often small (e.g., 2MB) to prevent DoS attacks. You must explicitly increase this limit for file upload routes using `DefaultBodyLimit::max(size)`. -let app = RustApi::new() - .route("/upload", post(upload_file)) - // Limit request body to 10MB - .layer(DefaultBodyLimit::max(10 * 1024 * 1024)); -``` +### 3. Security +- **Path Traversal**: Malicious users can send filenames like `../../system32/cmd.exe`. Always rename files or sanitize filenames strictly. +- **Content Type Validation**: The `Content-Type` header is client-controlled and can be spoofed. Do not rely on it for security execution checks (e.g., preventing `.php` execution). +- **Executable Permissions**: Store uploads in a directory where script execution is disabled. -## Validating Content Type +## Testing with cURL -Never trust the `Content-Type` header sent by the client implicitly for security (e.g., executing a PHP script uploaded as an image). +You can test this endpoint using `curl`: -Verify the "magic bytes" of the file content itself if strictly needed, or ensure uploaded files are stored in a non-executable directory (or S3 bucket). +```bash +curl -X POST http://localhost:8080/upload \ + -F "file1=@./image.png" \ + -F "file2=@./document.pdf" +``` -```rust -// Simple check on the header (not fully secure but good UX) -if let Some(ct) = field.content_type() { - if !ct.starts_with("image/") { - return Err(ApiError::BadRequest("Only images are allowed".into())); - } +Response: +```json +{ + "message": "Upload successful", + "files": [ + { + "original_name": "image.png", + "stored_name": "550e8400-e29b-41d4-a716-446655440000-image.png", + "content_type": "image/png" + }, + ... + ] } ``` diff --git a/docs/cookbook/src/recipes/testing.md b/docs/cookbook/src/recipes/testing.md new file mode 100644 index 0000000..7805be9 --- /dev/null +++ b/docs/cookbook/src/recipes/testing.md @@ -0,0 +1,141 @@ +# Testing Strategies + +RustAPI provides robust tools for testing your application, ensuring reliability from unit tests to full integration scenarios. + +## Dependencies + +Add `rustapi-testing` to your `Cargo.toml`. It is usually added as a dev-dependency. + +```toml +[dev-dependencies] +rustapi-testing = "0.1.335" +tokio = { version = "1", features = ["full"] } +``` + +## Integration Testing with TestClient + +The `TestClient` allows you to test your API handlers without binding to a network port. It interacts directly with the service layer, making tests fast and deterministic. + +```rust +use rustapi_rs::prelude::*; +use rustapi_testing::TestClient; + +#[rustapi_rs::get("/hello")] +async fn hello() -> &'static str { + "Hello, World!" +} + +#[tokio::test] +async fn test_hello_endpoint() { + // 1. Build your application + let app = RustApi::new().route("/hello", get(hello)); + + // 2. Create a TestClient + let client = TestClient::new(app); + + // 3. Send requests + let response = client.get("/hello").send().await; + + // 4. Assert response + assert_eq!(response.status(), 200); + assert_eq!(response.text().await, "Hello, World!"); +} +``` + +### Testing JSON APIs + +`TestClient` has built-in support for JSON serialization and deserialization. + +```rust +#[derive(Serialize, Deserialize, PartialEq, Debug)] +struct User { + id: u64, + name: String, +} + +#[rustapi_rs::post("/users")] +async fn create_user(Json(user): Json) -> Json { + Json(user) +} + +#[tokio::test] +async fn test_create_user() { + let app = RustApi::new().route("/users", post(create_user)); + let client = TestClient::new(app); + + let new_user = User { id: 1, name: "Alice".into() }; + + let response = client.post("/users") + .json(&new_user) + .send() + .await; + + assert_eq!(response.status(), 200); + + let returned_user: User = response.json().await; + assert_eq!(returned_user, new_user); +} +``` + +## Mocking External Services + +When your API calls external services (e.g., payment gateways, third-party APIs), you should mock them in tests to avoid network calls and ensure reproducibility. + +`rustapi-testing` provides `MockServer` for this purpose. + +```rust +use rustapi_testing::{MockServer, MockResponse}; + +#[tokio::test] +async fn test_external_integration() { + // 1. Start a mock server + let mock_server = MockServer::start().await; + + // 2. Define an expectation + mock_server.expect( + rustapi_testing::RequestMatcher::new() + .method("GET") + .path("/external-data") + ).respond_with( + MockResponse::new() + .status(200) + .body(r#"{"data": "mocked"}"#) + ).await; + + // 3. Use the mock server's URL in your app configuration + let mock_url = mock_server.url("/external-data"); + + // Simulating your app logic calling the external service + let client = reqwest::Client::new(); + let res = client.get(&mock_url).send().await.unwrap(); + + assert_eq!(res.status(), 200); + let body = res.text().await.unwrap(); + assert_eq!(body, r#"{"data": "mocked"}"#); +} +``` + +## Testing Authenticated Routes + +You can simulate authenticated requests by setting headers directly on the `TestClient` request builder. + +```rust +#[tokio::test] +async fn test_protected_route() { + let app = RustApi::new().route("/protected", get(protected_handler)); + let client = TestClient::new(app); + + let response = client.get("/protected") + .header("Authorization", "Bearer valid_token") + .send() + .await; + + assert_eq!(response.status(), 200); +} +``` + +## Best Practices + +1. **Keep Tests Independent**: Each test should setup its own app instance and state. `TestClient` is lightweight enough for this. +2. **Mock I/O**: Use `MockServer` for HTTP, and in-memory implementations for databases (e.g., `sqlite::memory:`) or traits for logic. +3. **Test Edge Cases**: Don't just test the "happy path". Test validation errors, 404s, and error handling.