feat(api): complete DynamoDB API gaps (UpdateItem, Batch, TransactGet, Table ops)

README.md

@@ -1,57 +1,98 @@
 # cloudDB
 
-`cloudDB` is a distributed, DynamoDB-compatible database system. It features a high-performance C++ LSM-tree storage engine, Zig Bloom filters, Multi-Paxos consensus, Gossip cluster membership, Percolator transactions, and a Spring Boot Java control plane.
+`cloudDB` is a distributed, DynamoDB-compatible database system. It features a high-performance Zig-based LSM-tree storage engine with Bloom filters, Multi-Paxos consensus, Gossip cluster membership, Percolator transactions, and a Spring Boot Java control plane.
 
 ---
 
 ## Prerequisites
 
 Ensure you have the following installed:
+* **Zig 0.16+** (for the storage engine)
 * **C++17 Compiler** (Clang or GCC)
 * **CMake** (version >= 3.14)
-* **Zig** (for Bloom filter compilation)
 * **JDK 17** & **Maven** (for the Java Control Plane)
 
 ---
 
 ## How to Build and Run
 
-### 1. Build and Run the C++ Storage Engine
+### 1. Build and Run the Zig Storage Engine
 The storage engine handles local LSM storage, Paxos consensus, and network requests.
 
 ```bash
-# Compile the C++ storage server
+# Build the Zig storage server
 cd storage-engine
-mkdir build && cd build
-cmake -DBUILD_TESTS=ON ..
-make -j$(sysctl -n hw.ncpu || nproc)
+zig build
 
 # Start the storage node
-./cloudDBServer --port 9099 --gossip-port 9098 --rf 1 --w 1 --r 1
+./zig-out/bin/cloudDBServer --port 9099 --gossip-port 9098 --rf 1 --w 1 --r 1
 ```
 
 ### 2. Build and Run the Java Control Plane
-The control plane exposes a DynamoDB-compatible REST API that translates HTTP requests to TCP requests for the C++ engine.
+The control plane exposes a DynamoDB-compatible REST API that translates HTTP requests to TCP requests for the Zig storage engine.
 
 ```bash
 # Start the Spring Boot REST application (exposes port 8080)
-cd ../../control-plane
+cd control-plane
 mvn clean install
 mvn spring-boot:run
 ```
 
 ---
 
+## Supported DynamoDB Operations
+
+cloudDB supports the following DynamoDB operations:
+
+| Operation | Type | Description |
+|-----------|------|-------------|
+| `PutItem` | Write | Write an item to a table |
+| `GetItem` | Read | Read a single item by primary key |
+| `UpdateItem` | Write | Update an existing item (SET, REMOVE, ADD) |
+| `DeleteItem` | Write | Delete an item (tombstone) |
+| `BatchGetItem` | Read | Read up to 25 items in a single request |
+| `BatchWriteItem` | Write | Write or delete up to 25 items |
+| `Query` | Read | Query a table by partition key and sort key conditions |
+| `Scan` | Read | Scan the entire table with optional filters |
+| `TransactWriteItems` | Write | Write multiple items atomically |
+| `TransactGetItems` | Read | Read multiple items transactionally |
+| `CreateTable` | DDL | Create a table with optional LSIs and GSIs |
+| `DeleteTable` | DDL | Delete a table |
+| `DescribeTable` | DDL | Get table metadata and schema |
+| `ListTables` | DDL | List all tables |
+
+### Expression Support
+
+**Filter Expressions:**
+- Comparison operators: `=`, `<>`, `>`, `<`, `>=`, `<=`
+- Logical operators: `AND`, `OR`, `NOT`
+- Functions: `begins_with()`, `contains()`, `attribute_exists()`, `attribute_not_exists()`
+- `BETWEEN` operator (e.g., `age BETWEEN 18 AND 65`)
+- `IN` operator (e.g., `status IN ('active', 'pending')`)
+
+**Update Expressions:**
+- `SET` — set attribute values
+- `REMOVE` — remove attributes
+- `ADD` — add to numbers or sets
+
+### Attribute Types
+- `S` (String)
+- `N` (Number)
+- `SS` (String Set)
+- `NS` (Number Set)
+
+---
+
 ## How to Run Tests
 
-### C++ Tests
-From the `storage-engine/build` directory, you can run all 35 tests (including unit, chaos, and fuzz tests):
+### Zig Tests
+From the `storage-engine` directory:
 ```bash
-ctest --output-on-failure
+zig build test
 ```
 
 ### Java Tests
-From the `control-plane` directory, run:
+From the `control-plane` directory:
 ```bash
 mvn test
 ```
@@ -60,7 +101,44 @@ mvn test
 
 ## Project Structure
 
-* **`control-plane/`**: Java Spring Boot application (REST APIs, HTTP endpoints, client connection pool).
-* **`storage-engine/`**: C++ LSM storage engine, Paxos, Gossip, and Percolator.
-  * `storage/bloom_filter.zig`: High-performance Bloom Filter written in Zig.
-* **`README.md`**: This documentation file.
+* **`control-plane/`**: Java Spring Boot application (REST API, DynamoDB-compatible endpoints, TCP client to storage engine)
+* **`storage-engine/`**: Zig storage engine
+  * `src/storage/` — LSM storage engine, memtable, SSTable, Bloom filters
+  * `src/distributed/` — Multi-Paxos, gossip, Percolator transactions, vector clocks
+  * `src/network/` — RPC server/client
+  * `src/recovery/` — Write-Ahead Log (WAL)
+* **`docs/`** — Architecture Decision Records (ADRs)
+* **`README.md`** — This documentation file
+
+---
+
+## Architecture
+
+```
+                    ┌─────────────────┐
+                    │   AWS CLI /     │
+                    │   HTTP Client   │
+                    └────────┬────────┘
+                             │ HTTP
+                             ▼
+                   ┌─────────────────────┐
+                   │  Java Control Plane │
+                   │  (Spring Boot)     │
+                   │  Port 8080          │
+                   └────────┬───────────┘
+                            │ TCP
+                            ▼
+              ┌─────────────────────────────┐
+              │     Zig Storage Engine       │
+              │  (LSM tree, Paxos, gossip)  │
+              │  Port 9099                   │
+              └─────────────────────────────┘
+```
+
+### Key Components
+
+1. **Java Control Plane** — DynamoDB-compatible REST API; translates HTTP to TCP RPC
+2. **Zig Storage Engine** — High-performance LSM-tree with memtable, SSTable, Bloom filters
+3. **Multi-Paxos** — Leader-based consensus for strong consistency
+4. **Percolator Transactions** — Distributed ACID semantics with snapshot isolation
+5. **Gossip Protocol** — Dynamic cluster membership and failure detection

control-plane/src/main/java/com/clouddb/controlplane/client/EngineTcpClient.java

@@ -63,6 +63,16 @@ public RpcReply getItem(String partitionKey, String table, String sortKey) throw
     return sendRequest(RpcType.GetItem, args.serialize());
   }
 
+  public RpcReply updateItem(
+      String partitionKey, String table, String sortKey, String updateOpsJson) throws IOException {
+    ItemArgs args = new ItemArgs();
+    args.partitionKey = partitionKey;
+    args.table = table;
+    args.sortKey = sortKey;
+    args.value = updateOpsJson;
+    return sendRequest(RpcType.UpdateItem, args.serialize());
+  }
+
   public RpcReply queryItems(
       String table,
       String partitionKey,
@@ -104,6 +114,45 @@ public RpcReply transactWrite(java.util.List<ItemArgs> writeItems) throws IOExce
     return sendRequest(RpcType.TransactWrite, baos.toByteArray());
   }
 
+  public RpcReply batchGetItem(java.util.List<ItemArgs> keys) throws IOException {
+    java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
+    java.io.DataOutputStream out = new java.io.DataOutputStream(baos);
+
+    int count = keys.size();
+    out.write(intToLittleEndian(count));
+    for (ItemArgs args : keys) {
+      out.write(args.serialize());
+    }
+
+    return sendRequest(RpcType.BatchGetItem, baos.toByteArray());
+  }
+
+  public RpcReply batchWriteItems(java.util.List<ItemArgs> items) throws IOException {
+    java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
+    java.io.DataOutputStream out = new java.io.DataOutputStream(baos);
+
+    int count = items.size();
+    out.write(intToLittleEndian(count));
+    for (ItemArgs args : items) {
+      out.write(args.serialize());
+    }
+
+    return sendRequest(RpcType.BatchWriteItem, baos.toByteArray());
+  }
+
+  public RpcReply transactGetItems(java.util.List<ItemArgs> getItems) throws IOException {
+    java.io.ByteArrayOutputStream baos = new java.io.ByteArrayOutputStream();
+    java.io.DataOutputStream out = new java.io.DataOutputStream(baos);
+
+    int count = getItems.size();
+    out.write(intToLittleEndian(count));
+    for (ItemArgs args : getItems) {
+      out.write(args.serialize());
+    }
+
+    return sendRequest(RpcType.TransactGetItems, baos.toByteArray());
+  }
+
   private byte[] intToLittleEndian(int value) {
     return new byte[] {
       (byte) (value & 0xff),

control-plane/src/main/java/com/clouddb/controlplane/client/RpcType.java

@@ -13,6 +13,9 @@ public enum RpcType {
   Query(10),
   Scan(11),
   TransactWrite(12),
+  BatchGetItem(13),
+  BatchWriteItem(14),
+  TransactGetItems(15),
   Error(255);
 
   private final int value;