perf: pre-serialize environment-document bytes on poll

[gagantrivedi]

Summary

Move JSON serialization of the environment document off the request path. The released /api/v1/environment-document path runs serde_json::to_vec(&Arc<Value>) on every request and the handler then body.to_vec() clones the resulting Arc<[u8]> into a fresh Vec<u8>. Two allocations and a full JSON encoding per request — for a document that only changes on the polling refresh interval.

This PR moves the serialization to the cache layer:

• New EnvironmentsCache::get_environment_bytes returns Bytes.
• LocalMemEnvironmentsCache::put_environment produces the bytes once, alongside the parsed Arc<Value> and the pre-computed evaluation context.
• EnvironmentService::get_environment_bytes becomes a thin delegate.
• The route handler returns the Bytes directly as the response body — axum accepts Bytes with no copy.

Per-request work on the env-doc endpoint goes from "serialize a 1-11 MB JSON tree + memcpy" to "refcounted clone".

Measured on Fargate (1 vCPU / 2 GB)

Endpoint cache disabled. Image-vs-image A/B with the released flagsmith/edge-proxy-rs:0.1.1 baseline.

Small project (1 MB env-doc)

c	Baseline RPS	Fix RPS	Multiplier	Fix p50	Fix p99
10	343	543	1.58×	13 ms	201 ms
25	325	566	1.74×	36 ms	260 ms
50	339	556	1.64×	80 ms	300 ms
100	341	545	1.60×	172 ms	453 ms
200	342	525	1.54×	318 ms	1.81 s
500	310	502	1.62×	617 ms	10.29 s
1000	290	425	1.47×	1.34 s	17.69 s

Medium project (11 MB env-doc)

c	Baseline RPS	Fix RPS	Multiplier	Fix p50	Fix p99
10	21	51.7	2.46×	144 ms	428 ms
25	21	51.4	2.45×	451 ms	824 ms
50	19	51.1	2.69×	905 ms	4.64 s
100	22	50.3	2.29×	1.52 s	7.57 s

Medium gets a bigger relative win because at 11 MB per response the released path's per-request serde_json::to_vec was dominating CPU. After the fix, throughput is bandwidth-bound rather than CPU-bound and reaches roughly the same physical limit as the Python edge-proxy on the same task spec.

p99 caveat

At high concurrency (c≥500 small, c≥50 medium) the fix's p99 tail is worse than baseline. The bottleneck has shifted from CPU (serializing) to socket-write queueing on the response body — some requests get a fast slot, others queue behind 1 MB / 11 MB of in-flight bytes. p50 and p90 improve across the curve; p99 only matters past the useful operating range for the endpoint.

Test plan

• cargo build --release clean from origin/main
• cargo test --release --lib — all 19 unit tests pass
• cargo clippy --release --all-targets -- -D warnings clean
• End-to-end Fargate A/B with synthetic small (50 features, 750 overrides, 1 MB env-doc) and medium (200 features, 8.7K overrides, 11 MB env-doc) projects (numbers above)
• Manual smoke after redeploy: GET /api/v1/environment-document returns identical byte output as the baseline image for both project sizes
• Observe env-doc p50/p99 in staging do not regress

[gagantrivedi]

Code review

Found 1 issue:

1. put_environment runs serde_json::to_vec(&document) while holding all four RwLock write guards (environments, environment_bytes, contexts, identity_overrides). For large environment documents (the paginated, multi-MB case from fix: follow environment-document pagination Link headers #11), the serialize call adds milliseconds of CPU work to the critical section, blocking every concurrent reader of those locks (i.e. every in-flight flag-evaluation request). Since document is owned and not yet shared, to_vec could run before the locks are acquired and only the resulting Bytes inserted under the lock — keeping the perf win without widening the write window.

edge-proxy-rs/src/cache/environment.rs

Lines 74 to 128 in 84137d2

	async fn put_environment(&self, environment_key: &str, document: Value) -> bool {
	let mut environments = self.environments.write().await;
	let mut environment_bytes = self.environment_bytes.write().await;
	let mut contexts = self.contexts.write().await;
	let mut identity_overrides = self.identity_overrides.write().await;
	// Check if document changed
	let changed = environments
	.get(environment_key)
	.map(|existing| existing.as_ref() != &document)
	.unwrap_or(true);
	if changed {
	// Extract identity overrides
	if let Some(overrides_array) = document
	.get("identity_overrides")
	.and_then(|v| v.as_array())
	{
	let mut env_identities = HashMap::new();
	for override_obj in overrides_array {
	if let Some(identifier) =
	override_obj.get("identifier").and_then(|v| v.as_str())
	{
	env_identities.insert(identifier.to_string(), override_obj.clone());
	}
	}
	identity_overrides.insert(environment_key.to_string(), env_identities);
	}
	// Pre-compute the evaluation context
	if let Ok(environment) = serde_json::from_value::<Environment>(document.clone()) {
	let flagsmith_env: FlagsmithEnvironment = environment.to_flagsmith_environment();
	let context = environment_to_context(flagsmith_env);
	contexts.insert(environment_key.to_string(), context);
	}
	// Serialize once here so /environment-document requests are an Arc-clone.
	// Failure leaves the byte cache empty for this key — handler returns 503.
	match serde_json::to_vec(&document) {
	Ok(bytes) => {
	environment_bytes.insert(environment_key.to_string(), Bytes::from(bytes));
	}
	Err(err) => {
	error!(
	environment_key,
	error = %err,
	"failed to serialize environment document for byte cache"
	);
	environment_bytes.remove(environment_key);
	}
	}
	environments.insert(environment_key.to_string(), Arc::new(document));
	}

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}