refactor(server): establish owned generation lifecycle

[Graffioh]

Summary

Second PR in the server-refactor stack, directly following #691.

• move generation request/result types out of the broad backend header
• make retained GenerateRequest token sequences owned values
• add move-only Generation and GenerationSource channel endpoints
• bound request and token buffering and publish typed terminal results
• make LuceEngine own ModelBackend, the serving thread, mode selection, and orderly join
• make HttpServer borrow the engine while preserving the established serial worker and concurrent scheduler
• document the engine components and runtime flow as they exist today

Stack

This is PR 2 of 2:

1. refactor(server): seal backend construction behind a plan #691 — backend planning, owned configs, and grouped effective values, head a567baef9
2. refactor(server): establish owned generation lifecycle #692 — owned generation lifecycle and runtime owner, head 2e6309477

The #692 commit is directly parented by the current #691 head. Because the branches live in a fork, this PR targets main and temporarily displays #691's commits.

Review only this PR's 15-file delta: Graffioh/lucebox-hub@codex/backend-plan-boundary...codex/luce-engine-implementation

Current boundary

server_main
  -> LuceEngine owns ModelBackend
       -> owns one serving thread
       -> selects serial worker or concurrent scheduler once
  -> HttpServer borrows LuceEngine
       -> owns HTTP, sockets, response formatting, and the current job queue

The server no longer owns backend lifetime and worker-thread lifetime as separate facts. Shutdown asks LuceEngine to stop and join its selected loop before transport and cache teardown continues.

The generation channel is implemented and tested, but is not presented as a live HTTP entry point yet. HttpServer still supplies the established serving loops and retains the ServerJob queue. The remaining request-channel and LuceEngine::generate() cutover stays in this PR rather than creating another stacked PR.

The current components, ownership, request flows, and shutdown order are documented in server/docs/ENGINE_COMPONENTS.md. It intentionally documents the current system without proposals or open questions.

Verification

• direct ancestry: refactor(server): seal backend construction behind a plan #691 a567baef9 -> refactor(server): establish owned generation lifecycle #692 2e6309477
• dflash_server, the consolidated test_server_unit, and test_feature_gate build on HIP/gfx1151
• 451 model-free server, plan/feature, engine-lifecycle, and generation tests pass
• focused LuceEngineFixture and GenerationFixture suites pass
• typed KVFlash Off/Auto/Fixed behavior from refactor(server): seal backend construction behind a plan #691 remains preserved
• git diff --check

A broad all-target build reaches and builds dflash_common, but stops in the unrelated existing HIP test test_rms_norm_hip because cudaErrorNoDevice is not defined by the compatibility shim.

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 23 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

Review completed against the latest diff

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 53 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Graffioh]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@Graffioh I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>