fix(app_utils): fix apm capture fastapi calls, reorder apm middle ware

[alip990]

Description

Elastic APM was capturing zero transactions for any FastAPI service that also enabled the Prometheus metric interceptor (PROMETHEUS.IS_ENABLED=true). Errors were still reported, and functions (which use the @async_capture_transaction decorator, not FastAPI middleware) were unaffected — so the breakage looked intermittent and service-specific.

Root cause: in AppUtils.create_fastapi_app the Elastic APM middleware was registered before the Prometheus metric interceptor. Because FastAPI.add_middleware prepends to the middleware list, the last-registered middleware runs outermost. This left the order as:

FastAPIMetricInterceptor (Starlette BaseHTTPMiddleware)   ← outermost
    └── ElasticAPM (elasticapm.contrib.starlette)
            └── CORS → routing → endpoint

The fix swaps the registration order so Elastic APM is registered last and therefore becomes the outermost middleware. No API or configuration changes; purely middleware registration order.

Fixes # (issue)

Type of change

• Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Minimal Starlette app using the real elasticapm.contrib.starlette.ElasticAPM with a queue counter (transactions queued to the agent), DISABLE_SEND=True:

[APM outermost (no BaseHTTPMiddleware wrapper)] transactions queued = 3
[BaseHTTPMiddleware wraps APM (previous order)] transactions queued = 0

After the fix, services with both Prometheus and Elastic APM enabled capture request transactions again.

• [*] Run a FastAPI app with both ELASTIC_APM.IS_ENABLED=true and PROMETHEUS.IS_ENABLED=true; confirm request transactions appear in APM under transaction type request.
• [*] Confirm Prometheus HTTP metrics are still emitted.
• [*] Confirm make lint / make behave pass.

Checklist:

• [*] My code follows the style guidelines of this project
• [*] I have performed a self-review of my own code
• [*] I have commented my code, particularly in hard-to-understand areas
• [*] I have made corresponding changes to the documentation
• [*] My changes generate no new warnings
• [*] I have added tests that prove my fix is effective or that my feature works
• [*] New and existing unit tests pass locally with my changes
• [*] Any dependent changes have been merged and published in downstream modules
• [*] I have checked my code and corrected any misspellings

Additional context

Why this caused every transaction to be dropped

1. Elastic APM tracks the "current transaction" in a ContextVar.
   The Starlette ElasticAPM middleware calls client.begin_transaction("request")
   (which does ContextVar.set(transaction)) on the way in, and
   client.end_transaction(name, result) (which does ContextVar.get() then
   finalizes and queues the transaction) on the way out. The begin and the end
   must run in the same execution context, or end_transaction cannot find the
   transaction and nothing is queued.

2. ContextVar writes do not propagate across asyncio tasks.
   A new task receives a copy of the context at creation time; a ContextVar.set()
   inside that task is invisible to the parent and to sibling tasks.

3. Starlette's BaseHTTPMiddleware runs the downstream app in a separate task.
   To bridge the streaming ASGI send/receive into its request/response model,
   BaseHTTPMiddleware does task_group.start_soon(self.app, ...). The Prometheus
   FastAPIMetricInterceptor extends BaseHTTPMiddleware, so everything below it —
   including the ElasticAPM middleware — executed inside a forked context.

4. Result: the APM transaction was begun and ended in a forked context that was
   disconnected from (a) the agent's top-level tracing state established by
   elasticapm.instrument() patching Starlette.middleware_stack, and (b) the real
   send (BaseHTTPMiddleware substitutes its own send, so the
   http.response.start used to derive result/outcome was severed). The
   transaction object simply went out of scope without being collected — no error,
   no log, just silently missing transactions.

Standalone errors (capture_exception) do not depend on this ContextVar round-trip,
which is why errors still showed up while transactions did not.

The fix

# Before
FastAPIUtils.setup_elastic_apm(app, config)
FastAPIUtils.setup_metric_interceptor(app, config)

# After — Elastic APM registered last => outermost middleware
FastAPIUtils.setup_metric_interceptor(app, config)
FastAPIUtils.setup_elastic_apm(app, config)

With Elastic APM outermost, begin_transaction and end_transaction both run in
the original request task, before any BaseHTTPMiddleware forks a child task. The
transaction lives in one context for its whole lifetime, the wrapped send sees the
genuine http.response.start, and the transaction is finalized and queued. The
Prometheus BaseHTTPMiddleware now forks its child task below APM, which is
harmless to the request transaction.

This matches Elastic's own guidance that the Starlette/FastAPI APM middleware must be
the first (outermost) middleware and is incompatible with running underneath a
BaseHTTPMiddleware.

Impact

• Affects every FastAPI service created via AppUtils.create_fastapi_app with
  ELASTIC_APM.IS_ENABLED=true and PROMETHEUS.IS_ENABLED=true.
• Elastic APM now also observes CORS handling/preflight, which is the intended setup.