refactor(config): replace bottlecap Config with upstream Config<LambdaConfig>#1251
Conversation
duncanista
left a comment
There was a problem hiding this comment.
Reviewed the migration diff (feat/use-datadog-agent-config-crate...feat/migrate-bottlecap-to-upstream-config). This is a mechanical refactor — Config → Config<LambdaConfig> with .ext. indirection on 19 Lambda fields, plus a PropConfig newtype to ferry the foreign Config<E> past the orphan rule. No correctness bugs found. The structural pieces (PropConfig impl, mod.rs rewrite, ~70 read-site rewrites, ~14 test struct-literal rewrites) all check out.
What I verified
- Field migration completeness. Grepped every one of the 19 Lambda field names across
bottlecap/src/andbottlecap/tests/. The only remaining bareconfig.<lambda_field>(no.ext.) accesses are insideLambdaConfig/LambdaConfigSourcestruct literals or comments. Nothing missed. - Test struct-literal rewrites. Every
Config { lambda_field: …, ..Default::default() }is correctly rewritten to nest the Lambda fields underext: LambdaConfig { …, ..Default::default() }with the outer..Config::default()(or..no_config.as_ref().clone()) preserved. Test intent is preserved in the two cases where a Lambda field was bundled with core fields (e.g.lifecycle/invocation/processor.rstest at the new line 2407 hasservice: Some("test-service".to_string())at the top level ANDserverless_appsec_enabled: truecorrectly placed underext:). metrics/enhanced/lambda.rstest_disabled / test_snapstart_restore_duration_metric_disabled. These overrideenhanced_metricsover a non-defaultno_configbase. The rewrite correctly uses..no_config.ext.clone()for the extension and..no_config.as_ref().clone()for the rest — both layers of "keep everything else from no_config" are preserved.mod.rsrewrite (2243 → 602 lines). Verified that nothing outside the deleted files references the droppedConfigBuilder/ConfigSource/ConfigError/merge_*macros. The re-exports cover every existingcrate::config::{env, flush_strategy, processing_rule, log_level, …}::Ximport I could find (lifecycle/invocation_times.rs,lifecycle/flush_control.rs,logs/processor.rs, etc.).PropConfigimpl parity. The newPropagationConfig for PropConfigimpl is a verbatim forward of the oldimpl PropagationConfig for Config(same emptiness checks, sameNonefor inject, same hard-coded512fordatadog_tags_max_length). Theself.0.fooderefs throughArc<Config>to the same fields as the priorself.fooaccess.- Stacked-PR base. Re-targeting onto main after #1249 lands should be clean — this PR's diff touches only files modified in this branch's HEAD commit (
e5cebae0), and #1249 doesn't touch the same lines in those files.
Findings are non-blocking nits — see inline comments.
Three non-blocking nits from the independent review: 1. PropConfig's inner Arc<Config> is no longer pub — keeps the wrapper boundary tight; the new() constructor is sufficient and no callers outside the module reach for the inner field directly. 2. PropConfig::new returns Arc<Self> instead of Self. Drops the redundant `Arc::new(...)` wrap at the single call site in traces/propagation/mod.rs. 3. Documents the hard-coded 512 in datadog_tags_max_length — it matches dd-trace-rs's default; bottlecap intentionally doesn't expose DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH. Saves the next reader a round-trip through upstream to confirm parity. No behavior change. All 505 lib tests still pass.
e5cebae to
561096a
Compare
Three non-blocking nits from the independent review: 1. PropConfig's inner Arc<Config> is no longer pub — keeps the wrapper boundary tight; the new() constructor is sufficient and no callers outside the module reach for the inner field directly. 2. PropConfig::new returns Arc<Self> instead of Self. Drops the redundant `Arc::new(...)` wrap at the single call site in traces/propagation/mod.rs. 3. Documents the hard-coded 512 in datadog_tags_max_length — it matches dd-trace-rs's default; bottlecap intentionally doesn't expose DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH. Saves the next reader a round-trip through upstream to confirm parity. No behavior change. All 505 lib tests still pass.
561096a to
a223aee
Compare
This comment has been minimized.
This comment has been minimized.
| #[must_use] | ||
| pub fn new(config: Arc<Config>) -> Arc<Self> { | ||
| Arc::new(Self(config)) | ||
| } |
There was a problem hiding this comment.
nit: new() usually returns Self, so returning Arc::new() feels a bit counterintuitive or I missed something?
There was a problem hiding this comment.
you are right that it is a bit unusual. i did it that way because every caller wraps it in an arc immediately, so returning arc skips one extra line at each call site. happy to switch it back to return self if you prefer the more conventional shape.
litianningdatadog
left a comment
There was a problem hiding this comment.
Approved with nit comment
|
Can you add some notes in the description about why we want to do this. |
f636493 to
2686077
Compare
…nt-config (#1249) ## Overview First step of migrating bottlecap's in-tree configuration module onto the shared `datadog-agent-config` crate (lives in [DataDog/serverless-components](https://github.com/DataDog/serverless-components/tree/main/crates/datadog-agent-config)). This PR adds the foundation only — it introduces a `LambdaConfig` extension struct (in `bottlecap/src/config/mod.rs`, alongside the existing legacy code) that implements the upstream `ConfigExtension` trait, plus a `LambdaConfigSource` deserialization shape that figment uses for both env-var and YAML loading (dual extraction). Nothing in bottlecap consumes the extension yet — the existing `bottlecap::config::Config` struct is untouched. The follow-up #1251 (stacked on this) replaces it with `Config<LambdaConfig>`, deletes the duplicated env.rs / yaml.rs / deserializer modules, and removes the legacy `#[macro_export]` `merge_*` macros at the top of `mod.rs` (`LambdaConfig::merge_from` deliberately uses fully-qualified upstream macro paths today to coexist with them). ### Why this shape The upstream crate is purpose-built for this migration — see PR DataDog/serverless-components#111 ("ConfigExtension trait") which landed exactly the extensibility hook we need. Each consumer (bottlecap here, future serverless agents elsewhere) supplies its own extension type for fields that don't generalize, while sharing one canonical implementation for all the core agent fields (site, api_key, logs/APM/OTLP/proxy/trace propagation, etc.). ### Fields in the extension The 19 Lambda-specific fields that have no upstream equivalent: - `api_key_secret_arn`, `kms_api_key`, `api_key_ssm_arn`, `api_key_secret_reload_interval` - `serverless_logs_enabled` (OR-merged with the `DD_LOGS_ENABLED` alias) - `serverless_flush_strategy` (custom `FlushStrategy` deserializer for `"end" | "end,N" | "periodically,N" | "continuously,N"`) - `enhanced_metrics`, `lambda_proc_enhanced_metrics` - `capture_lambda_payload`, `capture_lambda_payload_max_depth` - `compute_trace_stats_on_extension`, `span_dedup_timeout` - `dd_org_uuid` (sourced from `DD_ORG_UUID`, source field `org_uuid` → config field `dd_org_uuid`) - `serverless_appsec_enabled`, `appsec_rules`, `appsec_waf_timeout` - `api_security_enabled`, `api_security_sample_delay` - `custom_metrics_exclude_tags` (sourced from `DD_LAMBDA_CUSTOMER_METRICS_EXCLUDE_TAGS` / `lambda_customer_metrics_exclude_tags:` in YAML) Kept in the extension rather than upstreamed because they're Lambda-runtime concerns. `custom_metrics_exclude_tags` is arguably generalizable to other serverless targets but stays here for now; we can migrate it upstream when another consumer needs it. ### Pre-requisites that already merged - DataDog/serverless-components#135 — bumps libdatadog rev across all serverless-components crates from `0a3304c` to `48da0d8` to match bottlecap. - DataDog/serverless-components#136 — switches `datadog-agent-config`'s libdd transitive deps to `default-features = false` and exposes `https` / `fips` opt-in features. Without this, the FIPS dep tree was poisoned by an implicit `https` (ring) path. Bottlecap pins all three serverless-components crates (`dogstatsd`, `datadog-fips`, `datadog-agent-config`) to the merge SHA `bb4dedee` so cargo deduplicates `dogstatsd`. ## Testing 37 new tests in `bottlecap/src/config/mod.rs::lambda_config_tests` cover each extension field from both env vars and YAML where applicable, plus: - Per-field env round-trip (`api_key_secret_arn`, `kms_api_key`, `api_key_ssm_arn`, …) - YAML coverage for the two source-to-config renames: `org_uuid` → `dd_org_uuid` and `lambda_customer_metrics_exclude_tags` → `custom_metrics_exclude_tags` - `DD_LOGS_ENABLED` ↔ `DD_SERVERLESS_LOGS_ENABLED` OR-merge semantics - `FlushStrategy` — `"end"`, `"end,N"` (`EndPeriodically`), `"periodically,N"`, `"continuously,N"`, invalid → `Default` - Duration parsing — seconds, microseconds, `_ignore_zero` variant - env precedence over YAML - Forgiving fallback when a single field is malformed (default preserved instead of failing the whole extraction) ``` $ cargo test --lib config::lambda_config_tests test result: ok. 37 passed; 0 failed; 0 ignored ``` Existing bottlecap config behavior is unchanged (this PR adds; it doesn't replace yet). ## Follow-up - #1251 — stacked on this branch — does the actual wire-in: replaces `bottlecap::config::Config` with `Config<LambdaConfig>`, deletes the duplicated config files, and removes the legacy `#[macro_export]` macros at the top of `mod.rs`.
|
@jchrostek-dd will update the description with the motivation. short version: bottlecap had its own hand-rolled config that has drifted from the upstream datadog-agent-config crate, so we ended up reimplementing parsing, defaults, and env var handling for things the agent already supports. this migration swaps it for config so we get all the upstream parsing for free and only keep a small lambda-specific extension on top. it also unblocks pulling in upstream features (like the propagation config trait used by dd-trace-rs) without forking them. i will add this to the pr body now. |
Introduces a `LambdaExtension` struct that implements the upstream
`ConfigExtension` trait from `datadog-agent-config`, plus a
`LambdaSource` deserialization shape used for both env-var and YAML
loading via figment's dual-extraction.
This is the first step of migrating bottlecap's in-tree config module
onto the shared serverless-components `datadog-agent-config` crate.
The extension carries the 19 Lambda-specific fields with no upstream
equivalent (api_key_secret_arn, kms_api_key, api_key_ssm_arn,
serverless_logs_enabled, serverless_flush_strategy, enhanced_metrics,
lambda_proc_enhanced_metrics, capture_lambda_payload,
capture_lambda_payload_max_depth, compute_trace_stats_on_extension,
span_dedup_timeout, api_key_secret_reload_interval, dd_org_uuid,
serverless_appsec_enabled, appsec_rules, appsec_waf_timeout,
api_security_enabled, api_security_sample_delay,
custom_metrics_exclude_tags).
Behavior preserved end-to-end with 33 tests covering each field from
both DD_* env vars and datadog.yaml, plus:
- DD_LOGS_ENABLED <-> DD_SERVERLESS_LOGS_ENABLED OR-merge
- FlushStrategy ("end", "periodically,N", invalid -> Default)
- Duration parsing (seconds, microseconds, ignore-zero)
- org_uuid -> dd_org_uuid and lambda_customer_metrics_exclude_tags ->
custom_metrics_exclude_tags source-to-config field mappings
- env precedence over YAML
- Forgiving fallback when a single field is malformed
The dep is pinned to the pre-merge SHA of
DataDog/serverless-components#135 (libdatadog rev alignment) for
development; will be re-pinned to the merged SHA before opening
the migration PR.
Follow-ups (in subsequent commits):
- Replace bottlecap::config::Config with
datadog_agent_config::Config<LambdaExtension>
- Delete duplicated env.rs / yaml.rs / deserializer modules
… mod.rs `LambdaExtension` collides nominally with "the Datadog Lambda Extension" (this entire project). `LambdaConfig` reads more naturally for the extension type that holds Lambda-specific configuration fields. While here, fold the standalone lambda_extension.rs into config/mod.rs so consumers don't need a separate module hop. The inlined code uses fully-qualified `datadog_agent_config::merge_fields!` / `datadog_agent_config::merge_string!` invocations to coexist with the legacy `#[macro_export]` macros still at the top of mod.rs — both go away together once the migration onto the upstream Config lands. Renames carried through: LambdaExtension -> LambdaConfig LambdaSource -> LambdaConfigSource mod lambda_extension -> (gone; inlined) All 33 LambdaConfig tests still pass.
…aConfig>
Completes the migration started in the previous PR onto the shared
datadog-agent-config crate. bottlecap::config::Config is now a type
alias for datadog_agent_config::Config<LambdaConfig>; Lambda-specific
fields live under .ext (per the upstream ConfigExtension trait), and
the 10 in-tree config submodules that duplicated upstream
implementations are removed entirely.
What changes structurally:
- bottlecap/src/config/mod.rs shrinks from 2243 lines to ~600. The
legacy Config struct, ConfigBuilder, ConfigSource, ConfigError,
the #[macro_export] merge_* macros, all the per-field deserializer
helpers, and the entire test module that mirrored upstream's
behavior are gone. What remains: a `type Config` alias, a
`get_config(path)` wrapper, the LambdaConfig extension itself, and
re-exports of upstream modules under the same `crate::config::*`
paths so existing imports keep working.
- New module bottlecap/src/config/propagation_wrapper.rs holds a
newtype `PropConfig(Arc<Config>)` so we can implement the foreign
PropagationConfig trait on the foreign Config<E> type without
running afoul of Rust's orphan rule. The wrapper is scoped to the
dd-trace-rs propagator boundary; nothing else uses it.
- bottlecap/src/traces/propagation/mod.rs wraps the inner propagator
in PropConfig instead of passing Config directly. All call sites
that previously handed `Arc<Config>` to the propagator are
unchanged - the wrapping happens inside DatadogCompositePropagator::new.
- Deleted files (each redundant with upstream):
additional_endpoints.rs, apm_replace_rule.rs, env.rs,
flush_strategy.rs, log_level.rs, logs_additional_endpoints.rs,
processing_rule.rs, service_mapping.rs,
trace_propagation_style.rs, yaml.rs
- All ~70 field-access sites referencing Lambda-specific fields
(config.api_key_secret_arn, config.serverless_logs_enabled,
config.enhanced_metrics, etc.) updated to read through
config.ext.X. Test struct literals that construct Config { ... }
with Lambda fields now nest them under
`ext: LambdaConfig { ..., ..Default::default() }`.
What stays the same:
- LambdaConfig itself (and its 33 tests) - already shipped in the
parent PR; no behavior change in this commit.
- All other tests pass: 501 lib + 5 integration tests green.
- The .ext indirection is invisible to callers that hold an
`Arc<Config>` thanks to Rust's field-access auto-deref - they just
go from `config.foo` to `config.ext.foo` for the 19 Lambda fields.
Stacked on top of the LambdaConfig foundation PR (#1249).
Three non-blocking nits from the independent review: 1. PropConfig's inner Arc<Config> is no longer pub — keeps the wrapper boundary tight; the new() constructor is sufficient and no callers outside the module reach for the inner field directly. 2. PropConfig::new returns Arc<Self> instead of Self. Drops the redundant `Arc::new(...)` wrap at the single call site in traces/propagation/mod.rs. 3. Documents the hard-coded 512 in datadog_tags_max_length — it matches dd-trace-rs's default; bottlecap intentionally doesn't expose DD_TRACE_X_DATADOG_TAGS_MAX_LENGTH. Saves the next reader a round-trip through upstream to confirm parity. No behavior change. All 505 lib tests still pass.
a223aee to
15150c4
Compare
There was a problem hiding this comment.
Pull request overview
This PR completes the migration of Bottlecap’s configuration layer to the shared datadog-agent-config crate by replacing the in-tree bottlecap::config::Config struct with datadog_agent_config::Config<LambdaConfig> (Lambda-specific settings now live under .ext), and by removing duplicated local config parsing modules.
Changes:
- Replaces the legacy Bottlecap config implementation with an upstream-backed
Config<LambdaConfig>and updates call sites to useconfig.ext.*for Lambda-only fields. - Adds a
PropConfignewtype wrapper to satisfy Rust’s orphan rules when implementingPropagationConfigfor the upstream config type. - Deletes redundant in-tree config modules (env/yaml/deserializers/etc.) now covered by
datadog-agent-config.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| bottlecap/src/config/mod.rs | Switches to datadog_agent_config::Config<LambdaConfig>, re-exports upstream modules, and extends LambdaConfig with lambda_durable_function_log_buffer_size. |
| bottlecap/src/config/propagation_wrapper.rs | Introduces PropConfig wrapper to implement PropagationConfig without orphan rule conflicts. |
| bottlecap/src/traces/propagation/mod.rs | Uses PropConfig when constructing the dd-trace-rs composite propagator. |
| bottlecap/src/traces/trace_processor.rs | Updates trace stats gating to read compute_trace_stats_on_extension from config.ext. |
| bottlecap/src/traces/trace_agent.rs | Updates span de-dup timeout access to config.ext.span_dedup_timeout. |
| bottlecap/src/otlp/agent.rs | Updates stats computation toggle to config.ext.compute_trace_stats_on_extension. |
| bottlecap/src/appsec/mod.rs | Switches AppSec enablement check to cfg.ext.serverless_appsec_enabled. |
| bottlecap/src/appsec/processor/mod.rs | Reads AppSec/API Security settings from cfg.ext.* and updates tests accordingly. |
| bottlecap/src/proxy/mod.rs | Updates proxy start condition and tests to use config.ext.serverless_appsec_enabled. |
| bottlecap/src/lifecycle/invocation/processor.rs | Updates invocation processing logic to read Lambda-specific flags from config.ext.*. |
| bottlecap/src/lifecycle/invocation/span_inferrer.rs | Updates AppSec propagation toggle to config.ext.serverless_appsec_enabled and adjusts test config construction. |
| bottlecap/src/logs/lambda/processor.rs | Reads serverless_logs_enabled and durable log buffer size from datadog_config.ext.*. |
| bottlecap/src/metrics/enhanced/lambda.rs | Updates enhanced-metrics checks and tests to use config.ext.enhanced_metrics. |
| bottlecap/src/tags/lambda/tags.rs | Uses config.ext.serverless_appsec_enabled for runtime-family tagging and updates tests. |
| bottlecap/src/secrets/decrypt.rs | Updates secret-resolution decision logic to use config.ext.* Lambda fields. |
| bottlecap/src/secrets/delegated_auth/client.rs | Switches delegated-auth proof generation to use config.ext.dd_org_uuid. |
| bottlecap/src/bin/bottlecap/main.rs | Updates extension loop setup and secrets reload interval to use .ext Lambda fields. |
| bottlecap/tests/appsec_processor_test.rs | Adjusts test config literals to nest Lambda fields under ext: LambdaConfig { ... }. |
| bottlecap/tests/apm_integration_test.rs | Adjusts test config literals to nest Lambda fields under ext: LambdaConfig { ... }. |
| bottlecap/src/config/env.rs | Deleted: replaced by upstream env loading. |
| bottlecap/src/config/yaml.rs | Deleted: replaced by upstream YAML loading. |
| bottlecap/src/config/flush_strategy.rs | Deleted: replaced by upstream flush strategy implementation. |
| bottlecap/src/config/log_level.rs | Deleted: replaced by upstream log level implementation. |
| bottlecap/src/config/processing_rule.rs | Deleted: replaced by upstream processing rule implementation. |
| bottlecap/src/config/service_mapping.rs | Deleted: replaced by upstream service mapping implementation. |
| bottlecap/src/config/trace_propagation_style.rs | Deleted: replaced by upstream trace propagation style parsing. |
| bottlecap/src/config/logs_additional_endpoints.rs | Deleted: replaced by upstream logs additional endpoints parsing. |
| bottlecap/src/config/apm_replace_rule.rs | Deleted: replaced by upstream APM replace-rule parsing. |
| bottlecap/src/config/additional_endpoints.rs | Deleted: replaced by upstream additional endpoints parsing. |
…xtension_compute_stats (#1266) ## Summary Renames the public env var \`DD_COMPUTE_TRACE_STATS_ON_EXTENSION\` to \`DD_LAMBDA_EXTENSION_COMPUTE_STATS\`, plus the matching Rust field, so the name reflects the actual scope: this toggle only governs the Lambda extension's APM trace-stats pipeline and is not a generic Datadog Agent concept. ## Why Auditing the \`LambdaConfig\` extension fields surfaced this one as the one with an ambiguous env var name. \"Compute trace stats on extension\" reads as if there's a generic \"extension\" concept; in practice it's specifically the Lambda extension. Making that explicit avoids confusion if/when the same idea surfaces in other embedders (which would then need a differently-named knob). ## Changes - \`LambdaConfigSource\` field renamed: \`compute_trace_stats_on_extension\` → \`lambda_extension_compute_stats\`. Figment's \`DD_\` prefix maps to the new env var automatically. - \`LambdaConfig\` field renamed in lockstep. All call sites in \`trace_processor.rs\`, \`otlp/agent.rs\`, \`lifecycle/invocation/processor.rs\`, \`tags/lambda/tags.rs\`, and the integration test now read \`config.ext.lambda_extension_compute_stats\`. - \`merge_from\` keeps the field in the existing \`value:\` group of \`merge_fields!\` — no source-to-config rename needed since field names match. - **No back-compat alias** for the old env var. The \`.ext\` migration this depends on has not shipped, so the legacy name was never released externally. - Test renamed; default-false test added. ## Test plan - [x] \`cargo test\` — \`config::lambda_config_tests::lambda_extension_compute_stats_from_env\` and \`lambda_extension_compute_stats_defaults_false\` pass; 38/38 in the module. - [x] \`cargo clippy --workspace --all-targets --features default\` — clean. - [x] \`cargo fmt -- --check\` — clean. ## Notes for review Stacked on top of \`feat/migrate-bottlecap-to-upstream-config\` (the .ext migration PR). Merges into that branch.
Overview
Stacked on top of #1249. Completes the migration to the shared
datadog-agent-configcrate:bottlecap::config::Configis now a type alias fordatadog_agent_config::Config<LambdaConfig>, Lambda-specific fields live under.ext, and the 10 in-tree config submodules that duplicated upstream implementations are removed.Structural changes
bottlecap/src/config/mod.rsshrinks from 2243 → 602 lines. The legacyConfigstruct,ConfigBuilder,ConfigSource,ConfigError, the#[macro_export]merge_*macros, all per-field deserializer helpers, and the test module that mirrored upstream's behavior — all gone. What remains: atype Configalias, aget_config(path)wrapper, theLambdaConfigextension itself (from the parent PR), and re-exports of upstream modules under the samecrate::config::*paths so existing imports keep working.New module
bottlecap/src/config/propagation_wrapper.rsholds a newtypePropConfig(Arc<Config>)so we can implement the foreignPropagationConfigtrait on the foreignConfig<E>type without tripping Rust's orphan rule. The wrapper is scoped to the dd-trace-rs propagator boundary — nothing else uses it.bottlecap/src/traces/propagation/mod.rswraps the inner propagator inPropConfiginstead of passingConfigdirectly. All call sites that previously handedArc<Config>to the propagator are unchanged — the wrapping happens insideDatadogCompositePropagator::new.Deleted files (each redundant with upstream):
additional_endpoints.rs,apm_replace_rule.rs,env.rs,flush_strategy.rs,log_level.rs,logs_additional_endpoints.rs,processing_rule.rs,service_mapping.rs,trace_propagation_style.rs,yaml.rs~70 field-access sites referencing Lambda-specific fields (
config.api_key_secret_arn,config.serverless_logs_enabled,config.enhanced_metrics, etc.) updated to read throughconfig.ext.X. Test struct literals that constructedConfig { ... }with Lambda fields now nest them underext: LambdaConfig { ..., ..Default::default() }.What stays the same
LambdaConfigitself (and its 33 tests) — already shipped in the parent PR; no behavior change in this commit..extindirection is invisible to callers that hold anArc<Config>thanks to Rust's field-access auto-deref — they just go fromconfig.footoconfig.ext.foofor the 19 Lambda fields.Testing
Follow-ups
feat/use-datadog-agent-config-cratetomainonce feat(config): introduce extension config on top of shared datadog-agent-config #1249 merges.custom_metrics_exclude_tagsupstream so it becomes part of coreConfigrather than living in the Lambda extension (discussed in feat(config): introduce extension config on top of shared datadog-agent-config #1249).