diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 28f0c62c14ff..86a009439858 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -319,6 +319,7 @@ * Fix: `layer-extensions.yml` is now excluded from the `skywalking-oap` jar and shipped to the distribution `config/` directory, so an operator-edited `config/layer-extensions.yml` is no longer shadowed by the empty template bundled in the jar. Because the OAP launch script puts `oap-libs/*.jar` ahead of `config/` on the classpath, `ResourceUtils.read("layer-extensions.yml")` previously always resolved the jar-bundled `layers: []` and silently ignored the operator's file — custom layers declared there never registered. The file now follows the same exclude-from-jar + copy-to-`config/` packaging as every other operator-editable config (`application.yml`, `alarm-settings.yml`, etc.). * Fix: the v2 MAL compiler now resolves custom layers referenced as `Layer.NAME` in an expression. A custom layer declared through a `layerDefinitions:` block (or `layer-extensions.yml` / the `LayerExtension` SPI) has no generated `Layer.*` static field, so `service(['svc'], Layer.IOT_FLEET)` previously failed code generation because `Layer` has no `IOT_FLEET` field. The compiler now lowers every `Layer.NAME` static-field reference to a runtime `Layer.nameOf("NAME")` registry lookup, so a custom layer can be referenced exactly like a built-in one (`Layer.GENERAL`). For a built-in layer this is equivalent, because `Layer.nameOf("GENERAL")` returns the same instance as the `Layer.GENERAL` field. The lowering is scoped to `Layer` only; the other MAL enum types (`DetectPoint`, `DownsamplingType`, etc.) are real Java enums and keep their direct static-field reference. * Fix Envoy ALS rendering for the LAL live-debugger and the persisted log `content`: an Istio metadata-exchange peer in `common_properties.filter_state_objects` (legacy Wasm `wasm.*_peer` = `Any{BytesValue}` wrapping a FlatBuffer, or modern `*_peer` = `Any{Struct}`) is now decoded into the readable peer metadata (pod / namespace / labels) instead of an opaque `jsonformat-failed` envelope or base64. The serialization is hardened so a single un-printable field can no longer blank the whole entry — the `LalPayloadDebugDump` printer carries a well-known-type `TypeRegistry` and sanitizes every value `JsonFormat` would reject (an unresolvable, no-slash, or corrupt-bytes `Any` degrades to an `@unresolved` placeholder; a non-finite `Value` double `NaN`/`Infinity` is rendered as a string), keeping the rest of the entry readable. Because the LAL output builder's `bindInput` runs eagerly before the debug capture, this also stops an unregistered `filter_state_objects` type from throwing and aborting the whole rule (dropping the mesh log). Decoding is wired through a new `LalInputDebugRenderer` SPI (`EnvoyAlsHttpDebugRenderer` / `EnvoyAlsTcpDebugRenderer`) so `log-analyzer` reaches the receiver-side decoders without depending on the Envoy receiver, and covers both HTTP and TCP access logs. +* Surface the effective BanyanDB configuration (`bydb.yml` / `bydb-topn.yml`) in the `/debugging/config/dump` admin API. Because the BanyanDB config moved to a separate file in 10.2.0, a BanyanDB deployment previously showed an empty `storage.banyandb` block in the dump; its post-environment-resolution values are now merged into the same response under `storage.banyandb.*` (TopN rules under `storage.banyandb.topN.*`), masked by the same secret-keyword list, via a generic `ConfigDumpExtension` SPI on `ServerStatusService` that any module loading config from a secondary file can implement. #### UI * Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7). diff --git a/docs/en/debugging/config_dump.md b/docs/en/debugging/config_dump.md index 753f8dd31d8b..d521cd26b0a1 100644 --- a/docs/en/debugging/config_dump.md +++ b/docs/en/debugging/config_dump.md @@ -75,12 +75,12 @@ storage: password: ${SW_ES_PASSWORD:""} ``` -It would be masked and shown as `********` in the dump result. +It would be masked and shown as `******` in the dump result. ```shell > curl http://127.0.0.1:17128/debugging/config/dump ... -storage.elasticsearch.password=******** +storage.elasticsearch.password=****** ... ``` @@ -89,4 +89,26 @@ By default, we mask the config keys through the following configurations. ```yaml # Include the list of keywords to filter configurations including secrets. Separate keywords by a comma. keywords4MaskingSecretsOfConfig: ${SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS:user,password,trustStorePass,keyStorePass,token,accessKey,secretKey,authentication} -``` \ No newline at end of file +``` + +## BanyanDB Storage Configurations + +When BanyanDB is the active storage, the OAP loads its configuration from the dedicated +`bydb.yml` and `bydb-topn.yml` files (separated out from `application.yml` since 10.2.0). The +effective, environment-resolved values of these files are included in the same dump under the +`storage.banyandb.*` keys (TopN rules under `storage.banyandb.topN.*`), for example: + +```shell +> curl http://127.0.0.1:17128/debugging/config/dump +... +storage.banyandb.global.targets=127.0.0.1:17912 +storage.banyandb.global.user=****** +storage.banyandb.global.password=****** +storage.banyandb.metricsMinute.ttl=7 +storage.banyandb.topN.endpoint_cpm.endpoint_cpm-service.countersNumber=1000 +... +``` + +These rows reflect what the server actually loaded from `bydb.yml` / `bydb-topn.yml` (after +environment-variable overrides), so editing `storage.banyandb.*` under `application.yml` has no +effect. Secret values are masked using the same keyword list described above. \ No newline at end of file diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ConfigDumpExtension.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ConfigDumpExtension.java new file mode 100644 index 000000000000..ec85397c24b9 --- /dev/null +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ConfigDumpExtension.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.core.status; + +import java.util.Map; + +/** + * Contributes additional effective (post-environment-resolution) configurations to the + * {@code /debugging/config/dump} output. It exists for modules whose runtime configuration is + * loaded from a secondary file outside {@code application.yml} — for example the BanyanDB storage + * plugin loads {@code bydb.yml} / {@code bydb-topn.yml} into its own POJO, which the boot-time + * {@link ServerStatusService#dumpBootingConfigurations(String)} cannot otherwise see. + * + *

Implementations register themselves through + * {@link ServerStatusService#registerConfigDumpExtension(ConfigDumpExtension)}. + * + * @since 11.0.0 + */ +public interface ConfigDumpExtension { + /** + * @return the effective configurations as fully-qualified {@code module.provider.key} to value + * pairs, keyed exactly like the boot dump so they interleave with it. Values are returned raw; + * the dump masks secrets centrally by key, so each key should carry its field name as the last + * segment (e.g. {@code storage.banyandb.global.password}). + */ + Map dumpConfigurations(); +} diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java index db9141e8aaee..c97984b26364 100644 --- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java +++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/status/ServerStatusService.java @@ -48,6 +48,8 @@ public class ServerStatusService implements Service { private List statusWatchers = new CopyOnWriteArrayList<>(); + private final List configDumpExtensions = new CopyOnWriteArrayList<>(); + private List configurations; public void bootedNow(List configurations, long uptime) { @@ -81,6 +83,18 @@ public void registerWatcher(ServerStatusWatcher watcher) { this.statusWatchers.add(watcher); } + /** + * Register a {@link ConfigDumpExtension} whose effective configurations are merged into the + * {@code /debugging/config/dump} output. Used by modules (e.g. the BanyanDB storage plugin) + * that load configuration from a secondary file outside {@code application.yml}. + * + * @param extension the extension contributing extra effective configurations to the dump + * @since 11.0.0 + */ + public void registerConfigDumpExtension(ConfigDumpExtension extension) { + this.configDumpExtensions.add(extension); + } + /** * @return a complete list of booting configurations with effected values. * @since 9.7.0 @@ -117,6 +131,10 @@ public ConfigList dumpBootingConfigurations(String keywords4MaskingSecretsOfConf ) ); } + for (ConfigDumpExtension extension : configDumpExtensions) { + extension.dumpConfigurations().forEach( + (key, value) -> configList.put(key, maskConfigValue(key, value, keywords))); + } return configList; } diff --git a/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/status/ServerStatusServiceTest.java b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/status/ServerStatusServiceTest.java new file mode 100644 index 000000000000..b49c50c2d374 --- /dev/null +++ b/oap-server/server-core/src/test/java/org/apache/skywalking/oap/server/core/status/ServerStatusServiceTest.java @@ -0,0 +1,83 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.core.status; + +import java.lang.reflect.Field; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import org.apache.skywalking.oap.server.core.status.ServerStatusService.ConfigList; +import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration; +import org.apache.skywalking.oap.server.library.module.ModuleManager; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.mockito.Mockito.mock; + +public class ServerStatusServiceTest { + private static final String KEYWORDS = "user,password"; + + @Test + public void shouldMergeAndMaskConfigDumpExtensions() throws Exception { + ServerStatusService service = new ServerStatusService(mock(ModuleManager.class)); + seedBootingConfigurations(service); + + service.registerConfigDumpExtension(() -> Map.of( + "storage.banyandb.global.targets", "127.0.0.1:17912", + "storage.banyandb.global.user", "admin", + "storage.banyandb.global.password", "s3cret", + "storage.banyandb.metricsMinute.ttl", "7" + )); + + ConfigList dump = service.dumpBootingConfigurations(KEYWORDS); + + // Secrets masked by their field-name keyword, applied centrally to extension rows too. + assertEquals("******", dump.get("storage.banyandb.global.user")); + assertEquals("******", dump.get("storage.banyandb.global.password")); + // Non-secret extension values pass through, merged into the same dump. + assertEquals("127.0.0.1:17912", dump.get("storage.banyandb.global.targets")); + assertEquals("7", dump.get("storage.banyandb.metricsMinute.ttl")); + // The application.yml-derived rows are still present. + assertEquals("12800", dump.get("core.default.restPort")); + } + + @Test + public void shouldNotLeakExtensionsBeforeBoot() { + ServerStatusService service = new ServerStatusService(mock(ModuleManager.class)); + service.registerConfigDumpExtension( + () -> Map.of("storage.banyandb.global.targets", "127.0.0.1:17912")); + + // configurations not set yet (pre-boot) -> dump is empty; extensions are not dumped early. + assertFalse(service.dumpBootingConfigurations(KEYWORDS) + .containsKey("storage.banyandb.global.targets")); + } + + private void seedBootingConfigurations(ServerStatusService service) throws Exception { + ApplicationConfiguration appConfig = new ApplicationConfiguration(); + Properties props = new Properties(); + props.put("restPort", "12800"); + ApplicationConfiguration.ModuleConfiguration core = appConfig.addModule("core"); + core.addProviderConfiguration("default", props); + + Field field = ServerStatusService.class.getDeclaredField("configurations"); + field.setAccessible(true); + field.set(service, List.of(core)); + } +} diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigDumpExtension.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigDumpExtension.java new file mode 100644 index 000000000000..bc047509b9b8 --- /dev/null +++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigDumpExtension.java @@ -0,0 +1,137 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.storage.plugin.banyandb; + +import com.google.common.base.Strings; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import org.apache.skywalking.oap.server.core.status.ConfigDumpExtension; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.Global; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.GroupResource; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.Stage; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.TopN; + +/** + * Flattens the loaded, environment-resolved {@link BanyanDBStorageConfig} (from {@code bydb.yml} + * and {@code bydb-topn.yml}) into {@code /debugging/config/dump} rows. Without this, a BanyanDB + * deployment shows an empty {@code storage.banyandb} block in the dump, because that configuration + * lives in a separate file the boot-time dump does not parse. + * + *

Keys read straight from the loaded POJO, never re-read from the file, so the values are the + * effective post-environment-override ones. Group segment names mirror the {@code groups:} keys in + * {@code bydb.yml} so the dump reads like the source file. Secret masking (e.g. {@code global.user} + * / {@code global.password}) is applied centrally by the dump, so this extension returns raw values. + * + * @since 11.0.0 + */ +public class BanyanDBConfigDumpExtension implements ConfigDumpExtension { + private final String prefix; + private final BanyanDBStorageConfig config; + + public BanyanDBConfigDumpExtension(final String prefix, final BanyanDBStorageConfig config) { + this.prefix = prefix; + this.config = config; + } + + @Override + public Map dumpConfigurations() { + final Map dump = new LinkedHashMap<>(); + flattenGlobal(prefix + ".global", config.getGlobal(), dump); + + // Only the groups populated from bydb.yml are emitted; segment names match its `groups:` keys. + final Map groups = new LinkedHashMap<>(); + groups.put("records", config.getRecordsNormal()); + groups.put("recordsLog", config.getRecordsLog()); + groups.put("trace", config.getTrace()); + groups.put("zipkinTrace", config.getZipkinTrace()); + groups.put("recordsBrowserErrorLog", config.getRecordsBrowserErrorLog()); + groups.put("metricsMinute", config.getMetricsMin()); + groups.put("metricsHour", config.getMetricsHour()); + groups.put("metricsDay", config.getMetricsDay()); + groups.put("metadata", config.getMetadata()); + groups.put("property", config.getProperty()); + groups.forEach((name, group) -> flattenGroup(prefix + "." + name, group, dump)); + + config.getTopNConfigs().forEach((metric, rules) -> + rules.forEach((ruleName, topN) -> + flattenTopN(prefix + ".topN." + metric + "." + ruleName, topN, dump))); + return dump; + } + + private void flattenGlobal(final String p, final Global g, final Map dump) { + // getTargets()/getCompatibleServerApiVersions() split the backing String into an array; + // join them back so the dump shows the raw comma-separated form the operator configured. + dump.put(p + ".targets", String.join(",", g.getTargets())); + dump.put(p + ".maxBulkSize", String.valueOf(g.getMaxBulkSize())); + dump.put(p + ".flushInterval", String.valueOf(g.getFlushInterval())); + dump.put(p + ".flushTimeout", String.valueOf(g.getFlushTimeout())); + dump.put(p + ".concurrentWriteThreads", String.valueOf(g.getConcurrentWriteThreads())); + dump.put(p + ".profileTaskQueryMaxSize", String.valueOf(g.getProfileTaskQueryMaxSize())); + dump.put(p + ".asyncProfilerTaskQueryMaxSize", String.valueOf(g.getAsyncProfilerTaskQueryMaxSize())); + dump.put(p + ".pprofTaskQueryMaxSize", String.valueOf(g.getPprofTaskQueryMaxSize())); + dump.put(p + ".resultWindowMaxSize", String.valueOf(g.getResultWindowMaxSize())); + dump.put(p + ".metadataQueryMaxSize", String.valueOf(g.getMetadataQueryMaxSize())); + dump.put(p + ".segmentQueryMaxSize", String.valueOf(g.getSegmentQueryMaxSize())); + dump.put(p + ".profileDataQueryBatchSize", String.valueOf(g.getProfileDataQueryBatchSize())); + dump.put(p + ".cleanupUnusedTopNRules", String.valueOf(g.isCleanupUnusedTopNRules())); + dump.put(p + ".namespace", Strings.nullToEmpty(g.getNamespace())); + dump.put(p + ".compatibleServerApiVersions", String.join(",", g.getCompatibleServerApiVersions())); + dump.put(p + ".user", Strings.nullToEmpty(g.getUser())); + dump.put(p + ".password", Strings.nullToEmpty(g.getPassword())); + dump.put(p + ".sslTrustCAPath", Strings.nullToEmpty(g.getSslTrustCAPath())); + } + + private void flattenGroup(final String p, final GroupResource group, final Map dump) { + dump.put(p + ".shardNum", String.valueOf(group.getShardNum())); + dump.put(p + ".segmentInterval", String.valueOf(group.getSegmentInterval())); + dump.put(p + ".ttl", String.valueOf(group.getTtl())); + dump.put(p + ".replicas", String.valueOf(group.getReplicas())); + dump.put(p + ".enableWarmStage", String.valueOf(group.isEnableWarmStage())); + dump.put(p + ".enableColdStage", String.valueOf(group.isEnableColdStage())); + dump.put(p + ".defaultQueryStages", group.getDefaultQueryStages().toString()); + final List stages = group.getAdditionalLifecycleStages(); + for (int i = 0; i < stages.size(); i++) { + flattenStage(p + ".additionalLifecycleStages." + i, stages.get(i), dump); + } + } + + private void flattenStage(final String p, final Stage stage, final Map dump) { + dump.put(p + ".name", stage.getName() == null ? "" : stage.getName().name()); + dump.put(p + ".nodeSelector", Strings.nullToEmpty(stage.getNodeSelector())); + dump.put(p + ".shardNum", String.valueOf(stage.getShardNum())); + dump.put(p + ".segmentInterval", String.valueOf(stage.getSegmentInterval())); + dump.put(p + ".ttl", String.valueOf(stage.getTtl())); + dump.put(p + ".replicas", String.valueOf(stage.getReplicas())); + dump.put(p + ".close", String.valueOf(stage.isClose())); + } + + private void flattenTopN(final String p, final TopN topN, final Map dump) { + dump.put(p + ".lruSizeMinute", String.valueOf(topN.getLruSizeMinute())); + dump.put(p + ".lruSizeHourDay", String.valueOf(topN.getLruSizeHourDay())); + dump.put(p + ".countersNumber", String.valueOf(topN.getCountersNumber())); + dump.put(p + ".sort", topN.getSort() == null ? "" : topN.getSort().name()); + dump.put(p + ".groupByTagNames", + topN.getGroupByTagNames() == null ? "[]" : topN.getGroupByTagNames().toString()); + dump.put(p + ".excludes", topN.getExcludes().stream() + .map(kv -> kv.getKey() + "=" + kv.getValue()) + .collect(Collectors.joining(",", "[", "]"))); + } +} diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java index fd858ab68868..dbb2b7f3dffc 100644 --- a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java +++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBStorageProvider.java @@ -27,6 +27,7 @@ import org.apache.skywalking.library.banyandb.v1.client.grpc.exception.BanyanDBException; import org.apache.skywalking.oap.server.core.CoreModule; import org.apache.skywalking.oap.server.core.RunningMode; +import org.apache.skywalking.oap.server.core.status.ServerStatusService; import org.apache.skywalking.oap.server.core.storage.IBatchDAO; import org.apache.skywalking.oap.server.core.storage.IHistoryDeleteDAO; import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory; @@ -244,6 +245,11 @@ public void start() throws ServiceNotProvidedException, ModuleStartException { this.modelInstaller.start(); getManager().find(CoreModule.NAME).provider().getService(ModelRegistry.class).addModelListener(modelInstaller); + + ServerStatusService serverStatusService = + getManager().find(CoreModule.NAME).provider().getService(ServerStatusService.class); + serverStatusService.registerConfigDumpExtension( + new BanyanDBConfigDumpExtension(StorageModule.NAME + "." + name(), this.config)); } catch (Exception e) { throw new ModuleStartException(e.getMessage(), e); } diff --git a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigDumpExtensionTest.java b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigDumpExtensionTest.java new file mode 100644 index 000000000000..daeef8fac5e2 --- /dev/null +++ b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/test/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/BanyanDBConfigDumpExtensionTest.java @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.storage.plugin.banyandb; + +import java.util.Map; +import org.apache.skywalking.oap.server.core.query.type.KeyValue; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.MetricsMin; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.Stage; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.StageName; +import org.apache.skywalking.oap.server.storage.plugin.banyandb.BanyanDBStorageConfig.TopN; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; + +public class BanyanDBConfigDumpExtensionTest { + @Test + public void shouldFlattenLoadedConfig() { + BanyanDBStorageConfig config = new BanyanDBStorageConfig(); + config.getGlobal().setTargets("a:1,b:2"); + config.getGlobal().setUser("admin"); + config.getGlobal().setPassword("admin"); + config.getGlobal().setCompatibleServerApiVersions("0.10"); + config.getGlobal().setNamespace("sw"); + + // A metrics-minute group carrying a warm lifecycle stage. + MetricsMin metricsMin = config.getMetricsMin(); + metricsMin.setShardNum(1); + metricsMin.setTtl(7); + metricsMin.setEnableWarmStage(true); + Stage warm = new Stage(); + warm.setName(StageName.warm); + warm.setTtl(30); + metricsMin.getAdditionalLifecycleStages().add(warm); + + // A TopN rule under bydb-topn.yml. + TopN topN = new TopN(); + topN.setCountersNumber(1000); + topN.setSort(TopN.Sort.des); + topN.getExcludes().add(new KeyValue("a", "b")); + config.getTopNConfigs().put("endpoint_cpm", Map.of("endpoint_cpm-service", topN)); + + Map dump = + new BanyanDBConfigDumpExtension("storage.banyandb", config).dumpConfigurations(); + + // global: targets joined back to the raw comma form, not the split array. + assertEquals("a:1,b:2", dump.get("storage.banyandb.global.targets")); + // Raw values; masking is the dump's responsibility, not the extension's. + assertEquals("admin", dump.get("storage.banyandb.global.user")); + assertEquals("admin", dump.get("storage.banyandb.global.password")); + assertEquals("0.10", dump.get("storage.banyandb.global.compatibleServerApiVersions")); + // Group + indexed lifecycle stage (enum -> name()). + assertEquals("7", dump.get("storage.banyandb.metricsMinute.ttl")); + assertEquals("true", dump.get("storage.banyandb.metricsMinute.enableWarmStage")); + assertEquals("warm", dump.get("storage.banyandb.metricsMinute.additionalLifecycleStages.0.name")); + assertEquals("30", dump.get("storage.banyandb.metricsMinute.additionalLifecycleStages.0.ttl")); + // TopN nested under topN... + assertEquals("1000", dump.get("storage.banyandb.topN.endpoint_cpm.endpoint_cpm-service.countersNumber")); + assertEquals("des", dump.get("storage.banyandb.topN.endpoint_cpm.endpoint_cpm-service.sort")); + assertEquals("[a=b]", dump.get("storage.banyandb.topN.endpoint_cpm.endpoint_cpm-service.excludes")); + } +} diff --git a/test/e2e-v2/cases/storage/banyandb/e2e.yaml b/test/e2e-v2/cases/storage/banyandb/e2e.yaml index c59388a5d59b..8c9d9a7ff92e 100644 --- a/test/e2e-v2/cases/storage/banyandb/e2e.yaml +++ b/test/e2e-v2/cases/storage/banyandb/e2e.yaml @@ -66,6 +66,25 @@ verify: - query: | swctl --display json --admin-url=http://${oap_host}:${oap_17128} admin config ttl expected: ../expected/ttl-config-banyandb.yml + # The effective bydb.yml / bydb-topn.yml config is surfaced in /debugging/config/dump + # (one representative key per section) and the credentials are masked. Assertions live + # in the query; it prints `config-dump-ok` only when all of them pass. + - query: | + dump=$(curl -s http://${oap_host}:${oap_17128}/debugging/config/dump) + exact() { echo "$dump" | grep -Fqx "$1" || { echo "missing or unexpected: $1"; exit 1; }; } + exists() { echo "$dump" | grep -Fq "$1" || { echo "missing: $1"; exit 1; }; } + exact 'storage.provider=banyandb' + exact 'storage.banyandb.global.user=******' + exact 'storage.banyandb.global.password=******' + exact 'storage.banyandb.global.namespace=sw' + exists 'storage.banyandb.global.targets=' + exact 'storage.banyandb.records.ttl=3' + exact 'storage.banyandb.metricsMinute.ttl=7' + exact 'storage.banyandb.metricsMinute.enableWarmStage=true' + exact 'storage.banyandb.metricsMinute.additionalLifecycleStages.0.name=warm' + exists 'storage.banyandb.topN.endpoint_cpm.endpoint_cpm-service.countersNumber=' + echo config-dump-ok + expected: ../expected/config-dump-banyandb.yml cleanup: on: always collect: diff --git a/test/e2e-v2/cases/storage/expected/config-dump-banyandb.yml b/test/e2e-v2/cases/storage/expected/config-dump-banyandb.yml new file mode 100644 index 000000000000..070a4d3de88e --- /dev/null +++ b/test/e2e-v2/cases/storage/expected/config-dump-banyandb.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The representative bydb config-dump assertions run inside the query (it prints this +# sentinel only when every picked key is present and the secrets are masked). +config-dump-ok