Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/en/changes/changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,7 @@
* Fix: `layer-extensions.yml` is now excluded from the `skywalking-oap` jar and shipped to the distribution `config/` directory, so an operator-edited `config/layer-extensions.yml` is no longer shadowed by the empty template bundled in the jar. Because the OAP launch script puts `oap-libs/*.jar` ahead of `config/` on the classpath, `ResourceUtils.read("layer-extensions.yml")` previously always resolved the jar-bundled `layers: []` and silently ignored the operator's file — custom layers declared there never registered. The file now follows the same exclude-from-jar + copy-to-`config/` packaging as every other operator-editable config (`application.yml`, `alarm-settings.yml`, etc.).
* Fix: the v2 MAL compiler now resolves custom layers referenced as `Layer.NAME` in an expression. A custom layer declared through a `layerDefinitions:` block (or `layer-extensions.yml` / the `LayerExtension` SPI) has no generated `Layer.*` static field, so `service(['svc'], Layer.IOT_FLEET)` previously failed code generation because `Layer` has no `IOT_FLEET` field. The compiler now lowers every `Layer.NAME` static-field reference to a runtime `Layer.nameOf("NAME")` registry lookup, so a custom layer can be referenced exactly like a built-in one (`Layer.GENERAL`). For a built-in layer this is equivalent, because `Layer.nameOf("GENERAL")` returns the same instance as the `Layer.GENERAL` field. The lowering is scoped to `Layer` only; the other MAL enum types (`DetectPoint`, `DownsamplingType`, etc.) are real Java enums and keep their direct static-field reference.
* Fix Envoy ALS rendering for the LAL live-debugger and the persisted log `content`: an Istio metadata-exchange peer in `common_properties.filter_state_objects` (legacy Wasm `wasm.*_peer` = `Any{BytesValue}` wrapping a FlatBuffer, or modern `*_peer` = `Any{Struct}`) is now decoded into the readable peer metadata (pod / namespace / labels) instead of an opaque `jsonformat-failed` envelope or base64. The serialization is hardened so a single un-printable field can no longer blank the whole entry — the `LalPayloadDebugDump` printer carries a well-known-type `TypeRegistry` and sanitizes every value `JsonFormat` would reject (an unresolvable, no-slash, or corrupt-bytes `Any` degrades to an `@unresolved` placeholder; a non-finite `Value` double `NaN`/`Infinity` is rendered as a string), keeping the rest of the entry readable. Because the LAL output builder's `bindInput` runs eagerly before the debug capture, this also stops an unregistered `filter_state_objects` type from throwing and aborting the whole rule (dropping the mesh log). Decoding is wired through a new `LalInputDebugRenderer` SPI (`EnvoyAlsHttpDebugRenderer` / `EnvoyAlsTcpDebugRenderer`) so `log-analyzer` reaches the receiver-side decoders without depending on the Envoy receiver, and covers both HTTP and TCP access logs.
* Surface the effective BanyanDB configuration (`bydb.yml` / `bydb-topn.yml`) in the `/debugging/config/dump` admin API. Because the BanyanDB config moved to a separate file in 10.2.0, a BanyanDB deployment previously showed an empty `storage.banyandb` block in the dump; its post-environment-resolution values are now merged into the same response under `storage.banyandb.*` (TopN rules under `storage.banyandb.topN.*`), masked by the same secret-keyword list, via a generic `ConfigDumpExtension` SPI on `ServerStatusService` that any module loading config from a secondary file can implement.

#### UI
* Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7).
Expand Down
28 changes: 25 additions & 3 deletions docs/en/debugging/config_dump.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,12 @@ storage:
password: ${SW_ES_PASSWORD:""}
```

It would be masked and shown as `********` in the dump result.
It would be masked and shown as `******` in the dump result.

```shell
> curl http://127.0.0.1:17128/debugging/config/dump
...
storage.elasticsearch.password=********
storage.elasticsearch.password=******
...
```

Expand All @@ -89,4 +89,26 @@ By default, we mask the config keys through the following configurations.
```yaml
# Include the list of keywords to filter configurations including secrets. Separate keywords by a comma.
keywords4MaskingSecretsOfConfig: ${SW_DEBUGGING_QUERY_KEYWORDS_FOR_MASKING_SECRETS:user,password,trustStorePass,keyStorePass,token,accessKey,secretKey,authentication}
```
```

## BanyanDB Storage Configurations

When BanyanDB is the active storage, the OAP loads its configuration from the dedicated
`bydb.yml` and `bydb-topn.yml` files (separated out from `application.yml` since 10.2.0). The
effective, environment-resolved values of these files are included in the same dump under the
`storage.banyandb.*` keys (TopN rules under `storage.banyandb.topN.*`), for example:

```shell
> curl http://127.0.0.1:17128/debugging/config/dump
...
storage.banyandb.global.targets=127.0.0.1:17912
storage.banyandb.global.user=******
storage.banyandb.global.password=******
storage.banyandb.metricsMinute.ttl=7
storage.banyandb.topN.endpoint_cpm.endpoint_cpm-service.countersNumber=1000
...
```

These rows reflect what the server actually loaded from `bydb.yml` / `bydb-topn.yml` (after
environment-variable overrides), so editing `storage.banyandb.*` under `application.yml` has no
effect. Secret values are masked using the same keyword list described above.
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.core.status;

import java.util.Map;

/**
* Contributes additional effective (post-environment-resolution) configurations to the
* {@code /debugging/config/dump} output. It exists for modules whose runtime configuration is
* loaded from a secondary file outside {@code application.yml} — for example the BanyanDB storage
* plugin loads {@code bydb.yml} / {@code bydb-topn.yml} into its own POJO, which the boot-time
* {@link ServerStatusService#dumpBootingConfigurations(String)} cannot otherwise see.
*
* <p>Implementations register themselves through
* {@link ServerStatusService#registerConfigDumpExtension(ConfigDumpExtension)}.
*
* @since 11.0.0
*/
public interface ConfigDumpExtension {
/**
* @return the effective configurations as fully-qualified {@code module.provider.key} to value
* pairs, keyed exactly like the boot dump so they interleave with it. Values are returned raw;
* the dump masks secrets centrally by key, so each key should carry its field name as the last
* segment (e.g. {@code storage.banyandb.global.password}).
*/
Map<String, String> dumpConfigurations();
}
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ public class ServerStatusService implements Service {

private List<ServerStatusWatcher> statusWatchers = new CopyOnWriteArrayList<>();

private final List<ConfigDumpExtension> configDumpExtensions = new CopyOnWriteArrayList<>();

private List<ApplicationConfiguration.ModuleConfiguration> configurations;

public void bootedNow(List<ApplicationConfiguration.ModuleConfiguration> configurations, long uptime) {
Expand Down Expand Up @@ -81,6 +83,18 @@ public void registerWatcher(ServerStatusWatcher watcher) {
this.statusWatchers.add(watcher);
}

/**
* Register a {@link ConfigDumpExtension} whose effective configurations are merged into the
* {@code /debugging/config/dump} output. Used by modules (e.g. the BanyanDB storage plugin)
* that load configuration from a secondary file outside {@code application.yml}.
*
* @param extension the extension contributing extra effective configurations to the dump
* @since 11.0.0
*/
public void registerConfigDumpExtension(ConfigDumpExtension extension) {
this.configDumpExtensions.add(extension);
}

/**
* @return a complete list of booting configurations with effected values.
* @since 9.7.0
Expand Down Expand Up @@ -117,6 +131,10 @@ public ConfigList dumpBootingConfigurations(String keywords4MaskingSecretsOfConf
)
);
}
for (ConfigDumpExtension extension : configDumpExtensions) {
extension.dumpConfigurations().forEach(
(key, value) -> configList.put(key, maskConfigValue(key, value, keywords)));
}
return configList;
}

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
*/

package org.apache.skywalking.oap.server.core.status;

import java.lang.reflect.Field;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.skywalking.oap.server.core.status.ServerStatusService.ConfigList;
import org.apache.skywalking.oap.server.library.module.ApplicationConfiguration;
import org.apache.skywalking.oap.server.library.module.ModuleManager;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.mockito.Mockito.mock;

public class ServerStatusServiceTest {
private static final String KEYWORDS = "user,password";

@Test
public void shouldMergeAndMaskConfigDumpExtensions() throws Exception {
ServerStatusService service = new ServerStatusService(mock(ModuleManager.class));
seedBootingConfigurations(service);

service.registerConfigDumpExtension(() -> Map.of(
"storage.banyandb.global.targets", "127.0.0.1:17912",
"storage.banyandb.global.user", "admin",
"storage.banyandb.global.password", "s3cret",
"storage.banyandb.metricsMinute.ttl", "7"
));

ConfigList dump = service.dumpBootingConfigurations(KEYWORDS);

// Secrets masked by their field-name keyword, applied centrally to extension rows too.
assertEquals("******", dump.get("storage.banyandb.global.user"));
assertEquals("******", dump.get("storage.banyandb.global.password"));
// Non-secret extension values pass through, merged into the same dump.
assertEquals("127.0.0.1:17912", dump.get("storage.banyandb.global.targets"));
assertEquals("7", dump.get("storage.banyandb.metricsMinute.ttl"));
// The application.yml-derived rows are still present.
assertEquals("12800", dump.get("core.default.restPort"));
}

@Test
public void shouldNotLeakExtensionsBeforeBoot() {
ServerStatusService service = new ServerStatusService(mock(ModuleManager.class));
service.registerConfigDumpExtension(
() -> Map.of("storage.banyandb.global.targets", "127.0.0.1:17912"));

// configurations not set yet (pre-boot) -> dump is empty; extensions are not dumped early.
assertFalse(service.dumpBootingConfigurations(KEYWORDS)
.containsKey("storage.banyandb.global.targets"));
}

private void seedBootingConfigurations(ServerStatusService service) throws Exception {
ApplicationConfiguration appConfig = new ApplicationConfiguration();
Properties props = new Properties();
props.put("restPort", "12800");
ApplicationConfiguration.ModuleConfiguration core = appConfig.addModule("core");
core.addProviderConfiguration("default", props);

Field field = ServerStatusService.class.getDeclaredField("configurations");
field.setAccessible(true);
field.set(service, List.of(core));
}
}
Loading
Loading