Error scraping some metrics

[mbndiaye05]

We are getting 401 errors on several endpoints when retrieving metrics from a device, This is causing data loss on the graph (see screnshot)

• OS: Ubuntu 24.04.3 LTS
• Browser : Microsoft Edge

Logs for the collector :

otel-collector-1 | 2025-12-10T10:20:59.936Z error scraperhelper@v0.119.0/obs_metrics.go:61 Error scraping metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "bigip/device_name", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "endpoint /mgmt/tm/cm/device returned code 401; endpoint /mgmt/tm/sys/cpu returned code 401; endpoint /mgmt/tm/sys/memory returned code 401; endpoint /mgmt/tm/sys/proc-info returned code 401; endpoint /mgmt/tm/ltm/virtual/stats returned code 401; endpoint /mgmt/tm/ltm/pool returned code 401; endpoint /mgmt/tm/ltm/pool/stats returned code 401; endpoint /mgmt/tm/net/interface returned code 401; endpoint /mgmt/tm/net/vlan/stats returned code 401; endpoint /mgmt/tm/ltm/node/stats returned code 401; endpoint /mgmt/tm/sys/provision returned code 401; endpoint /mgmt/tm/sys/license returned code 401; endpoint /mgmt/tm/ltm/rule/stats returned code 401; endpoint /mgmt/tm/ltm/eviction-policy returned code 401; endpoint /mgmt/tm/sys/file/ssl-cert returned code 401; endpoint /mgmt/tm/ltm/profile/fastl4 returned code 401; endpoint /mgmt/tm/ltm/profile/udp returned code 401; endpoint /mgmt/tm/ltm/profile/tcp returned code 401; endpoint /mgmt/tm/ltm/profile/http returned code 401; endpoint /mgmt/tm/ltm/profile/http returned code 401; endpoint /mgmt/tm/ltm/profile/http3 returned code 401; endpoint /mgmt/tm/ltm/profile/one-connect returned code 401; endpoint /mgmt/tm/ltm/profile/client-ssl returned code 401; endpoint /mgmt/tm/ltm/profile/server-ssl returned code 401; endpoint /mgmt/tm/sys/disk/logical-disk returned code 401"}
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.wrapObsMetrics.func1
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/obs_metrics.go:61
otel-collector-1 | go.opentelemetry.io/collector/scraper.ScrapeMetricsFunc.ScrapeMetrics
otel-collector-1 | go.opentelemetry.io/collector/scraper@v0.119.0/metrics.go:24
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.scrapeMetrics
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/controller.go:256
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.NewMetricsController.func1
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/controller.go:228
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.(*controller[...]).startScraping.func1
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/controller.go:175
otel-collector-1 | 2025-12-10T10:21:00.100Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 6, "data points": 154}
otel-collector-1 | 2025-12-10T10:21:00.500Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 174, "data points": 1164}
otel-collector-1 | 2025-12-10T10:21:00.701Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 240, "data points": 1701}
otel-collector-1 | 2025-12-10T10:21:00.902Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 283, "data points": 2052}
otel-collector-1 | 2025-12-10T10:21:01.104Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 2, "metrics": 547, "data points": 4873}
otel-collector-1 | 2025-12-10T10:21:01.705Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 286, "data points": 3283}
otel-collector-1 | 2025-12-10T10:21:02.025Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 286, "data points": 8341}
otel-collector-1 | 2025-12-10T10:21:02.182Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 289, "data points": 8880}
otel-collector-1 | 2025-12-10T10:21:59.950Z error scraperhelper@v0.119.0/obs_metrics.go:61 Error scraping metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "bigip/device_name", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "endpoint /mgmt/tm/cm/device returned code 401; endpoint /mgmt/tm/sys/cpu returned code 401; endpoint /mgmt/tm/sys/memory returned code 401; endpoint /mgmt/tm/sys/proc-info returned code 401; endpoint /mgmt/tm/ltm/virtual/stats returned code 401; endpoint /mgmt/tm/ltm/pool returned code 401; endpoint /mgmt/tm/ltm/pool/stats returned code 401; endpoint /mgmt/tm/net/interface returned code 401; endpoint /mgmt/tm/net/vlan/stats returned code 401; endpoint /mgmt/tm/ltm/node/stats returned code 401; endpoint /mgmt/tm/sys/provision returned code 401; endpoint /mgmt/tm/sys/license returned code 401; endpoint /mgmt/tm/ltm/rule/stats returned code 401; endpoint /mgmt/tm/ltm/eviction-policy returned code 401; endpoint /mgmt/tm/sys/file/ssl-cert returned code 401; endpoint /mgmt/tm/ltm/profile/fastl4 returned code 401; endpoint /mgmt/tm/ltm/profile/udp returned code 401; endpoint /mgmt/tm/ltm/profile/tcp returned code 401; endpoint /mgmt/tm/ltm/profile/http returned code 401; endpoint /mgmt/tm/ltm/profile/http returned code 401; endpoint /mgmt/tm/ltm/profile/http3 returned code 401; endpoint /mgmt/tm/ltm/profile/one-connect returned code 401; endpoint /mgmt/tm/ltm/profile/client-ssl returned code 401; endpoint /mgmt/tm/ltm/profile/server-ssl returned code 401; endpoint /mgmt/tm/sys/disk/logical-disk returned code 401"}
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.wrapObsMetrics.func1
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/obs_metrics.go:61
otel-collector-1 | go.opentelemetry.io/collector/scraper.ScrapeMetricsFunc.ScrapeMetrics
otel-collector-1 | go.opentelemetry.io/collector/scraper@v0.119.0/metrics.go:24
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.scrapeMetrics
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/controller.go:256
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.NewMetricsController.func1
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/controller.go:228
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper.(*controller[...]).startScraping.func1
otel-collector-1 | go.opentelemetry.io/collector/scraper/scraperhelper@v0.119.0/controller.go:175
otel-collector-1 | 2025-12-10T10:22:00.068Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 6, "data points": 154}
otel-collector-1 | 2025-12-10T10:22:00.470Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 174, "data points": 1164}
otel-collector-1 | 2025-12-10T10:22:00.670Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 240, "data points": 1701}
otel-collector-1 | 2025-12-10T10:22:00.871Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 283, "data points": 2052}
otel-collector-1 | 2025-12-10T10:22:01.273Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 2, "metrics": 547, "data points": 4873}
otel-collector-1 | 2025-12-10T10:22:01.875Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 286, "data points": 3283}
otel-collector-1 | 2025-12-10T10:22:02.293Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 286, "data points": 8341}
otel-collector-1 | 2025-12-10T10:22:02.406Z info Metrics {"resource": {"service.instance.id": "4c67f2d0-c284-4ab2-91b1-2f8fb61afa06", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "debug/bigip", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "resource metrics": 1, "metrics": 289, "data points": 8880}

[mbndiaye05]

We need support on this problem because we don't understand the issue.

[mikempw]

You're receiving a 401 authentication issue. Can you please ensure the appropriate user permissions and that the control plane is not overwhelmed?

[mbndiaye05]

We're actually using an admin account but we gonna check the control plane part (maybe improve the restjavad RAM), will let you know if it will solve the problem

[mikempw]

You can also change the concurrent worker count. I'd recommend changing that to 1 vs default of 2.

[mbndiaye05]

It's already set on 1 "concurrent_workers: 1" in the receivers.yaml (/application-study-tool/services/otel_collector) file of the otel-collector for the device with the issue and also in bigip_receivers.yaml (/application-study-tool/config) . Is that the right files or should I configure the parameter somewhere else?

[mikempw]

https://f5devcentral.github.io/application-study-tool/troubleshooting Has the guide for it.

[mbndiaye05]

Hello,

We continue to receive 401 errors every 1 to 2 minutes on the same device after making all the changes you indicated (concurrent worker count is set to, control plane is not overwhelmed & the user have the appropriate permissions)
You can see on the dashboard data loss (exemple see below screenshoot)

The recurring error:
otel-collector-1 | 2026-01-07T09:45:16.815Z error scraperhelper@v0.119.0/obs_metrics.go:61 Error scraping metrics {"resource": {"service.instance.id": "c9735c10-9bf6-4e69-a06c-be06e88419c4", "service.name": "otelcol-custom", "service.version": ""}, "otelcol.component.id": "bigip/device-name", "otelcol.component.kind": "receiver", "otelcol.signal": "metrics", "error": "endpoint /mgmt/tm/cm/device returned code 401; endpoint /mgmt/tm/sys/cpu returned code 401; endpoint /mgmt/tm/sys/memory returned code 401; endpoint /mgmt/tm/sys/proc-info returned code 401; endpoint /mgmt/tm/ltm/virtual/stats returned code 401; endpoint /mgmt/tm/ltm/pool returned code 401; endpoint /mgmt/tm/ltm/pool/stats returned code 401; endpoint /mgmt/tm/net/interface returned code 401; endpoint /mgmt/tm/net/vlan/stats returned code 401; endpoint /mgmt/tm/ltm/node/stats returned code 401; endpoint /mgmt/tm/sys/provision returned code 401; endpoint /mgmt/tm/sys/license returned code 401; endpoint /mgmt/tm/ltm/rule/stats returned code 401; endpoint /mgmt/tm/ltm/eviction-policy returned code 401; endpoint /mgmt/tm/sys/file/ssl-cert returned code 401; endpoint /mgmt/tm/ltm/profile/fastl4 returned code 401; endpoint /mgmt/tm/ltm/profile/udp returned code 401; endpoint /mgmt/tm/ltm/profile/tcp returned code 401; endpoint /mgmt/tm/ltm/profile/http returned code 401; endpoint /mgmt/tm/ltm/profile/http returned code 401; endpoint /mgmt/tm/ltm/profile/http3 returned code 401; endpoint /mgmt/tm/ltm/profile/one-connect returned code 401; endpoint /mgmt/tm/ltm/profile/client-ssl returned code 401; endpoint /mgmt/tm/ltm/profile/server-ssl returned code 401; endpoint /mgmt/tm/sys/disk/logical-disk returned code 401"}

Can you please investigate further and provide us support on this issue, as it preventing us from moving forward?
Thanks