[WIP] Add opt-in Usage Reporter (call-home) for the CloudStack project - #13985
Draft
wido wants to merge 3 commits into
Draft
[WIP] Add opt-in Usage Reporter (call-home) for the CloudStack project#13985wido wants to merge 3 commits into
wido wants to merge 3 commits into
Conversation
The Management Server periodically sends an anonymous usage report to an API endpoint of the CloudStack project. This is opt-in and disabled by default (usage.report.interval = 0); only aggregated statistics are sent, nothing that can directly identify an environment. Reports are only sent over HTTPS. Includes the server-side collector, a Python Flask/WSGI application which validates incoming reports and stores them as JSON files on the local filesystem, in a directory per environment with the receive timestamp as filename. Submissions are rate limited and bounded per environment and reports with unexpected structure are rejected.
|
This is a really nice idea and will help us understand better the CloudStack users, their stack and environment specifics. IMO the information shall be limited to PMCs only, as it opens a bit of security gap for proprietary vendors to target the CloudStack users. But we can create Qly reports and publish them as blog posts on the ACS website. I was hoping for years to get such information, so really nice job! |
…enses
Pin down the JSON that the management server POSTs to usage.report.uri so
any change to the wire format has to be a deliberate one.
UsageReporterTest mocks the seven DAOs the report is built from and drives
the real report builders through the real AtomicGsonAdapter, comparing the
result against a checked-in fixture, usage-report-expected.json. Keys are
sorted on both sides before comparing: AtomicLongMap is backed by a
ConcurrentHashMap and the report itself by a HashMap, so key order on the
wire is not deterministic and must not be part of the contract. Sorting
also makes a mismatch print a readable diff.
The tests document three things that are not obvious from reading the code:
- provisioning_type keys are lowercase ("thin"/"fat"). Storage.Provisio-
ningType overrides toString() and the adapter keys on String.valueOf(),
so the payload does not carry the enum constant names that Gson would
emit by default.
- boolean counters reach the wire as the string keys "true" and "false",
used by ha_enabled, dynamically_scalable, compute_only and
use_local_storage.
- a report from an empty install still carries all eight sections with
empty counter objects, and avg_disk_size falls back to 0 rather than
dividing by zero.
AtomicGsonAdapterTest covers the adapter on its own: null and empty maps,
counts as numbers, boolean and enum keys, and that read() consumes a null.
Also add the ASF license headers that apache-rat flags on reporter/README.md
and reporter/requirements.txt.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #13985 +/- ##
============================================
+ Coverage 19.74% 19.75% +0.01%
- Complexity 19960 19996 +36
============================================
Files 6371 6374 +3
Lines 575784 576075 +291
Branches 70478 70516 +38
============================================
+ Hits 113665 113827 +162
- Misses 449765 449889 +124
- Partials 12354 12359 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
It would be definitely useful but I'm not sure how many users would want to share it. It would be useful to some control on toggling configs on the first login screen. |
DaanHoogland
approved these changes
Aug 27, 2026
DaanHoogland
left a comment
DaanHoogland
left a comment
Contributor
There was a problem hiding this comment.
code looks generally good, but prove is in the eating of the pudding.
Contributor
|
I think if we do not disclose the information to the general public we should abandon this project. It is definitely not for PMC only, or even restricted to any apache circle. Its use is for any contributor from anywhere. |
…n identity util Address the first points of the architectural review on the usage reporter. The reporting destination is no longer a Global Setting. The point of telemetry is to give the Apache CloudStack project authoritative project-wide statistics, so "telemetry enabled" has to mean the data reaches the project rather than wherever an operator points it. The endpoint is now the constant https://call-home.cloudstack.org/report, with DNS providing whatever indirection the receiving infrastructure needs. Rename usage.report.interval to telemetry.interval. The usage.* prefix collides with the Usage Server / usage records subsystem and reads like an accounting setting. Zero still means disabled. The installation identity stays derived rather than generated: the version the database was created with plus the moment it was created already identifies an installation, is shared by every Management Server on that database, and survives restarts and upgrades without anything being stored. Move the derivation out of UsageReporter into InstallationIdentity in the utils module, fed by a new VersionDao.getInitialVersion(). That removes the raw JDBC from UsageReporter and makes both halves unit testable. Two changes to the hash follow from taking the timestamp as a Date instead of whatever string the JDBC driver returned: it is formatted as UTC, so Management Servers in different timezones sharing a database derive the same identity, and the two inputs are separated so different pairs cannot concatenate into the same string. The derived value therefore differs from the previous implementation, which is a one-time change for anyone already running this branch and has no effect otherwise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is a WIP / RFC to gather feedback before it is finished. Not intended to be merged as-is.
This proposes an opt-in Usage Reporter ("call-home") for CloudStack. The goal is to give the project insight into how CloudStack is actually deployed in the wild: which hypervisors, storage types, network offerings and versions are in use, and how large environments typically are. Today we simply do not know, which makes it hard to decide what to prioritise, what to deprecate and what to test.
Two parts are included:
UsageReporter): periodically collects aggregated counters and POSTs them as JSON to an endpoint of the CloudStack project over HTTPS.reporter/): a small Python Flask/WSGI application that validates incoming reports and stores them as JSON files on disk, one directory per environment, with the receive timestamp as filename. Submissions are rate limited and bounded per environment, and reports with an unexpected structure are rejected.Opt-in and privacy
usage.report.interval = 0. An operator has to explicitly set an interval (7 days recommended) and restart the Management Server.usage.report.uridefaults tohttps://reporting.cloudstack.org/reportand can be pointed elsewhere. Only HTTPS is accepted; plain HTTP is refused.versiontable, so reports from the same environment can be correlated over time without identifying it.Open points for discussion
reporter/collector may be better off in a separate repository rather than in the main tree.Types of changes
How Has This Been Tested?
Manually against a local Management Server with
usage.report.intervalset to a low value and the collector running locally behind HTTPS. Verified that nothing is sent with the default configuration.