diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml new file mode 100644 index 0000000..ebd05c5 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -0,0 +1,39 @@ +name: Bug report +description: Something does not work as documented. +labels: ["bug"] +body: + - type: textarea + id: what + attributes: + label: What happened + description: What you saw, including error output. + validations: + required: true + - type: textarea + id: expected + attributes: + label: What you expected + validations: + required: true + - type: textarea + id: reproduce + attributes: + label: How to reproduce + description: Commands, pipeline YAML or config, and input data. Keep it minimal. + render: shell + validations: + required: true + - type: input + id: version + attributes: + label: Commit + description: Output of `git rev-parse HEAD`. + validations: + required: true + - type: textarea + id: environment + attributes: + label: Environment + description: OS, CPU, Rust version (`rustc --version`), and features used. + validations: + required: false diff --git a/.github/ISSUE_TEMPLATE/capability.yml b/.github/ISSUE_TEMPLATE/capability.yml new file mode 100644 index 0000000..516c5fd --- /dev/null +++ b/.github/ISSUE_TEMPLATE/capability.yml @@ -0,0 +1,46 @@ +name: New capability +description: A new sketch function or use case. +labels: ["capability"] +body: + - type: markdown + attributes: + value: | + A capability answers one question about telemetry with a sketch. + See RFC 0001 (docs/rfcs/0001-sketch-telemetry-functions.md) for the existing use cases. + - type: textarea + id: question + attributes: + label: Question it answers + description: The question a user asks, in one or two sentences. + placeholder: Which attribute keys explode in cardinality? + validations: + required: true + - type: input + id: sketch + attributes: + label: Sketch + description: The asap_sketchlib sketch and what it is keyed by. + placeholder: HLL per (metric, attribute key) + validations: + required: true + - type: textarea + id: output + attributes: + label: Output + description: The fields a user sees (keys, counts, quantiles, labels) and where they go (metrics, logs, ASAPQuery). + validations: + required: true + - type: textarea + id: bound + attributes: + label: Accuracy bound + description: The error bound reported next to each answer, and the sketch parameters it depends on. + validations: + required: true + - type: textarea + id: ground-truth + attributes: + label: Ground truth + description: The exact answer it is measured against, and how the workload makes that answer known. + validations: + required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 0000000..1262011 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,5 @@ +blank_issues_enabled: false +contact_links: + - name: Long or cross-cutting design (RFC) + url: https://github.com/ProjectASAP/ASAPCollector-public/tree/main/docs/rfcs + about: Write an RFC in docs/rfcs instead of an issue. See docs/rfcs/README.md. diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml new file mode 100644 index 0000000..2cb9c80 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -0,0 +1,36 @@ +name: Feature request +description: A change to existing behavior, tooling or docs. +labels: ["enhancement"] +body: + - type: markdown + attributes: + value: | + For a new sketch function or use case, use the capability form. + For a long or cross-cutting design, write an RFC in docs/rfcs. + - type: textarea + id: problem + attributes: + label: Problem + description: What is hard or impossible today. + validations: + required: true + - type: textarea + id: proposal + attributes: + label: Proposed change + description: Describe it as before -> after. + validations: + required: true + - type: textarea + id: performance + attributes: + label: Expected performance impact + description: Effect on throughput, CPU, memory or latency, or "none". + validations: + required: true + - type: textarea + id: alternatives + attributes: + label: Alternatives considered + validations: + required: false diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md new file mode 100644 index 0000000..eab2517 --- /dev/null +++ b/.github/pull_request_template.md @@ -0,0 +1,28 @@ +## What changed + + + +Before: + +After: + +## How verified + + + +## Performance impact + + + +## Linked issue or RFC + + + +## Checklist + +Run in `asap-precompute-rs/`: + +- [ ] `cargo fmt --check` +- [ ] `cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings` +- [ ] `cargo test --features otap-engine,stream-benchmark` +- [ ] This is one of at most two open PRs I have. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..7ad76d3 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,108 @@ +# Contributing + +This guide is for people and for AI agents. Follow it as written. If a step +does not fit your change, say so in the PR description. + +The project follows OpenTelemetry practice for roles, review and design +documents. + +## Roles + +| Role | Who | What they do | +| --- | --- | --- | +| Contributor | Anyone | Opens issues and PRs, reviews any PR. | +| Approver | Listed in [MAINTAINERS.md](MAINTAINERS.md) | Reviews PRs. An approver's approval counts toward merge. | +| Maintainer | Listed in [MAINTAINERS.md](MAINTAINERS.md) | Merges PRs, cuts releases, accepts RFCs, sets project direction. | + +To become an approver: + +- Contributed for at least one month. +- Authored several merged PRs that are not trivial. +- Reviewed several PRs with useful comments. +- Sponsored by one maintainer, with no objection from the others. + +To become a maintainer: + +- Has been an approver for at least two months. +- Has reviewed a substantial share of PRs in that time. +- Shows good judgement on design, performance and accuracy. +- Sponsored by one maintainer and approved by a majority of maintainers. + +Changes to roles are made by a PR to `MAINTAINERS.md`. People who stop +contributing move to the emeritus list. + +## Before you start + +- Bugs, small features and new capabilities: open an issue with the matching + form in `.github/ISSUE_TEMPLATE/`. +- Long or cross-cutting designs: write an RFC in [docs/rfcs](docs/rfcs/README.md). +- Discuss larger changes in an issue or in the weekly sync before writing code. + +## Pull requests + +- At most two open PRs per contributor. Finish or close one before opening a + third. +- Keep PRs small and reviewable. One concern per PR. Split refactors from + behavior changes. +- Fill in the PR template: + - What changed, written as before -> after. + - How you verified it: commands run, tests added, output checked. + - Performance impact: measured numbers when the change can affect + performance, otherwise "none". + - The linked issue or RFC. +- Keep the PR up to date with `main` and answer review comments. +- An approver or maintainer approves, and a maintainer merges. + +## Design principles + +1. **Performance is a feature.** Every change keeps the end-to-end benchmark + green. A change that can affect throughput, CPU, memory or latency reports + measured numbers. +2. **Accuracy is stated as a bound next to every answer.** An estimate without + its error bound is not a finished feature. +3. **Sketches use asap_sketchlib.** Use asap_sketchlib's implementations and the + ASAPv1 wire format only. Fix gaps in asap_sketchlib, not with a local copy. +4. **Upstream-compatible with OTAP, no forks.** Nodes register through the + official OTAP Dataflow factories and depend on pinned upstream crates. + +## Local checks + +Run these in `asap-precompute-rs/` before you push. They must all pass. + +```sh +cd asap-precompute-rs +cargo fmt --check +cargo clippy --all-targets --features otap-engine,stream-benchmark -- -D warnings +cargo test --features otap-engine,stream-benchmark +``` + +## Benchmarks + +The benchmark is the evidence that a function is cheap enough to leave on. +Three tiers run in CI: + +| Tier | When | Scope | Gate | Workflow | +| --- | --- | --- | --- | --- | +| Smoke | Every PR and every merge to `main` | fmt, clippy, tests, then `deploy/telemetry-functions/acceptance.py` on real collector containers | Fails on wrong answers | `ci.yml` | +| Nightly sweep | Every day | Full rate sweep per scenario | Fails when throughput drops more than the margin (default 10%) against the previous night | `asap-sketch-benchmark.yml` | +| PR comparison | A maintainer comments `/benchmark` on a PR | Reduced sweep, this PR vs `main` | Posts a comparison table | `benchmark-on-demand.yml` | + +Capacity numbers need dedicated CPUs: the nightly and on-demand tiers run on +the runner named by the `BENCH_RUNNER` repository variable (see +`benchmarks/README.md`). For a performance-related change, also paste the +relevant numbers into the PR. + +## Weekly sync + + +**Placeholder: time and link to be added by maintainers.** + +Anyone can join. Bring RFCs for review and PRs that are stuck. + +## Notes for AI agents + +- Follow every rule above, including the two-open-PR limit. +- Run the local checks and paste their result into "How verified". +- Do not invent benchmark numbers. Report only numbers you measured, with the + command that produced them. +- Do not change accepted RFCs except as described in [docs/rfcs](docs/rfcs/README.md). diff --git a/MAINTAINERS.md b/MAINTAINERS.md new file mode 100644 index 0000000..6be461a --- /dev/null +++ b/MAINTAINERS.md @@ -0,0 +1,12 @@ +# Maintainers and approvers + +See [CONTRIBUTING.md](CONTRIBUTING.md#roles) for what each role does and how +to join. + +| Name | GitHub | Affiliation | Role | +| --- | --- | --- | --- | +| _To be filled in by the repository owner_ | | | Maintainer | + +## Emeritus + +None yet. diff --git a/README.md b/README.md index 4c23198..07b52dc 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,12 @@ and `resolve()` — the `sketch_type` dispatch table (factory + observer + `SketchType`). All of this sits on the host-neutral runtime in the same crate (`precompute`, `window`, `sketches/*`). +## Contributing + +See [CONTRIBUTING.md](./CONTRIBUTING.md) for roles, PR rules, design +principles and the local checks. Long or cross-cutting designs are written +as RFCs in [`docs/rfcs/`](./docs/rfcs/README.md). + ## Build & test For a continuous KLL vs exact comparison with every processor in its own diff --git a/docs/rfcs/0001-sketch-telemetry-functions.md b/docs/rfcs/0001-sketch-telemetry-functions.md new file mode 100644 index 0000000..86577b8 --- /dev/null +++ b/docs/rfcs/0001-sketch-telemetry-functions.md @@ -0,0 +1,151 @@ +# RFC 0001: Sketch-based telemetry functions in the OTAP collector: design + +| Field | Value | +| --- | --- | +| Status | Accepted | +| Date | 2026-10-12 | +| Owner | ProjectASAP maintainers | + +Lines marked "Update after approval" record facts that changed after the design was accepted. Struck-through text is the original wording they replace. + +## Context and goals + +We will build three sketch-based telemetry functions inside the OTAP collector, one per category in the proposal, and show their cost and accuracy with measured figures. The first audience is a technical webinar (engineers and technical managers); the second is the OpenTelemetry community. + +What exists today (ASAPCollector-public): + +- A multi-process demo and benchmark harness: branch A/B, merge and estimate run as separate processes, each pinned to one CPU, connected by OTLP/HTTP over TCP, with Prometheus and Grafana (PR #20). +- KLL quantiles vs exact: exact sustains 10,000 signals/s per source; KLL at least 400,000 (the highest tested), at 0.031 vs 0.63 CPU cores for the four processors at 10,000/s per source (PR #20). +- CMS + heap top-k vs exact: exact sustains 50,000/s per source; CMS + heap at least 400,000, with recall@10 = 1.0 and 0.62% count error (PR #22). + +Goals for about three months: + +1. A working prototype of one function per category, each driven by a concrete use case with known ground truth. +2. Measured cost of adding each function inline: throughput per CPU core, memory, added CPU and added end-to-end latency. +3. Measured accuracy against exact answers, shown to users with confidence bounds. +4. An artifact others can run: open-source code, nightly benchmarks in CI, and a blog post per use case. + +Non-goals for this milestone: sampling plus sketches, and sketches as an input for agents. Both follow once the pipeline cost is understood. + +## Use cases + +We pick one use case per category, in priority order: cardinality and drift first (the proposal's two named categories), heavy hitters third as a full use case, since PR #22 already provides its sketch and benchmark. Each one is written backwards from the blog post it should produce. + +| Priority | Category | Use case | Question the user asks | Sketch per 1-minute window | Event emitted | Ground truth in synthetic data | +| --- | --- | --- | --- | --- | --- | --- | +| 1 | Cardinality and uniqueness | Attribute stability and identifier detection | Which attribute keys explode in cardinality or behave like unique IDs (possible privacy risk)? | HLL per (metric, attribute key), plus a count | Per attribute: distinct estimate, distinct/total ratio, trend vs baseline, a stable / growing / identifier-like label | Generator sets distinct counts per attribute, including one ID-like and one exploding attribute | +| 2 | Distribution drift | Latency regression vs baseline | Did p50/p99, or the whole distribution, of a metric shift against the last hour? | KLL per (service, route) | Per series: current vs baseline quantiles, a distance (max CDF gap), drift yes/no with a bound | Generator changes one route's distribution at a known time; the telemetry shape stays the same | +| 3 | Heavy hitters and novelty | Noisy routes and new error signatures | Which keys dominate now, and which are new compared with the baseline? | CMS + heap (top-k) per attribute | Top-k with counts and error bound; keys new to the top-k vs baseline | Generator injects a new heavy key at a known time | + +Common to all three: + +- **Two stages.** Sketches are built inline at each source (stage 1) and merged for a global view (stage 2). The global answer is what the detector uses. +- **Readable output.** A sketch alone is not useful to a user. Each event carries the answer in plain fields (keys, counts, quantiles, labels) next to the sketch bytes, plus the error bound from the sketch configuration. +- **Measured against exact.** Each use case also runs an exact pipeline on the same traffic, as PRs #20 and #22 do, so accuracy and cost are compared directly. + +## Architecture + +Each function runs in two stages: an inline sketch at every source, and one merge plus detector for the global view. This is the topology PRs #20 and #22 already run, with a detector added after the merge. + +![Pipeline architecture: sketches are built at each source and merged once for a global answer](images/0001-architecture.png) + +- **Stage 1, inline.** A processor in each source's collector updates one sketch per (metric, attribute) per 1-minute window and emits it as an event. The raw telemetry continues unchanged, so the function can be turned off with no other effect. +- **Stage 2, global.** Merge combines the sources' sketches for a window. The detector compares the merged window with a baseline (the merge of the previous N windows) and emits drift, novelty or stability events. +- **Outputs.** Answers become Prometheus metrics with confidence bands (Grafana can draw the band from the configured error bound). Detection events go to Loki as OTLP logs. Sketches can also be stored in ASAPQuery for later queries. Merging can instead happen in ASAPQuery when that is cheaper. +- **Transport.** Sketches travel as ASAPv1 envelopes inside OTLP summaries today. Between OTAP processes an Arrow-native layout could avoid the decode; experiment 3 measures whether that matters. +- **Deployment.** ~~Each stage runs in its own container, pinned to its own CPU, and later on separate machines connected over the network.~~ *Update after approval:* every pipeline stage runs as the official OTAP engine (`asap-collector`, the upstream controller with the ASAP nodes linked in) in its own pinned container; multi-machine runs come later when machines are available. + +## Workload and ground truth + +We start with synthetic data whose answers are known, generated by an extended OTAP traffic generator; traffic with realistic distribution properties follows later. + +*Update after approval:* the official OpenTelemetry traffic generator cannot yet shape attribute cardinality, popularity or value distributions, or change them over time; that is planned upstream, and until then a temporary local receiver, `urn:asap:receiver:scenario_generator`, in this repository stands in. + +The current benchmark replays one fixed corpus per window: durations for KLL, Zipf(1.1) keys over 2^20 ranks for top-k. That is enough for throughput, but every window looks the same, so nothing drifts. The extension adds scenario knobs to the generator: + +| Knob | Drives | Example setting | +| --- | --- | --- | +| Distinct values per attribute key | Cardinality | `route`: 200, `user.id`: unique per event, `pod`: grows 5% per minute | +| Value distribution per series | Drift | `checkout` latency lognormal, its median doubling at minute 30 | +| Key popularity | Heavy hitters, novelty | Zipf(s) over N keys, plus a new key reaching the top 10 at minute 45 | +| Change schedule | All | When each change starts and how fast it ramps | +| Rate and number of sources | Cost | Signals/s per source, 2 to N sources | + +The telemetry shape (metric names, attribute keys, resource attributes) stays fixed while values change, which matches the proposal's drift case. ~~The generator writes the ground truth (exact distinct counts, quantiles and top-k per window) to a side file, so every figure can be scored without a second exact pipeline.~~ *Update after approval:* instead of a generator side file, each source's stream is split with the official `fanout` processor into the sketch pipeline and an exact (raw-forwarding) pipeline, whose answers are the ground truth. + +Split of work: we extend the generator and the scoring; our collaborators supply the scenario definitions (which attributes, which shifts) and, later, traffic with real distribution properties. + +## Evaluation and success metrics + +The key question is whether inline measurement is cheap enough to leave on. We answer it by comparing the same pipeline with and without each function, on the same machine and traffic. + +| Metric | How it is measured | What counts as success | +| --- | --- | --- | +| Ingestion throughput per CPU core | Highest sustained signals/s at the capacity criteria (delivery, backlog, p99 window latency), per pinned core | Close to the \~1M points/s per core reference from Jake's benchmark framework, with and without the function | +| Added CPU | Process CPU of each stage, with minus without the function, at the same offered load | A small, stated fraction of the base pipeline | +| Memory | Sampled and peak RSS per process | Bounded and independent of cardinality | +| Accuracy | Error against ground truth per window: relative distinct-count error, quantile or CDF error, recall@k and count error | Within the configured error bound, shown as a confidence band | +| End-to-end latency | Window close to answer at the backend, with and without the detector stage | Added latency well under one window | +| Egress | Bytes on the wire per stage | Sketch egress orders of magnitude below raw or exact | + +Experiments, in order: + +1. **Inline cost.** Pass-through vs pass-through plus sketch creation, per function, swept over rate and sources. Output: an enable or disable rule per function. +2. **Accuracy vs size.** Sweep sketch parameters (HLL precision, KLL k, CMS width and heap size) and plot error against memory and bytes. +3. **Serialization.** Split sketch encode and decode from update and merge using the existing CPU-scope profiler. Compare ASAPv1 MessagePack with an Arrow-native layout, where OTAP could avoid a decode. +4. **Merge at scale.** Merge 10, 100 and 1,000 sketches per window and time it. 1,000-way merges were flagged as possibly slow. +5. **Distributed run.** Stages on separate machines over the network, ending in Prometheus, to check the single-host numbers. +6. **Later: sampling plus sketches.** Only after experiments 1 to 4 show where the cost goes. + +## Open source, adoption and governance + +Adoption depends on a team being able to switch a function on, see its cost, and trust its answers. Continuous benchmarks give the evidence; a clear contribution process makes it safe for more people to work on the code. + +**What would give an operator the confidence to deploy:** + +- A published cost per function (CPU per core, memory, added latency) at their traffic rate, with an off switch. +- Accuracy stated as a bound and shown next to every answer, not hidden in a paper. +- No regression across releases, proven by benchmarks in CI. +- Components upstream in, or compatible with, OTAP, so the deployment does not depend on a fork. + +**Benchmarks in CI** (OpenTelemetry practice): + +| Tier | When | Scope | Gate | +| --- | --- | --- | --- | +| Continuous | Every merge to main | Short smoke run of each scenario, correctness checks | Fails on wrong answers | +| Nightly | Every day | Full rate sweep per scenario (the PR #19/#20/#22 harness) | Alerts on a throughput or CPU regression beyond a set margin | +| On demand | Comment on a PR | Same sweep, this PR vs main | Posts a comparison table | + +**Process for many contributors** (following OpenTelemetry): + +- Roles: contributors (anyone), approvers (review), maintainers (merge and release); a written bar for becoming a maintainer. +- A CONTRIBUTING guide: PR expectations, templates, how review works, and the design principles. It is written for both people and AI agents. +- Long designs go in an RFC doc in the repository; everything else in GitHub issues with bug, feature and capability templates. +- A weekly sync meeting; at most two open PRs per contributor. +- Design principles and success metrics fixed early, for example that performance matters, so the end-to-end benchmark is part of every change. + +## Milestones + +We ship one measured use case per phase, so each phase can end in a blog post and a demo. The dates are a proposal to agree with our collaborators. + +![Proposed milestones: four phases over about 13 weeks, each ending with a measured result](images/0001-milestones.png) + +Each gate means: the function runs inline and in the exact baseline, the nightly benchmark covers it, and the figures from the evaluation section exist for it. The webinar date still needs to go on this plan; whichever phase it falls in, the demo uses the latest completed gate. + +## Open questions + +For our collaborators: + +- [ ] Which attributes and shifts should the first synthetic scenarios use, and when can traffic with real distribution properties be provided? +- [ ] Is a 1-minute window right for the events, or do detectors need shorter panes merged into longer windows? +- [ ] Where should answers land first: Prometheus metrics, Loki (OTLP logs) events, or ASAPQuery? +- [ ] Which parts belong in the open-source project, and which stay internal? +- [ ] What added CPU or latency would make a function "always on" rather than opt-in? + +For us: + +- [ ] Drift distance: max CDF gap (Kolmogorov–Smirnov style) from KLL, or quantile shifts at fixed points? The bound is easier to state for the first. +- [ ] Arrow-native sketch layout: worth prototyping before experiment 3, or after its numbers? +- [ ] Upstream path to OTAP: which pieces (processor, sketch carrier, generator knobs) to propose first, and through which OpenTelemetry SIG. + +Next, after this milestone: sketches as the input for agents, so an agent sees summaries instead of raw telemetry (smaller and more private). Planned for next month. diff --git a/docs/rfcs/README.md b/docs/rfcs/README.md new file mode 100644 index 0000000..e502627 --- /dev/null +++ b/docs/rfcs/README.md @@ -0,0 +1,53 @@ +# RFCs + +An RFC (request for comments) is a design document kept in this repository. +It records what we decided and why, so later contributors can find it. + +## RFC or issue? + +Write an RFC when the design is long or cross-cutting. Examples: + +- It changes more than one component, such as a processor, the sketch wire + format and the benchmark harness together. +- It changes a public interface: a node URN, a config shape, or the sketch + carrier. +- It sets a plan or principle that many later PRs will follow. + +Use a GitHub issue for everything else: a bug, a small feature, or one new +capability that fits the capability issue form. If an issue discussion grows +into a design, move it to an RFC and link the two. + +## Numbering and files + +- Name the file `NNNN-short-title.md`, with the next free four-digit number. + The first RFC is `0001`. +- Put images in `images/`, named with the RFC number as a prefix, for example + `images/0001-architecture.png`. +- Start the RFC with a header table: Status, Date, Owner. +- Numbers are never reused, even if an RFC is withdrawn. + +## Statuses + +| Status | Meaning | +| --- | --- | +| Draft | Open for review. The PR that adds it is not merged yet, or it is merged for discussion only. | +| Accepted | Maintainers approved it. Code should follow it. | +| Superseded | A later RFC replaces it. Add "Superseded by RFC NNNN" to the header and keep the file. | + +After an RFC is accepted, do not rewrite it. If a fact changes, add a short +line marked "Update after approval" next to the original text, or write a new +RFC that supersedes it. + +## Review + +1. Open a PR that adds the RFC with Status: Draft. +2. Announce it in the weekly sync or in an issue. +3. Approvers and maintainers review it in the PR. +4. A maintainer sets Status: Accepted and merges it, or closes the PR with a + reason. + +## Index + +| RFC | Title | Status | +| --- | --- | --- | +| [0001](0001-sketch-telemetry-functions.md) | Sketch-based telemetry functions in the OTAP collector | Accepted | diff --git a/docs/rfcs/images/0001-architecture.png b/docs/rfcs/images/0001-architecture.png new file mode 100644 index 0000000..f7a88db Binary files /dev/null and b/docs/rfcs/images/0001-architecture.png differ diff --git a/docs/rfcs/images/0001-milestones.png b/docs/rfcs/images/0001-milestones.png new file mode 100644 index 0000000..e66352b Binary files /dev/null and b/docs/rfcs/images/0001-milestones.png differ