Summary
Currently, metrics-by-tag reporting requires explicit @pytest.mark markers on each test. In practice, most tests don't have these markers — they rely on the folder structure to indicate what they test.
Since the test hierarchy already encodes the feature (e.g., core/operator/expressions/string/replaceAll/), we should derive tags automatically from the file path.
Current test distribution (449 test functions)
core/ (449)
operator/ (399)
expressions/ (260)
string/ (57) — replaceAll(12), replaceOne(14), strcasecmp(13), ...
accumulator/ (55) — max(13), min(13), first(7), last(7), setUnion(8), ...
arithmetic/ (20) — divide(4), add(1), abs(1), ...
date/ (22) — toDate(11), dateAdd(1), ...
boolean/ (12) — or(10), and(1), not(1)
conditional/ (11) — switch(9), cond(1), ifNull(1)
array/ (20), type/ (21), set/ (6), ...
stages/ (41) — group(6), match(5), ...
query/ (50) — bitwise(16), comparison(8), geospatial(11), ...
update/ (22) — array(8), fields(9), modifiers(4)
accumulators/ (23) — smoke tests
query-and-write/ (29) — find(22), insert(7)
collections/ (11), aggregation/ (4), cursors/ (2), indexes/ (4)
Proposed report format
Top-level summary (always shown):
| Area |
Tests |
Pass Rate |
| expressions |
260 |
97.7% |
| stages |
41 |
90.2% |
| query |
50 |
94.0% |
| update |
22 |
95.5% |
Collapsible drill-down (GitHub step summary):
See showcase comment below for a rendered example using <details> tags.
Failures-only view (flat, for quick triage):
| Path |
Failed |
Total |
core/operator/stages/match |
2 |
5 |
core/operator/expressions/arithmetic/abs |
1 |
1 |
core/operator/expressions/string/toLower |
1 |
1 |
Text output with configurable depth:
python -m documentdb_tests.compatibility.result_analyzer --group-by-path --depth 2
Tasks
Notes
- Aligns with the feature tree structure in
docs/feature-tree.csv
- The
nodeid is already available in pytest JSON reports — no test code changes needed
- Explicit markers are still useful for cross-cutting tags but not needed for feature categorization
Summary
Currently, metrics-by-tag reporting requires explicit
@pytest.markmarkers on each test. In practice, most tests don't have these markers — they rely on the folder structure to indicate what they test.Since the test hierarchy already encodes the feature (e.g.,
core/operator/expressions/string/replaceAll/), we should derive tags automatically from the file path.Current test distribution (449 test functions)
Proposed report format
Top-level summary (always shown):
Collapsible drill-down (GitHub step summary):
See showcase comment below for a rendered example using
<details>tags.Failures-only view (flat, for quick triage):
core/operator/stages/matchcore/operator/expressions/arithmetic/abscore/operator/expressions/string/toLowerText output with configurable depth:
Tasks
<details>tags--depthflag for text output to control grouping level@pytest.markmarkers for cross-cutting concerns (smoke,slow,replica)Notes
docs/feature-tree.csvnodeidis already available in pytest JSON reports — no test code changes needed