feat: heatmap label steps and margin, bar sort_by, thresholds on horizontal bars

CHANGELOG.md

@@ -5,8 +5,20 @@ while the major version is 0, minor bumps may include breaking changes and say s
 
 ## Unreleased
 
+### Added
+
+- `sort_by` on a categorical bar ranks it by something other than its first metric, largest first, on every supported release. `"total"` ranks stacked or grouped bars by their sum. Before, Superset drew several series in name order: two stacked metrics on 4.1.4 and 5.0.0, and a grouped bar on every release. A metric (`"sort_by": "SUM(revenue)"`) ranks by its value and orders the query too, so a `row_limit` keeps the top bars by it; it need not be drawn, so two charts can share one order. `advise` warns when a `row_limit` cuts a bar ranked by `"total"`, since the query is still ordered by the first metric.
+- A threshold on a horizontal bar: `annotations` there now say what Superset draws, a line standing upright at its value (a 1.0x threshold, a 4-hour limit), seen on every supported release, and the live CI fixture builds one.
+- Heatmap axes: `x_label_every` and `y_label_every` label every Nth column or row, counted from the first, so an hour axis with `"x_label_every": 6` reads 0, 6, 12, 18 instead of Superset's uneven automatic spacing. `left_margin` leaves room, in px, left of the row labels. Every supported release takes all three; `decompile` reads them back and `plan` reports a change made in the UI.
+
+### Changed
+
+- Documented: no supported release gives one line of a mixed chart its own width or dash. The plugin has no such control and its transform passes none, and 6.1's ECharts Options replace the series list whole. Set a ghost line apart by colour with `dashboard.label_colors`, or draw a fixed level as an annotation.
+- Design brain 12: `advise --fix` fills `left_margin: 16` on every heatmap (`default.heatmap-label-room`). On Superset 6.1.0 a heatmap's longest row label lost its first letters at the card's edge, and a theme can't reach the heatmap there. Write `left_margin` yourself, or ignore the rule for a chart, to keep your own value.
+
 ### Fixed
 
+- `decompile` names a bar ranked by its series' minimum, maximum or average, or ranked smallest first; before, apply quietly put the largest-first ranking back.
 - `decompile` (and so `adopt`) names more chart settings it can't carry when they differ from Superset's defaults: big-number font sizes and a hidden trendline, a smooth or step line, horizontal bars on a time axis, a pie's radius and hidden labels, a funnel's labels, tooltip and percentage calculation, treemap labels, a heatmap's axis sort, legend, margins, label intervals and value bounds, a histogram's normalize setting, and any chart's currency format. Before, they were dropped without a note.
 
 - `standards apply` no longer adds a header or footer row that the body already holds at its edge (the last rows for a footer, the first for a header). A dashboard built in the UI has no header or footer, so `decompile` and `adopt` read its legal line as a body row, and the standard's footer was then added below it, showing it twice. `--claim` moves such a row into the header or footer and records it; `explain` and the apply summary name it.

chartwright/compiler.py

@@ -301,6 +301,36 @@ def metric(m: str):
                     "hasCustomLabel": False,
                 }
                 p["order_desc"] = chart.category_sort == "desc"
+        elif chart.sort_by:
+            # Largest first, as the default ranking: on top of the bottom-up horizontal axis.
+            ascending = chart.orientation == "horizontal"
+            sort_metric = chart.sort_metric()
+            if sort_metric is None:
+                # "total": the plugin sorts the categories by the sum of their series
+                # (SortSeriesType.Sum, utils/series.ts sortRows: 4.1.4 :200-208, 6.1.0
+                # :564-572), reading x_axis_sort from 6.0.0 (Timeseries/transformProps.ts
+                # 6.1.0 :335-336) and x_axis_sort_series at 4.1.4 and 5.0.0 (:243-246,
+                # :246-249). "sum" is no label, so sortOperator.ts adds no sort of its own.
+                p["x_axis_sort"] = "sum"
+                p["x_axis_sort_series"] = "sum"
+                p["x_axis_sort_series_ascending"] = ascending
+            else:
+                # A metric: sortOperator.ts sorts the rows by it (no groupby, all three
+                # releases). As the "Sort query by" metric it also orders the query, so a
+                # row limit keeps the top bars by it (normalizeOrderBy, descending), and
+                # one that is not drawn is queried but never drawn: extractExtraMetrics.ts:35
+                # adds it because its label is x_axis_sort, and extractSeries skips it.
+                payload = metric(sort_metric)
+                p["x_axis_sort"] = payload["label"] if isinstance(payload, dict) else payload
+                p["timeseries_limit_metric"] = payload
+                if chart.several_series():
+                    # Several series at 4.1.4 and 5.0.0: the plugin re-sorts the rows by
+                    # x_axis_sort_series, whose panel default is the category name, unless
+                    # it is unset (isDefined, utils/series.ts extractSeries, 4.1.4
+                    # :296-307, 5.0.0 :302-313). Null keeps sortOperator's order there;
+                    # 6.0.0 dropped the control.
+                    p["x_axis_sort_series"] = None
+            p["x_axis_sort_asc"] = ascending
         else:
             # Rankings read sorted by their measure, not by label order. The
             # horizontal axis renders bottom-up, so ascending puts the largest
@@ -461,6 +491,18 @@ def metric(m: str):
             p["show_percentage"] = False  # the control's default is true
         if chart.number_format:
             p["y_axis_format"] = chart.number_format  # the cell values' format
+        # xscale_interval / yscale_interval: transformProps hands interval N - 1 to the
+        # category axis, so N labels every Nth value from the first (Heatmap/transformProps.ts
+        # 4.1.4 :231,238, 5.0.0 :234,241, 6.1.0 :436,445). left_margin is the grid's left
+        # edge, labels inside it (containLabel, :166-169, :168-171, :354-357); 'auto' is 0.
+        # On 6.1.0 Heatmap.tsx:25 renders <Echart> without vizType, so a theme's per-type
+        # overrides never reach it and these are the only way to set the axes.
+        if chart.x_label_every is not None:
+            p["xscale_interval"] = chart.x_label_every
+        if chart.y_label_every is not None:
+            p["yscale_interval"] = chart.y_label_every
+        if chart.left_margin is not None:
+            p["left_margin"] = chart.left_margin
     elif t == "histogram":
         p["column"] = chart.column
         p["bins"] = chart.bins

chartwright/decompile.py

@@ -161,6 +161,18 @@ def _x_labels_to_spec(p: dict, out: dict, losses: list, name: str) -> None:
 _COLUMN_CONFIG_SPEC = {v: k for k, v in COLUMN_CONFIG_KEYS.items()}  # d3NumberFormat -> number_formats
 
 
+# SortSeriesType (superset-ui-chart-controls types.ts, 4.1.4 :503, 6.1.0 :549): what a bar
+# with several series sorts its categories by.
+_SERIES_SORTS = ("name", "sum", "min", "max", "avg")
+
+
+def _stored_label(m) -> str | None:
+    """A stored metric's label, as getMetricLabel reads it: a saved metric is its name."""
+    if isinstance(m, str):
+        return m
+    return m.get("label") if isinstance(m, dict) else None
+
+
 def _number(v) -> float | int | None:
     """A stored bound or size as a number; None for null, "" or junk."""
     if isinstance(v, bool) or v in (None, ""):
@@ -655,12 +667,37 @@ def keep_row_limit() -> None:
         # series: x_axis_sort from 6.0.0, x_axis_sort_series at 4.1.4 and 5.0.0).
         several = bool(p.get("groupby")) or len(ms) > 1
         sort, stored_asc = p.get("x_axis_sort"), p.get("x_axis_sort_asc")
-        if several and sort in (None, "") and p.get("x_axis_sort_series") == "name":
-            sort, stored_asc = "name", p.get("x_axis_sort_series_ascending", True)
+        if several and sort in (None, "") and p.get("x_axis_sort_series") in _SERIES_SORTS:
+            sort = p["x_axis_sort_series"]
+            stored_asc = p.get("x_axis_sort_series_ascending", True)
         if sort == x or (several and sort == "name"):
             # In reading order (the horizontal axis runs bottom-up).
             ascending = bool(stored_asc) != (p.get("orientation") == "horizontal")
             out["category_sort"] = "asc" if ascending else "desc"
+        elif sort not in (None, ""):
+            # A ranking by a measure, largest first (on top of a horizontal bar): the
+            # series' sum ("total"), or a metric, which compile also makes the query's
+            # "Sort query by" (timeseries_limit_metric), drawn or not.
+            ranked = p.get("timeseries_limit_metric")
+            first = metric_label(ms[0])
+            if several and sort == "sum":
+                out["sort_by"] = "total"
+            elif several and sort in _SERIES_SORTS:
+                losses.append(Loss(name, f"bars ranked by the series' {sort!r} value not "
+                                         "preserved (the first metric on re-apply)"))
+            elif ranked and _stored_label(ranked) == sort and not p.get("groupby"):
+                m = metric_one(ranked)
+                if m:
+                    out["sort_by"] = m
+            elif sort != first and not p.get("groupby"):
+                m = next((m for m in ms if metric_label(m) == sort), None)
+                if m:
+                    out["sort_by"] = m
+            largest_first = p.get("orientation") == "horizontal"
+            if stored_asc is not None and bool(stored_asc) != largest_first and (
+                    "sort_by" in out or sort == first):
+                losses.append(Loss(name, "bars ranked smallest first not preserved "
+                                         "(largest first on re-apply)"))
         if p.get("y_axis_format") not in (None, "SMART_NUMBER"):
             out["number_format"] = p["y_axis_format"]
         gb = _groupby_one(p, losses, name)
@@ -832,6 +869,20 @@ def keep_row_limit() -> None:
             out["number_format"] = p["y_axis_format"]
         if p.get("show_legend") is False:
             out["show_legend"] = False
+        # Superset's own values (-1 = automatic spacing, 'auto' = no margin) map to nothing.
+        for key, spec_field in (("xscale_interval", "x_label_every"),
+                                ("yscale_interval", "y_label_every")):
+            every = _number(p.get(key))
+            if isinstance(every, int) and 1 <= every <= 50:
+                out[spec_field] = every
+            elif every not in (None, -1):
+                losses.append(Loss(name, f"heatmap {key} {p[key]!r} not preserved (automatic on re-apply)"))
+        margin = _number(p.get("left_margin"))
+        if isinstance(margin, int) and 0 <= margin <= 200:
+            out["left_margin"] = margin
+        elif p.get("left_margin") not in (None, "", "auto"):
+            losses.append(Loss(name, f"heatmap left_margin {p['left_margin']!r} not preserved "
+                                     "(none on re-apply)"))
         keep_row_limit()
     elif spec_type == "histogram":
         col = p.get("column")
@@ -958,6 +1009,8 @@ def keep_row_limit() -> None:
         mapped_here = mapped_here | {"x_axis_title", "y_axis_title"}
     if spec_type == "pie":
         mapped_here = mapped_here | {"show_total"}
+    if spec_type == "heatmap":
+        mapped_here = mapped_here | {"xscale_interval", "yscale_interval", "left_margin"}
     if spec_type in _AXIS_TYPES:
         _x_labels_to_spec(p, out, losses, name)
         mapped_here = mapped_here | _axis_to_spec(p, out, losses, name, spec_type)

chartwright/design/brief.py

@@ -89,8 +89,8 @@ def render_brief(audience: str, overlay: Overlay | None = None) -> str:
         "## Design defaults: `advise --fix` fills these, so leave them unset unless the user asks",
         "",
         "- x_label_format, compare_suffix, number_format of COUNT metrics, and on tables cell_bars,",
-        "  page_length and search_box; show_legend on a single named series; show_value on few bars.",
-        "  Set one yourself only when the user asks for it, and say so in your reply.",
+        "  page_length and search_box; show_legend on a single named series; show_value on few bars;",
+        "  a heatmap's left_margin. Set one yourself only when the user asks for it, and say so in your reply.",
         "- Run `chartwright advise <spec> --fix` (MCP: fix_spec) before you build. It writes each value",
         "  into the spec file and records it in design.filled; its `fixed` entries say kind \"fill\" and why.",
         "- Edit the spec it wrote; never regenerate the spec from your own copy, or the fills are lost.",

chartwright/design/defaults.py

@@ -135,15 +135,15 @@ def _findings(ctx: RuleContext, fill: Fill):
 
 
 def _fill(rule_id: str, field: str, types, doc: str, *, superset, superset_text: str,
-          override: str):
+          override: str, since: str = "5"):
     def deco(decide):
         f = Fill(rule_id, field, frozenset(types), decide, superset, superset_text, override)
         FILLS[rule_id] = f
 
         def fn(ctx: RuleContext):
             yield from _findings(ctx, f)
 
-        rule(rule_id, "info", doc, fixable=True, since="5")(fn)
+        rule(rule_id, "info", doc, fixable=True, since=since)(fn)
         return decide
     return deco
 
@@ -394,3 +394,25 @@ def _value_labels(ctx: RuleContext, c):
     if w < min_w:
         return None, f"{w}/12 wide, narrower than {min_w}/12"
     return True, f"at most {n} bars at {w}/12 wide: each value reads without the axis"
+
+
+# -- heatmaps -----------------------------------------------------------------------
+
+# A heatmap's y labels sit inside its grid (containLabel), whose left edge is the card's
+# edge while left_margin is Superset's 'auto' (Heatmap/transformProps.ts 6.1.0 :354-357).
+# Seen on 6.1.0: the longest label is drawn wider than the room the grid gave it and
+# loses its first letters there ('rucks and Buses', a 95 px label; 'ustralian Gift
+# Network, Co', 150 px). 8 px cleared the first and 16 px both; 4.1.4 and 5.0.0 drew
+# them whole. Its own spec field is the only way: on 6.1.0 Heatmap.tsx:25 renders
+# <Echart> without vizType, so a theme's per-chart-type overrides never reach it.
+HEATMAP_LABEL_ROOM = 16
+
+
+@_fill("default.heatmap-label-room", "left_margin", {"heatmap"},
+       "a heatmap keeps 16 px left of its y labels, where Superset 6.1.0 cuts off the "
+       "longest one's first letters",
+       superset=None, superset_text="no margin (Superset's 'auto')",
+       override="write left_margin yourself (0 for none)", since="12")
+def _heatmap_label_room(ctx: RuleContext, c):
+    return HEATMAP_LABEL_ROOM, ("Superset 6.1.0 cuts the first letters off the longest y "
+                                "label at the card's edge; 16 px keeps labels up to ~300 px whole")

chartwright/design/guidelines/chart-choice.md

@@ -22,11 +22,18 @@
   scrollbar; smoke warns at apply time when they do.
 - Sort order is per family: bars sort by their first metric (right for
   rankings, wrong for ordinals like weekday/month) unless `category_sort`
-  orders them by label; heatmap and pivot categories sort alphabetically
+  orders them by label. Stacked or grouped bars need `sort_by: "total"` to
+  rank: Superset puts several series in name order (4.1.4/5.0.0 always, a
+  groupby everywhere). `sort_by` a metric ranks by one the chart needn't draw,
+  so two charts share an order. Heatmap and pivot categories sort alphabetically
   (a pivot can sort by value: `row_order`). For ordinal dimensions, chart an
   order-encoded label column (labels prefixed with a sort index: '1-Mon')
   if the dataset has one, and give a bar `category_sort: "asc"`. An ordered
   axis such as hours stays vertical: add `x_label_every` so every label shows.
+  A heatmap's `x_label_every`/`y_label_every` is a step: 6 on hours reads 0, 6, 12, 18.
+- No release gives a mixed chart's line its own width or dash: set a ghost line
+  apart by colour (`dashboard.label_colors`, a pale grey on its series label);
+  a fixed level is an annotation, which takes `style` and `width`.
 - A paged table (`page_length`) needs room for one page and its pager, not
   for every row; a timeseries with many groups keeps the top few with
   `series_limit`.

chartwright/design/model.py

@@ -26,7 +26,12 @@
 # "7" = fills keep a null record when the author edits one (an edit later deleted stays
 # deleted), and default.stale-record: a fixable finding for a renamed or removed chart's
 # design.filled entry, which now validates. Both change what `--fix` writes.
-DESIGN_BRAIN_VERSION = "7"
+# "12" = default.heatmap-label-room, a fill (left_margin on every heatmap, which 6.1.0
+# otherwise draws with its longest y label cut off); it changes what `--fix` writes.
+# With it, a bar's sort_by: data.top-n-sort warns on a row_limit under sort_by "total"
+# (the query is still ordered by the first metric), and chart.ordinal-order leaves a
+# bar ranked by sort_by alone. ("8" to "11" are other branches'.)
+DESIGN_BRAIN_VERSION = "12"
 
 KPI_TYPES = {"big_number_total", "big_number_trend"}
 TIMESERIES_TYPES = {"timeseries_line", "timeseries_bar", "timeseries_area", "timeseries_scatter"}

chartwright/design/rules.py

@@ -766,6 +766,8 @@ def dims(c):
 
     for c in ctx.spec.charts:
         hits = [d for d in dims(c) if d and _ORDINAL_RE.search(d)]
+        if c.type == "bar" and c.sort_by:
+            continue  # ranked by a measure the author chose: the order is deliberate
         if hits and c.type == "bar" and not c.category_sort:
             # A bar sorts by its first metric unless category_sort is set, so an
             # order-encoded label alone changes nothing.
@@ -863,6 +865,21 @@ def top_n_sort(ctx: RuleContext):
                 "the query by the series ranking, so the axis shows gaps; drop row_limit, "
                 "or drop series_limit so the query is ordered by the category",
             )
+        elif (c.type == "bar" and c.sort_by == "total" and c.row_limit is not None
+                and c.row_limit < DEFAULT_ROW_LIMIT["bar"]):
+            # "total" is the plugin's sort of the rows it gets (SortSeriesType.Sum); the
+            # query is still ordered by the first metric (normalizeOrderBy.ts), so the
+            # row limit cuts by that metric before the bars are ranked by their sum.
+            remedy = ("write the total as a metric (e.g. \"SQL(SUM(a) + SUM(b)) AS Total\") "
+                      "and sort_by it, which orders the query too" if not c.groupby else
+                      "with a groupby the limit also counts each bar's segments, not bars: "
+                      "drop row_limit, or filter the categories")
+            yield Finding(
+                "data.top-n-sort", "warn", c.name, ctx.where(c.name),
+                f"row_limit {c.row_limit} with sort_by \"total\" keeps the rows with the "
+                f"largest {c.metrics[0]} and then ranks them by their total, so a bar with "
+                f"a large total but a small {c.metrics[0]} can be left out; {remedy}",
+            )
 
 
 _RANGE_DAYS = {"day": 1, "week": 7, "month": 30.4, "quarter": 91, "year": 365}

chartwright/resolver.py

@@ -317,6 +317,8 @@ def _check_chart_fields(chart, ds: ResolvedDataset, res: Resolution) -> None:
     elif t == "bar":
         for m in chart.metrics:
             _check_metric(m, chart.name, ds, res)
+        if chart.sort_metric() and chart.sort_metric() not in chart.metrics:
+            _check_metric(chart.sort_metric(), chart.name, ds, res)
         _check_column(chart.x_column, chart.name, ds, res, "x_column")
         if chart.groupby:
             _check_column(chart.groupby, chart.name, ds, res, "groupby")

chartwright/smoke.py

@@ -144,6 +144,9 @@ def metric(m: str):
     elif t == "bar":
         q["metrics"] = [metric(m) for m in chart.metrics]
         q["columns"] = [chart.x_column] + ([chart.groupby] if chart.groupby else [])
+        if chart.sort_metric() and chart.sort_metric() not in chart.metrics:
+            # The chart queries a sort metric it doesn't draw (extractExtraMetrics.ts).
+            q["metrics"].append(metric(chart.sort_metric()))
     elif t == "pie":
         q["metrics"] = [metric(chart.metric)]
         q["columns"] = [chart.groupby]