Repository navigation
Reports and CSV export #9827
Description
Activity
Hi @dfliess
Thanks for raising the issue.
Yes we are aware of this implementation gap and this is on our backlog.Thanks for confirming.
Do you have a rough timeframe for this? If it’s near-term, I’ll leave it with you. If it’s further out, I’d be happy to take it on.
I’ve gone through with Claude, and I’ve included a proposal below in case it’s useful. If you’d prefer to keep this in-house, no problem at all.The idea, in plain terms. The resolver framework already has an export hook, and nothing has ever called it. So this isn't about building an export mechanism — it's about connecting the one that already exists: let the download request carry a resolver instead of a legacy query, reuse the permission gate you already apply to resolvers elsewhere, and fill in the one missing implementation. Your own docs describe the end state already —
{{ .export }}is documented as "true when the API is being resolved for export (CSV, Excel, Parquet)", and it can never be true today.| `{{ .export }}` | `bool` | `true` when the API is being resolved for export (CSV, Excel, Parquet) | What's there today.
ResolveExporthas no call sites anywhere in the repo,ForExportis never set totrue, andsqlResolveris the only one of the 13 implementations that exports anything.Lines 56 to 57 in 545bf7b
// ResolveExport resolve data for export (e.g. downloads or reports). ResolveExport(ctx context.Context, w io.Writer, opts *ResolverExportOptions) error Proposal:
ExportRequestgainsresolver/resolver_properties/resolver_args— the same pairReportSpecandAlertSpecalready carry alongside their deprecated legacy fields — andExportgets the permission switchQueryResolveralready uses:metrics/metrics_sqlonReadMetrics, everything else onReadResolvers. Without that switch the new field would let any viewer mint a download token forresolver: sql, sinceExportonly checksReadMetricstoday.
rill/runtime/server/query_resolver.go
Lines 21 to 32 in 545bf7b
switch req.Resolver { case "metrics", "metrics_sql": // As a special case, we allow metrics resolvers for users with ReadMetrics permission (i.e. all users) if !claims.Can(runtime.ReadMetrics) { return nil, status.Error(codes.PermissionDenied, "not allowed to query metrics") } default: // Other resolvers require ReadResolvers permission (i.e. project admin) if !claims.Can(runtime.ReadResolvers) { return nil, status.Error(codes.PermissionDenied, "only project admins can query resolvers") } } ExportReportbuilds that request fromrep.Specfor any resolver other thanlegacy_metrics, which keeps its current translation, anddownloadHandlergrows one branch forrequest.Resolver != "".metricsResolver.ResolveExport, sharinggenerateExportHeadersand the format mapping withMetricsViewAggregation.Exportrather than duplicating them.metrics_sqlcomes along for free, since it returns ametricsResolver.IncludeHeader/OriginDashboard/OriginURLadded toResolverExportOptions. Priority and execution time stay inArgs, which every resolver already reads.- Keeping the transitive-access stripping exactly as it is, and
export.limitparity by injectingprops["limit"]the wayQueryResolverdoes.
Separately, and arguably its own bug:
sqlreports don't work for recipients at all today, before export enters the picture.sqlResolver.InferRequiredSecurityRulesreturns an error,ResolveTransitiveAccesspropagates it, andexpandTransitiveAccessRulesthen fails the whole security resolution — so a magic-token recipient of asqlreport can't resolve anything. Returningnil, nillooks right given the comment there, but that's a security call I'd rather you make.Lines 200 to 202 in 545bf7b
func (r *sqlResolver) InferRequiredSecurityRules() ([]*runtimev1.SecurityRule, error) { // NOTE - This is the regular SQL resolver, so the only refs would be to models, which don't have security policies / access checks return nil, errors.New("security rule inference not implemented") Happy to split it: proto + gate + handler; then
metricsexport with the shared headers; then thesqlsecurity fix; thenapiHandler ?format=so{{ .export }}matches its docs; then the UI mapper.Questions, if you do want a PR:
- Does extending
ExportRequestwith theQueryResolvergate work for you, or would you rather this stayed internal toExportReport? - Extend
ResolverExportOptionswith the three header fields, or converge it ontoruntime.ExportOptions? I'd extend — converging would duplicate whatArgsalready carries. Runtime.ResolveExportmirroringRuntime.Resolveso tracing and the billable query metric stay in one place, or initialise the resolver in the handler the wayalert.godoes?sqlResolver.InferRequiredSecurityRulesreturningnil, nil, or do you want real inference against the referenced models?- Backend first, then a shared mapper for the report and alert
openroutes? Both read the legacyquery_name/query_args_jsonpair today, so neither can open adata:report either.
Hey @dfliess
Unfortunately I don't have a concrete timeline for this.
We do appreciate your interest in contributing to this but we would like to keep this work in-house for now.Reacted by dfliess
Hi, I might be misunderstanding how reports are meant to work, so I would like to ask before assuming anything is broken.
The reports reference has an example titled "Example: query-based report with CSV export" that uses a
data:block withmetrics:, together withexport: format: csv:https://docs.rilldata.com/reference/project-files/reports#examples
When I copy that example, the report parses, reconciles and schedules without any error, and the email is delivered. But the download link in the email returns 400:
The check that rejects it seems to be here:
rill/runtime/server/downloads.go
Lines 78 to 83 in 14296a4
Reading around it, the export path looks older than the resolver abstraction: it rebuilds a legacy query proto from
query_nameandquery_args_jsonrather than going throughResolver.ResolveExport. And of the resolvers, onlysqlseems to implementResolveExport;metricsreturnsnot implemented:rill/runtime/resolvers/metrics.go
Lines 181 to 183 in 14296a4
Is
data:meant to support export for reports, so this is a gap that has not been filled yet? Or isquery:the only supported way to write a report that delivers a file, in which case the docs example may be worth updating?Happy to help with a PR if it would be useful, though I am not sure yet whether the right shape is routing downloads through
ResolveExportor something else.