Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
482e11e
add OpenTelemetry distributed tracing support
lovasoa Mar 8, 2026
f3a9ee2
fix OTel example: tracing init, Dockerfile, Tempo config
lovasoa Mar 8, 2026
c742abe
propagate trace context to PostgreSQL via application_name
lovasoa Mar 8, 2026
2bb370a
rewrite OTel example README with setup guides for all major providers
lovasoa Mar 8, 2026
9a879e5
fix todo example: use :title (POST variable) instead of $title
lovasoa Mar 8, 2026
ec9f3f9
set code.filepath and code.lineno span attributes to user SQL files
lovasoa Mar 8, 2026
0aebd0c
npm run fix
lovasoa Mar 8, 2026
712fb14
use stable OTel semantic convention attribute names
lovasoa Mar 8, 2026
85dd9b8
use OTel semantic convention values for db.system.name
lovasoa Mar 8, 2026
dafbac2
remove json-subscriber dependency
lovasoa Mar 8, 2026
2a8a89a
add Loki + Promtail log aggregation to OTel example
lovasoa Mar 8, 2026
d492ccc
add OTel spans for request parsing, rendering, sqlpage functions, and…
lovasoa Mar 8, 2026
cb7bdf6
add oidc.jwt.verify span for OIDC token verification
lovasoa Mar 8, 2026
0eeb5c3
add enduser.id attribute to oidc.jwt.verify span
lovasoa Mar 8, 2026
fab9350
add OTel user.* attributes to oidc.jwt.verify span
lovasoa Mar 8, 2026
ff8800e
add sqlpage.file.load span and attributes to http.parse_request
lovasoa Mar 8, 2026
f8fdc24
Fix clippy pedantic warnings
lovasoa Mar 8, 2026
3b0c37c
unify log format: logfmt with colors, no OTel noise
lovasoa Mar 8, 2026
1309725
use .instrument() instead of .entered() for async spans
lovasoa Mar 8, 2026
f884eab
preserve multi-line error formatting in terminal log output
lovasoa Mar 8, 2026
5bfaa08
use root Dockerfile for OTel example, add CARGO_PROFILE build arg
lovasoa Mar 8, 2026
3aaf82e
add db.query.parameter and db.response.returned_rows span attributes
lovasoa Mar 9, 2026
4c88b1e
Add official blog post about tracing
lovasoa Mar 9, 2026
bf7a863
add http.request.body.size and url.query span attributes
lovasoa Mar 9, 2026
10dec87
Replace Promtail with the OpenTelemetry Collector
lovasoa Mar 9, 2026
7e057e2
Refactor telemetry logging helpers
lovasoa Mar 9, 2026
7e156c5
Clamp traced fetch body size
lovasoa Mar 9, 2026
36aa465
Clamp traced fetch_with_meta body size
lovasoa Mar 9, 2026
e02ed67
Silence noisy PostgreSQL collector logs
lovasoa Mar 9, 2026
d285c57
make startup logs parseable
lovasoa Mar 9, 2026
0deb650
update terminal log formats
lovasoa Mar 9, 2026
8f7aeea
Ingest real PostgreSQL logs with trace IDs
lovasoa Mar 9, 2026
2ee9aa2
Use raw traceparent for PostgreSQL tracing
lovasoa Mar 9, 2026
80da041
Update opentelemetry example for PostgreSQL query events
lovasoa Mar 9, 2026
0a4920f
Add nginx logs to opentelemetry example
lovasoa Mar 9, 2026
a262215
Parse nginx error log severity correctly
lovasoa Mar 9, 2026
5704715
Log all span fields when debug logging is enabled
lovasoa Mar 9, 2026
ec43feb
Rename telemetry example directory
lovasoa Mar 9, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
479 changes: 472 additions & 7 deletions Cargo.lock

Large diffs are not rendered by default.

10 changes: 10 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,16 @@ openidconnect = { version = "4.0.0", default-features = false, features = ["acce
encoding_rs = "0.8.35"
odbc-sys = { version = "0.29.0", optional = true }

# OpenTelemetry / tracing
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["registry", "env-filter"] }
tracing-opentelemetry = "0.32"
tracing-actix-web = { version = "0.7", default-features = false, features = ["opentelemetry_0_31"] }
tracing-log = "0.2"
opentelemetry = "0.31"
opentelemetry_sdk = { version = "0.31", features = ["rt-tokio-current-thread"] }
opentelemetry-otlp = { version = "0.31", features = ["http-proto", "grpc-tonic"] }


[features]
default = []
Expand Down
2 changes: 2 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ FROM --platform=$BUILDPLATFORM rust:1.91-slim AS builder
WORKDIR /usr/src/sqlpage
ARG TARGETARCH
ARG BUILDARCH
ARG CARGO_PROFILE=superoptimized
ENV CARGO_PROFILE=$CARGO_PROFILE

COPY scripts/ /usr/local/bin/
RUN cargo init .
Expand Down
174 changes: 174 additions & 0 deletions examples/official-site/sqlpage/migrations/73_blog_tracing.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,174 @@
INSERT INTO blog_posts (title, description, icon, created_at, content)
VALUES
(
'Tracing SQLPage with OpenTelemetry and Grafana',
'How to inspect requests, SQL queries, and database wait time with distributed tracing',
'route-2',
'2026-03-09',
'
# Tracing SQLPage with OpenTelemetry and Grafana

When a page is slow, a log line telling you that the request took 1.8 seconds is only the start of the investigation. What you usually want to know next is where that time went:

- Did the request wait for a database connection?
- Which SQL file was executed?
- Which query took the longest?
- Did the delay start in SQLPage, in the reverse proxy, or in the database?

SQLPage now supports [OpenTelemetry](https://opentelemetry.io/), the standard way to emit distributed traces. Combined with Grafana, Tempo, Loki, and an OpenTelemetry collector, this gives you a detailed timeline of each request and lets you jump directly from logs to traces.

If you want a ready-to-run demo, see the [OpenTelemetry + Grafana example](https://github.com/sqlpage/SQLPage/tree/main/examples/telemetry), which this article is based on.

## What tracing gives you

With tracing enabled, one HTTP request becomes a tree of timed operations called spans. In a typical SQLPage app, you will see something like:

```text
[nginx] GET /todos
└─ [sqlpage] GET /todos
└─ [sqlpage] SQL website/todos.sql
├─ db.pool.acquire
└─ db.query
```

This is immediately useful for:

- Debugging slow pages by seeing exactly which query consumed the time
- Detecting connection pool pressure by measuring time spent in `db.pool.acquire`
- Following one request end-to-end from the reverse proxy to SQLPage to PostgreSQL

Tracing is especially helpful in SQLPage because one request often maps cleanly to one SQL file. That makes traces easy to interpret even when you are not used to application performance tooling.

## The easiest way to try it

The simplest way to explore tracing is to run the example shipped with SQLPage:

```bash
cd examples/telemetry
docker compose up --build
```

That stack starts:

- nginx as the reverse proxy
- SQLPage
- PostgreSQL
- an OpenTelemetry collector
- Grafana Tempo for traces
- Grafana Loki for logs
- Promtail for log shipping
- Grafana for visualization

Then:

1. Open [http://localhost](http://localhost) and use the sample todo application.
2. Open [http://localhost:3000](http://localhost:3000) to access Grafana.
3. Inspect recent traces and logs from the default dashboard.
4. Open a trace to see the full span waterfall for a single request.

This setup is useful both as a demo and as a reference architecture for production deployments.

## Enabling tracing in SQLPage

Tracing is built into SQLPage. There is no plugin to install and no SQLPage-specific tracing configuration file to write.

You only need to set standard OpenTelemetry environment variables before starting SQLPage:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_SERVICE_NAME="sqlpage"
sqlpage
```

The `OTEL_EXPORTER_OTLP_ENDPOINT` variable tells SQLPage where to send traces.
The `OTEL_SERVICE_NAME` variable controls how the service appears in your tracing backend.

If `OTEL_EXPORTER_OTLP_ENDPOINT` is not set, SQLPage falls back to normal logging and tracing stays disabled.

## What you will see in the trace

The most useful spans emitted by SQLPage are:

- The HTTP request span, with attributes such as method, path, status code, and user agent
- The SQL file execution span, showing which `.sql` file handled the request
- The `db.pool.acquire` span, showing time spent waiting for a database connection
- The `db.query` span, containing the SQL statement and database system

In practice, that means you can answer questions like:

- Is the page slow because the SQL itself is slow?
- Is the request queued because the connection pool is exhausted?
- Is the delay happening before SQLPage even receives the request?

This is much more actionable than a single request duration number.

## Logs and traces together

Tracing is even more useful when logs and traces are connected.

In the example stack, SQLPage writes structured logs to stdout, Promtail forwards them to Loki, and Grafana lets you move from a log line to the matching trace using the trace id. This makes it possible to start from an error log and immediately inspect the full request timeline.

That workflow is often the difference between guessing and knowing.

## PostgreSQL correlation

SQLPage also propagates trace context to PostgreSQL through the connection `application_name`.
This makes it possible to correlate live PostgreSQL activity or database logs with the trace that triggered it.

For example, inspecting `pg_stat_activity` can show which trace is attached to a running query:

```sql
SELECT application_name, query, state
FROM pg_stat_activity;
```

If you also include `%a` in PostgreSQL''s `log_line_prefix`, your database logs can contain the same trace context.

## A practical debugging example

Suppose users report that a page occasionally becomes slow under load.

With tracing enabled, you might see that:

- the HTTP span is long
- the SQL file execution span is also long
- the `db.query` span is short
- but `db.pool.acquire` takes several hundred milliseconds

That immediately tells you the database query itself is not the problem. The real issue is contention on the connection pool. You can then increase `max_database_pool_connections`, reduce concurrent load, or review long-running requests that keep connections busy.

Without tracing, this kind of diagnosis usually requires guesswork.

## Deployment options

The example uses Grafana Tempo and Loki, but SQLPage is not tied to a single backend. Because it emits standard OTLP traces, you can also send data to:

- Jaeger
- Grafana Cloud
- Datadog
- Honeycomb
- New Relic
- Axiom

In small setups, SQLPage can often send traces directly to the backend. In larger deployments, placing an OpenTelemetry collector in the middle is usually better because it centralizes routing, batching, and authentication.

## When to enable tracing

Tracing is particularly valuable when:

- you are running SQLPage behind a reverse proxy
- several SQL files participate in user-facing workflows
- you want to understand production latency, not just local development behavior
- you need a shared debugging tool for developers and operators

If your application is already important enough to monitor, it is important enough to trace.

## Conclusion

SQLPage already makes the application logic easy to inspect because it lives in SQL files. Tracing extends that visibility to runtime behavior.

By enabling OpenTelemetry and connecting SQLPage to Grafana, you can see not just that a request was slow, but why it was slow, where the time was spent, and which query or resource caused the delay.

For a complete working setup, start with the [OpenTelemetry + Grafana example](https://github.com/sqlpage/SQLPage/tree/main/examples/telemetry) and adapt it to your own deployment.
'
);
Loading