Add Ad Hoc Flamegraph Upload and Visualization

[artursarlo]

Add Ad Hoc Flamegraph Upload and Visualization

Summary

This PR introduces end-to-end support for ad hoc (on-demand) flamegraph profiles alongside the existing continuous profiling flow. When gprofiler agents run in ad hoc mode and emit a flamegraph_html field in the processed file payload, the system now:

1. Detects the profiling type (adhoc vs. continuous) in the indexer based on the continuous flag in file metadata.
2. Uploads the flamegraph HTML to a dedicated S3 path (products/{service}/stacks/flamegraph/), decoding base64-encoded payloads as needed.
3. Stores metadata about each adhoc flamegraph in a new PostgreSQL table (AdhocFlamegraphMetadata).
4. Exposes new REST endpoints for querying and retrieving adhoc flamegraph metadata and HTML content.
5. Displays a new "Adhoc Profiling" view in the frontend, showing a table of uploads with an inline iframe viewer.

Command tracking for heartbeats (received_command_ids / executed_command_ids) is also wired through the full stack as part of this change.

---

Changes by Component

Database / Migrations

scripts/setup/postgres/migrations/add_dynamic_profiling.up.sql
scripts/setup/postgres/gprofiler_recreate.sql

• Added AdhocFlamegraphMetadata table with columns: id, service_id, hostname, s3_key, start_time, end_time, perf_events, created_at.
• Added 3 indexes on AdhocFlamegraphMetadata: on service_id, on hostname, and a composite on (service_id, start_time, end_time).
• Added received_command_ids uuid[] NULL and executed_command_ids uuid[] NULL columns to HostHeartbeats.

scripts/setup/postgres/migrations/add_dynamic_profiling.down.sql

• Added DROP TABLE IF EXISTS AdhocFlamegraphMetadata CASCADE and the corresponding index drops.

---

Go Indexer (src/gprofiler_indexer/)

callstacks.go

• Added ProfilingTypeAdhoc = "adhoc" and ProfilingTypeContinuous = "continuous" constants.
• Extended RunArguments in the file metadata struct with Continuous bool, PerfEvents string, and PerfMode string fields.
• Extended FileInfo with an HTMLBlob and FlamegraphHTML field to receive the flamegraph HTML payload from the SQS message.
• Added upload logic: when FlamegraphHTML is non-empty, the indexer:
  • Replaces the hostname hash suffix in the filename with the actual hostname.
  • Determines profiling type from metadata.continuous.
  • Uploads the HTML to products/{service}/stacks/flamegraph/{filename}_{type}_flamegraph.html in S3 (supports both plain HTML and base64-encoded payloads).
  • If the type is adhoc, calls StoreAdhocFlamegraphMetadata() to persist the record in PostgreSQL.

postgres.go (new file)

• InitPostgres(connStr string) — opens a *sql.DB connection using lib/pq.
• StoreAdhocFlamegraphMetadata(serviceID, hostname, s3Key, startTime, endTime, perfEvents) — inserts a row into AdhocFlamegraphMetadata.
• ClosePostgres() — closes the connection pool.

args.go

• Added PostgresHost, PostgresPort, PostgresUser, PostgresPassword, PostgresDB CLI flag fields with defaults and GPROFILER_POSTGRES_* environment variable overrides.

main.go

• Builds a PostgreSQL DSN from CLI args and calls InitPostgres(connStr) at startup, with defer ClosePostgres().

go.mod / go.sum

• Added github.com/lib/pq v1.10.9 dependency.

---

Backend API — Python (src/gprofiler/backend/, src/gprofiler-dev/)

routers/metrics_routes.py

• Added FlamegraphFile Pydantic model (s3_key, hostname, start_time, end_time, perf_events, exists).
• Added FlamegraphContent Pydantic model (content string for HTML).
• Added GET /metrics/adhoc_flamegraphs endpoint — queries AdhocFlamegraphMetadata for a service/time range, checks S3 existence in parallel, and returns a list of FlamegraphFile objects (with exists=False for removed files).
• Added GET /metrics/adhoc_flamegraph_content endpoint — streams the flamegraph HTML from S3 and returns it as FlamegraphContent.
• Updated the heartbeat route to pass received_command_ids and executed_command_ids through to the DB upsert.

models/metrics_models.py

• Added received_command_ids: Optional[List[str]] and executed_command_ids: Optional[List[str]] to HeartbeatRequest.

gprofiler-dev/gprofiler_dev/postgres/db_manager.py

• Added get_adhoc_flamegraphs_metadata(service_id, start_time, end_time, hostname_filters) — queries AdhocFlamegraphMetadata with optional hostname filtering via RQL.
• Updated upsert_host_heartbeat to accept and persist received_command_ids and executed_command_ids.

gprofiler-dev/gprofiler_dev/s3_profile_dal.py

• Added check_keys_exist(keys: List[str]) -> Dict[str, bool] — checks S3 object existence in parallel using ThreadPoolExecutor with HEAD requests.
• Added list_files_with_prefix(prefix: str) -> List[str] — lists S3 keys under a given prefix.

backend/utils/filters_utils.py

• Added get_rql_all_eq_values(rql_filter, field_name) helper for extracting multi-value equality filters from RQL expressions, used for hostname filtering in the adhoc flamegraph query.

---

Frontend (src/gprofiler/frontend/src/)

api/urls.js

• Added GET_ADHOC_FLAMEGRAPHS and GET_ADHOC_FLAMEGRAPH_CONTENT URL constants.

utils/consts.js

• Added adhoc: 'adhoc' to the PROFILES_VIEWS enum.

components/profiles/views/adhoc/AdhocProfilingView.jsx (new file)

• Full React component (~260 lines) providing:
  • A collapsible table of adhoc flamegraph uploads (hostname, start/end time, perf events, S3 key).
  • A "Removed" chip for flamegraphs whose S3 object no longer exists.
  • An inline iframe viewer that loads the selected flamegraph HTML by fetching /adhoc_flamegraph_content.
  • Filtering by currently selected time range and hostname filters.

components/profiles/ProfilesWrapper.jsx

• Added shouldShowView = isFgDisplayed || viewMode === PROFILES_VIEWS.adhoc to bypass the flamegraph-empty guard when the adhoc view is active.

components/profiles/ProfilesViewsWrapper.jsx

• Added isAdhocView flag and shouldDisplayView = isFgDisplayed || isAdhocView to allow rendering without a loaded flamegraph.

components/profiles/ProfilesViews.jsx

• Added an early return that renders <AdhocProfilingView /> directly when viewMode === PROFILES_VIEWS.adhoc, before reaching the main flamegraph render path.
• Scoped the isFGEmpty side effect to skip navigation away from the adhoc view.

components/profiles/header/viewModeSwitch/ViewModeSwitch.jsx

• Added [PROFILES_VIEWS.adhoc]: ICONS_NAMES.FlameGraphView to the VIEW_TO_ICON_NAME map so the tab icon renders correctly.

components/profiles/header/viewModeSwitch/ViewModeTooltip.jsx

• Added adhoc entry to the viewModes object (fixes a TypeError: Cannot destructure property 'label' crash when the adhoc view mode was selected).

components/console/ProfilingStatusPage.jsx

• Added a buildProfileUrl(view) helper.
• Replaced the single "View Profile" link with two stacked links: "View Continuous Profile" (→ ?view=flamegraph) and "View Adhoc Profile" (→ ?view=adhoc).

---

Deploy

deploy/docker-compose.yml

• Added PostgreSQL connection environment variables (GPROFILER_POSTGRES_HOST, GPROFILER_POSTGRES_PORT, GPROFILER_POSTGRES_USER, GPROFILER_POSTGRES_PASSWORD, GPROFILER_POSTGRES_DB) to the indexer service definition.

---

Testing Checklist

• Indexer correctly routes a message with flamegraph_html set and continuous: false to the adhoc path and stores metadata in PostgreSQL.
• Indexer correctly routes a message with continuous: true to the S3 upload-only path (no PostgreSQL insert).
• GET /metrics/adhoc_flamegraphs returns the correct list of files with exists status.
• GET /metrics/adhoc_flamegraph_content returns valid HTML for an existing S3 key.
• Frontend "Adhoc Profiling" view tab renders, loads the table, and displays the iframe on row selection.
• No TypeError on the view mode tooltip when switching to the adhoc tab.
• Heartbeat route correctly persists received_command_ids and executed_command_ids.
• add_dynamic_profiling.up.sql migration applies cleanly; down.sql rolls back without errors.

[prashantbytesyntax]

once merged, @marc-queiroz can you help test the feature in dynamic profiling branch