Skip to content

Dynamic profiling#92

Open
ashokbytebytego wants to merge 178 commits intomasterfrom
dynamic_profiling
Open

Dynamic profiling#92
ashokbytebytego wants to merge 178 commits intomasterfrom
dynamic_profiling

Conversation

@ashokbytebytego
Copy link

@ashokbytebytego ashokbytebytego commented Feb 11, 2026

Add Dynamic Profiling Data Models and PostgreSQL Schemas

Summary

This PR contributes the foundational data model infrastructure for the dynamic profiling feature, including PostgreSQL database schemas and Python API models.

What's Included

Database Schema (scripts/setup/postgres/)

  • ProfilingRequests: Track start/stop profiling requests
  • ProfilingCommands: Store commands to be executed by agents
  • HostHeartbeats: Monitor agent health and status
  • ProfilingExecutions: Record profiling execution history
  • Supporting tables for namespace, service, container, and process mappings

API Models (src/gprofiler/backend/models/)

  • Pydantic models for request/response validation
  • Type-safe data structures for profiling operations
  • Support for host-level and process-level profiling

Documentation (docs/)

  • Comprehensive guide for dynamic profiling architecture
  • Database schema documentation with ERD
  • Setup and testing instructions

Tests (src/tests/)

  • Unit tests for profiling host status functionality
  • Test scripts for model validation

Benefits

  • ✅ Clean separation of dynamic profiling data from existing metrics
  • ✅ Support for both host-level and process-level profiling
  • ✅ Extensible schema for future profiling capabilities
  • ✅ Well-documented for easy onboarding

ashokbytebytego and others added 30 commits July 2, 2025 17:42
Adjust field name in hostheartbeat queries
Set debug port in docker
prashantbytesyntax and others added 18 commits November 16, 2025 09:06
Max Profiling Frequency and Profiler Selection in Dynamic Profiling
- Add PostgreSQL schemas for dynamic profiling tables
- Add Python Pydantic models for API requests/responses
- Add 4 REST API endpoints for dynamic profiling:
  * POST /api/metrics/profile_request
  * POST /api/metrics/heartbeat
  * POST /api/metrics/command_completion
  * GET /api/metrics/profiling/host_status
- Add comprehensive documentation and setup guides
- Add unit tests for profiling host status functionality
- Includes Slack notification support for profiling events

This contribution provides the complete dynamic profiling feature for
Intel's gprofiler-performance-studio, enabling start/stop profiling
commands, heartbeat monitoring, and host status tracking.

Co-authored-by: Pinterest Engineering <engineering@pinterest.com>
Remove files that are specific to Pinterest's infrastructure:
- .github/workflows/sync-upstream.yml (Pinterest CI/CD)
- DEPLOYMENT_CHECKLIST.md (Pinterest deployment docs)
- deploy/localstack_init/01_init_s3_sqs.sh (Pinterest testing)
- docs/METRICS_PUBLISHER_INDEXER_DOCUMENTATION.md (Pinterest metrics)
- docs/SLI_METRICS_IMPLEMENTATION.md (Pinterest SLI metrics)
- src/gprofiler/backend/utils/metrics_publisher.py (Pinterest code)
- src/gprofiler_indexer/metrics_publisher.go (Pinterest code)

These files are not part of the dynamic profiling contribution to Intel.
- Remove MetricsPublisher import from main.py
- Remove startup/shutdown event handlers for MetricsPublisher
- Remove all send_sli_metric() calls from profiles_routes.py

These are Pinterest-specific metrics tracking not needed for Intel PR.
- Remove MetricsPublisher configuration from args.go
- Remove metricsPublisher initialization from main.go
- Remove all SendSLIMetric() calls from queue.go and worker.go
- Remove related comments

These are Pinterest-specific metrics tracking not needed for Intel PR.
- Removed metricsPublisher initialization and cleanup calls from main.go
- This completes the cleanup of Pinterest-specific metrics code
- Indexer now builds successfully in Docker
- Fixed multiple statements on same line (syntax errors from previous cleanup)
- Split logger.error() and raise HTTPException() onto separate lines
- Fixed indentation after except blocks
- All services now build and run successfully
- Remove hardcoded AWS config and LocalStack from deploy/.env
- Remove LocalStack service from deploy/docker-compose.yml
- Remove exposed PostgreSQL port from docker-compose.yml
- Add database migrations (up/down) for dynamic profiling
…ntil agent implementation catches up with this feature
- Fix import sorting and formatting in ProfilingStatusPage.jsx
- Fix import sorting and formatting in ProfilingTopPanel.jsx
- Fix formatting in ProfilingHeader.jsx
- Fix long line formatting in iconsData.js
listen 80;
server_name your_domain.com www.your_domain.com; # Replace with your domain

location /api/v2/profiles {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only to help with local set up

Copy link
Contributor

@prashantbytesyntax prashantbytesyntax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved to unblock for upcoming changes to wall time, perf metrics based flamegraph and command completion api fallback to heartbeat request

@mlim19
Copy link
Contributor

mlim19 commented Mar 9, 2026

@prashantbytesyntax, @ashokbytebytego, @artursarlo, I don't see adhoc related view option in the following files. Is that the reason I am not able to see the adhoc view icon?

src/gprofiler/frontend/src/utils/consts.js (lines 85-90)
src/gprofiler/frontend/src/components/profiles/header/viewModeSwitch/ViewModeSwitch.jsx (lines 37-42)
src/gprofiler/frontend/src/components/common/icon/iconsData.js

And do I need any db schema update for adhoc data view?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants