diff --git a/README.md b/README.md index 4cf97ae..9e54fb4 100644 --- a/README.md +++ b/README.md @@ -11,23 +11,50 @@ specific functionality. The main components include: - **Data Ingestion Service**: Responsible for collecting data from various sensor endpoints. - **Data Processing Service**: Processes the ingested data, applying any necessary transformations or aggregations. -- **API Gateway**: Exposes a REST API for clients to access the processed data. +- **REST API**: Exposes a REST API for clients to access the processed data. +- **Application**: Sample frontend application to visualize and interact with the data. + +A technical overview of the architecture is shown below: + +![Architecture Diagram](./docs/architecture/architecture.png) + +_Components in grey are either optional or not implemented yet._ + +Legend: + +- Purple blocks represent external services / devices. +- Orange blocks represent services that are part of the system. +- Blue blocks represent data storage and visualization components. +- Green blocks represent user-facing applications. + +> [!NOTE] Kubernetes Autoscaling Kubernetes Autoscaling is an option, which can be implemented in +> the future if the application scales up. This would allow the services to automatically adjust +> their number of instances based on the current load and demand. ## Folder Structure ```text └── 📁ajdovscina-cloud-software └── 📁alertmanager + └── 📁application + └── 📁public + ├── favicon.ico + ├── index.html + └── 📁src + └── 📁components + └── 📁services + └── 📁types + ├── Makefile └── 📁common └── 📁logging └── 📁dockerfiles └── 📁docs └── 📁architecture - ├── ajdovscina-architecture.drawio └── 📁images └── 📁grafana └── 📁dashboards └── 📁ingest_service + └── 📁common └── 📁models └── 📁util ├── Makefile @@ -36,11 +63,13 @@ specific functionality. The main components include: └── 📁mock └── 📁ingest_service └── 📁processor_service + └── 📁common └── 📁src └── 📁util ├── Makefile └── 📁prometheus └── 📁rest_api + └── 📁common └── 📁models └── 📁src └── 📁util @@ -81,7 +110,61 @@ And to stop the production system: make prod-down ``` -> [!IMPORTANT] Each of the services can be deployed independently using their respective makefiles. +> [!IMPORTANT] Service local deployment Each of the services can be deployed independently locally +> using their respective makefiles. See the documentation in each service's directory for more +> details. This should be used in conjunction with the `make dev-up` command to start the +> dependencies. + +## Building Blocks + +The application is composed of several building blocks, each with its own purpose and functionality. +These blocks are defined in the `docker-compose` files located in the root directory. + +These are subject to change as the system evolves, but currently include: + +- **PostgreSQL with TimescaleDB**: The primary database for storing sensor data and application + metadata. +- **Kafka**: A message broker used for decoupling the ingestion and processing services. +- **Prometheus**: A monitoring and alerting toolkit used for collecting and querying metrics. +- **Grafana**: A visualization tool used for creating dashboards and visualizing metrics collected + by Prometheus. +- **Alertmanager**: A component of the Prometheus ecosystem used for handling alerts sent by + Prometheus. +- **NodeRED**: A flow-based development tool for visual programming, used for wiring together + hardware devices, APIs, and online services. +- **Ingest Service**: A FastAPI service responsible for receiving and validating incoming sensor + data. +- **Processor Service**: A service that processes the ingested data and stores it in the database. +- **Frontend Application**: A React-based application for visualizing and interacting with the + sensor data. + +The [dockerfiles](./dockerfiles) directory contains Dockerfiles for custom services: + +- `Dockerfile.ingest_service`: Dockerfile for the Ingest Service. +- `Dockerfile.processor_service`: Dockerfile for the Processor Service. +- `Dockerfile.rest_api`: Dockerfile for the REST API. +- `Dockerfile.application`: Dockerfile for the Frontend Application. + +## Service Accessibility + +The services are accessible via the following ports when deployed: + +| Service | Local Port | External Port Production | Description | +| -------------------- | ---------- | ------------------------ | ------------------------------------------ | +| PostgreSQL | 5432 | 13328 | Database service | +| Prometheus | 9090 | 19290 | Monitoring and alerting toolkit | +| Grafana | 3000 | 13201 | Visualization tool | +| Alertmanager | 9093 | 19193 | Alert management | +| NodeRED | 1880 | 11880 | Flow-based development tool | +| Ingest Service | 5000 | 40005 | Data ingestion endpoint | +| REST API | 5005 | 41005 | REST API for accessing sensor data | +| Frontend Application | 3001 | 8004 | User interface for visualizing sensor data | + +External ports are used to access the services from outside the Docker network, while local ports +are used for communication between services within the Docker network. + +If the services are run using their respective makefiles, the ports may differ. See the Makefile in +each service's directory for more details. ### Configuration and Constants @@ -110,3 +193,46 @@ DB_CONFIG = { API_KEY = "your-secret-api-key" # Replace with your actual key ``` + +Detailed descriptions of each constant: + +- `LOG_LEVEL`: Sets the logging level for the application. +- `DB_CONFIG`: A dictionary containing the database connection parameters: + - `dbname`: The name of the PostgreSQL database. Set to `ajdovscina` by default; This is defined + in the `docker-compose` files - `POSTGRES_DB` variable. + - `user`: The username used to connect to the database. Set to `ajdovscina-geospatial-user` by + default; This is defined in the `docker-compose` files - `POSTGRES_USER` variable. + - `password`: The password for the database user. Set to `password` by default; This is defined in + the `docker-compose` files - `POSTGRES_PASSWORD` variable. + - `host`: The hostname or IP address of the database server. Set to `localhost` for local + development or `geospatial-data` when using Docker Compose - that's the service name defined in + the `docker-compose` files. + - `port`: The port number on which the database server is listening. Set to `5432` for local + deployment or `13328` for Docker Compose - this is defined in the `docker-compose` files - + `ports` mapping in the `geospatial-data` service. + +#### ingest_service/constants.py + +The constants found in the ingest_service/constants.py file are specific to the Ingest Service. + +```python +import os + +LOCALHOST = os.environ.get("LOCALHOST") == "true" + +KAFKA_HOST = "localhost:9992" if LOCALHOST else "kafka:9993" # Kafka broker to connect to and produce messages to +# kafka:9993 is the service name defined in the docker-compose files +``` + +#### processor_service/constants.py + +The constants found in the processor_service/constants.py file are specific to the Processor +Service. + +```python +import os + +LOCALHOST = os.environ.get("LOCALHOST") == "true" +KAFKA_HOST = "localhost:9992" if LOCALHOST else "kafka:9993" # Kafka broker to connect to and consume messages from +# kafka:9993 is the service name defined in the docker-compose files +``` diff --git a/application/README.md b/application/README.md index 27f4f11..3daad83 100644 --- a/application/README.md +++ b/application/README.md @@ -1,55 +1,26 @@ -# Getting Started with Create React App +# Sample Frontend Application This project was bootstrapped with [Create React App](https://github.com/facebook/create-react-app). -## Available Scripts +This simple frontend application allows users to visualize and interact with the sensor data +ingested by the system. It fetches data from the REST API and displays it in a user-friendly manner. -In the project directory, you can run: +## Configuration -### `npm start` +A `.env` file is used to configure the application. The `.env` file should contain the following +variables: -Runs the app in the development mode.\ -Open [http://localhost:3000](http://localhost:3000) to view it in the browser. +```env +PORT=3001 -The page will reload if you make edits.\ -You will also see any lint errors in the console. +REACT_APP_API_TOKEN=your-secret-api-key +REACT_APP_API_BASE_URL=http://localhost:51005 +``` -### `npm test` +When deploying the application, make sure to set `REACT_APP_API_BASE_URL` to the URL where the REST +API is hosted, and `REACT_APP_API_TOKEN` to a secure value that is set by the REST API. -Launches the test runner in the interactive watch mode.\ -See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) -for more information. +## Running the Application -### `npm run build` - -Builds the app for production to the `build` folder.\ -It correctly bundles React in production mode and optimizes the build for the best performance. - -The build is minified and the filenames include the hashes.\ -Your app is ready to be deployed! - -See the section about [deployment](https://facebook.github.io/create-react-app/docs/deployment) for -more information. - -### `npm run eject` - -**Note: this is a one-way operation. Once you `eject`, you can’t go back!** - -If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. -This command will remove the single build dependency from your project. - -Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, -ESLint, etc) right into your project so you have full control over them. All of the commands except -`eject` will still work, but they will point to the copied scripts so you can tweak them. At this -point you’re on your own. - -You don’t have to ever use `eject`. The curated feature set is suitable for small and middle -deployments, and you shouldn’t feel obligated to use this feature. However we understand that this -tool wouldn’t be useful if you couldn’t customize it when you are ready for it. - -## Learn More - -You can learn more in the -[Create React App documentation](https://facebook.github.io/create-react-app/docs/getting-started). - -To learn React, check out the [React documentation](https://reactjs.org/). +To run the application locally, run `make run-local-dev` from this directory. This will run the +`npm run start` command and launch the application in development mode. diff --git a/application/src/components/Measurements.tsx b/application/src/components/Measurements.tsx index 4539880..b6f9f8b 100644 --- a/application/src/components/Measurements.tsx +++ b/application/src/components/Measurements.tsx @@ -100,7 +100,9 @@ const Measurements: React.FC = ({ selectedSensor: propSelecte return (
{key}: - {value.toFixed(2)} + + {typeof value === 'number' ? value.toFixed(2) : value} +
); }; diff --git a/common/README.md b/common/README.md new file mode 100644 index 0000000..e0c1ca5 --- /dev/null +++ b/common/README.md @@ -0,0 +1,12 @@ +# Common Utilities + +The common directory contains shared utilities and modules used across different services in the +project. This includes database connection management, logging setup, and other helper functions. + +## Configuration + +The [constants.py](./constants.py) file contains configuration constants for the services. These +should be configured based on the deployment environment. + +When deploying to the cloud, make sure you change the API key and database credentials to secure +values. diff --git a/common/constants.py b/common/constants.py index fe0f2b0..4518a47 100644 --- a/common/constants.py +++ b/common/constants.py @@ -9,7 +9,7 @@ DB_CONFIG = { "dbname": "ajdovscina", "user": "ajdovscina-geospatial-user", - "password": "password", + "password": "password", # In production, use a secure password and manage it safely "host": "localhost" if localhost else "geospatial-data", "port": 13328 if localhost else 5432, } diff --git a/docs/architecture/ajdovscina-architecture.png b/docs/architecture/ajdovscina-architecture.png deleted file mode 100644 index 45c7e5f..0000000 Binary files a/docs/architecture/ajdovscina-architecture.png and /dev/null differ diff --git a/docs/architecture/ajdovscina-architecture.drawio b/docs/architecture/architecture.drawio similarity index 79% rename from docs/architecture/ajdovscina-architecture.drawio rename to docs/architecture/architecture.drawio index bee8f72..fc009da 100644 --- a/docs/architecture/ajdovscina-architecture.drawio +++ b/docs/architecture/architecture.drawio @@ -1,76 +1,76 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - + - + - + - + @@ -85,25 +85,25 @@ - + - + - + - + - - + + - + @@ -111,9 +111,9 @@ - - - + + + @@ -124,45 +124,45 @@ - + - - + + - - + + - - + + - + - - + + - - + + - - + + - - + + @@ -170,8 +170,8 @@ - - + + @@ -195,106 +195,106 @@ - - + + - - + + - - + + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - - + + - + - + - + - + - + - - + + - + - + - + - - + + @@ -305,8 +305,8 @@ - - + + diff --git a/docs/architecture/architecture.png b/docs/architecture/architecture.png new file mode 100644 index 0000000..77e9188 Binary files /dev/null and b/docs/architecture/architecture.png differ diff --git a/docs/images/ttn_integration.png b/docs/images/ttn_integration.png new file mode 100644 index 0000000..86e248f Binary files /dev/null and b/docs/images/ttn_integration.png differ diff --git a/grafana/dashboards/postgresql-exporter.json b/grafana/dashboards/postgresql-exporter.json index c6dc85d..d195483 100644 --- a/grafana/dashboards/postgresql-exporter.json +++ b/grafana/dashboards/postgresql-exporter.json @@ -1,14 +1,15 @@ { "__inputs": [ { - "name": "DS_OBSERVE-PROMETHEUS02", - "label": "prometheus", + "name": "DS_PROMETHEUS", + "label": "Prometheus", "description": "", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ], + "__elements": {}, "__requires": [ { "type": "panel", @@ -20,12 +21,12 @@ "type": "grafana", "id": "grafana", "name": "Grafana", - "version": "7.0.3" + "version": "10.2.2" }, { "type": "panel", "id": "graph", - "name": "Graph", + "name": "Graph (old)", "version": "" }, { @@ -36,8 +37,8 @@ }, { "type": "panel", - "id": "singlestat", - "name": "Singlestat", + "id": "stat", + "name": "Stat", "version": "" } ], @@ -45,7 +46,10 @@ "list": [ { "builtIn": 1, - "datasource": "-- Grafana --", + "datasource": { + "type": "datasource", + "uid": "grafana" + }, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", @@ -54,16 +58,21 @@ } ] }, + "description": "postgres exporter Dashboard for display important postgreSQL metric", "editable": true, + "fiscalYearStartMonth": 0, "gnetId": 12485, "graphTooltip": 0, "id": null, - "iteration": 1592399036513, "links": [], + "liveNow": false, "panels": [ { "collapsed": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "d9592e16-ddb7-4e6e-a53f-b31701d78c14" + }, "gridPos": { "h": 1, "w": 24, @@ -72,34 +81,57 @@ }, "id": 18, "panels": [], + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "d9592e16-ddb7-4e6e-a53f-b31701d78c14" + }, + "refId": "A" + } + ], "title": "Global Statistics", "type": "row" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: server_version_num", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -107,89 +139,78 @@ "y": 1 }, "id": 11, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "tableColumn": "", + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "max(pg_settings_server_version_num)", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "PostgreSQL Version", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Max Replication lag behind master in seconds\n\nOnly available on a standby system.\n\nSource: pg_last_xact_replay_timestamp\n\nUse: pg_stat_replication for Details.", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" }, "overrides": [] }, - "format": "s", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -197,90 +218,79 @@ "y": 1 }, "id": 84, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "max" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "max(pg_replication_lag{instance=\"$Instance\"})", "interval": "", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Max Replication Lag", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "max" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Clients executing Statements.\n\nSource: pg_stat_activity", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -288,81 +298,59 @@ "y": 1 }, "id": 23, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum(pg_stat_activity_count{state=\"active\",instance=\"$Instance\"})", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Active clients", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Shared buffer hits vs reads from disc", "fieldConfig": { "defaults": { - "custom": {}, "decimals": 2, "mappings": [ { - "id": 0, - "op": "=", - "text": "N/A", - "type": 1, - "value": "null" + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" } ], - "nullValueMode": "connected", + "max": 100, + "min": 0, "thresholds": { "mode": "absolute", "steps": [ @@ -393,6 +381,8 @@ "id": 16, "links": [], "options": { + "minVizHeight": 75, + "minVizWidth": 75, "orientation": "horizontal", "reduceOptions": { "calcs": [ @@ -404,36 +394,42 @@ "showThresholdLabels": false, "showThresholdMarkers": true }, - "pluginVersion": "7.0.3", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum(pg_stat_database_blks_hit{instance=~\"$Instance\"})/(sum(pg_stat_database_blks_hit{instance=~\"$Instance\"})+sum(pg_stat_database_blks_read{instance=~\"$Instance\"}))*100", "refId": "A" } ], - "timeFrom": null, - "timeShift": null, "title": "Shared Buffer Hits", "type": "gauge" }, { - "cacheTimeout": null, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Percentage of max_connections used", "fieldConfig": { "defaults": { - "custom": {}, "decimals": 0, "mappings": [ { - "id": 0, - "op": "=", - "text": "N/A", - "type": 1, - "value": "null" + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" } ], - "nullValueMode": "connected", + "max": 1, + "min": 0, "thresholds": { "mode": "absolute", "steps": [ @@ -464,6 +460,8 @@ "id": 9, "links": [], "options": { + "minVizHeight": 75, + "minVizWidth": 75, "orientation": "horizontal", "reduceOptions": { "calcs": [ @@ -475,36 +473,42 @@ "showThresholdLabels": false, "showThresholdMarkers": true }, - "pluginVersion": "7.0.3", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum(pg_stat_database_numbackends)/max(pg_settings_max_connections)", "refId": "A" } ], - "timeFrom": null, - "timeShift": null, "title": "Connections used", "type": "gauge" }, { - "cacheTimeout": null, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Transaction committed vs rollbacked", "fieldConfig": { "defaults": { - "custom": {}, "decimals": 2, "mappings": [ { - "id": 0, - "op": "=", - "text": "N/A", - "type": 1, - "value": "null" + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" } ], - "nullValueMode": "connected", + "max": 1, + "min": 0, "thresholds": { "mode": "absolute", "steps": [ @@ -535,6 +539,8 @@ "id": 15, "links": [], "options": { + "minVizHeight": 75, + "minVizWidth": 75, "orientation": "horizontal", "reduceOptions": { "calcs": [ @@ -546,44 +552,60 @@ "showThresholdLabels": false, "showThresholdMarkers": true }, - "pluginVersion": "7.0.3", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum(pg_stat_database_xact_commit{instance=\"$Instance\"})/(sum(pg_stat_database_xact_commit{instance=\"$Instance\"}) + sum(pg_stat_database_xact_rollback{instance=\"$Instance\"}))", "refId": "A" } ], - "timeFrom": null, - "timeShift": null, "title": "Commit Ratio", "type": "gauge" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", - "decimals": 1, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_postmaster_start_time()", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "mode": "thresholds" + }, + "decimals": 1, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" }, "overrides": [] }, - "format": "s", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -591,89 +613,78 @@ "y": 4 }, "id": 57, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "tableColumn": "{instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}", + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^{instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}$/", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "time()-(pg_postmaster_start_time_seconds{instance=\"$Instance\"})", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "PostgreSQL Uptime", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Transactions committed + roolback per minute\n\nSource: pg_stat_database,xact_commit + xact_rollback", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -683,88 +694,78 @@ "id": 14, "interval": "", "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_xact_commit{instance=\"$Instance\"}[$Interval])))+sum((rate(pg_stat_database_xact_rollback{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Transaction rate", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Statements executed per Minute.\n\nSource: pg_stat_statements.calls", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -772,91 +773,80 @@ "y": 4 }, "id": 93, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_statements_calls{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Query rate", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", - "decimals": 2, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Size of all databases in $Instance.\n\nSource: pg_database_size()", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "decimals": 2, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" }, "overrides": [] }, - "format": "bytes", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -864,88 +854,77 @@ "y": 7 }, "id": 37, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum(pg_database_size{instance=\"$Instance\"})", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Total database size", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_stat_statements.total_time / pg_stat_statements.calls", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "s" }, "overrides": [] }, - "format": "s", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -953,91 +932,79 @@ "y": 7 }, "id": 102, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((delta(pg_stat_statements_total_time_seconds{instance=\"$Instance\"}[$Interval])))/sum((delta(pg_stat_statements_calls{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Average query runtime", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", - "decimals": 0, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Setting: shared_buffers in PostgreSQL.conf", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "mode": "thresholds" + }, + "decimals": 0, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" }, "overrides": [] }, - "format": "bytes", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -1045,88 +1012,76 @@ "y": 7 }, "id": 66, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "tableColumn": "pg_settings_shared_buffers_bytes{instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}", + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^pg_settings_shared_buffers_bytes{instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}$/", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "pg_settings_shared_buffers_bytes{instance=\"$Instance\"}", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Shared Buffers", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Setting: max_connections in postgreSQL.conf", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -1134,74 +1089,51 @@ "y": 7 }, "id": 75, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": false, - "ymax": null, - "ymin": null - }, - "tableColumn": "pg_settings_max_connections{instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}", + "options": { + "colorMode": "none", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^pg_settings_max_connections{instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}$/", + "values": false + }, + "textMode": "auto", + "wideLayout": true + }, + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "pg_settings_max_connections{instance=\"$Instance\"}", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Max Connections", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { "aliasColors": {}, "bars": true, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "decimals": 0, "description": "View: pg_stat_activity", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1240,9 +1172,10 @@ ], "nullPointMode": "null as zero", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1252,15 +1185,17 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (state) (pg_stat_activity_count{instance=\"$Instance\"})", "legendFormat": "{{state}}", "refId": "A" } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Connections by state (stacked)", "tooltip": { "shared": true, @@ -1269,9 +1204,7 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, @@ -1279,25 +1212,18 @@ { "decimals": 0, "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true }, { "decimals": 0, "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1305,12 +1231,15 @@ "bars": true, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "decimals": 0, "description": "View: pg_stat_activity", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1349,9 +1278,10 @@ ], "nullPointMode": "null as zero", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1361,15 +1291,17 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (datname) (pg_stat_activity_count{instance=\"$Instance\"})", "legendFormat": "{{datname}}", "refId": "A" } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Connections by database (stacked)", "tooltip": { "shared": true, @@ -1378,9 +1310,7 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, @@ -1388,25 +1318,18 @@ { "decimals": 0, "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true }, { "decimals": 0, "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1414,12 +1337,15 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "decimals": 2, "description": "1 Minute rate of transactions committed or rollback.", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1451,9 +1377,10 @@ "linewidth": 1, "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1463,12 +1390,20 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum ((rate(pg_stat_database_xact_commit[$Interval])))", "interval": "", "legendFormat": "committed", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum ((rate(pg_stat_database_xact_rollback[$Interval])))", "hide": false, "interval": "", @@ -1477,9 +1412,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Transactions", "tooltip": { "shared": true, @@ -1488,33 +1421,24 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1522,11 +1446,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_stat_database", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1555,9 +1482,10 @@ "linewidth": 1, "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1567,18 +1495,30 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_tup_inserted{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "Inserts", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_tup_updated{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "Updates", "refId": "B" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_tup_deleted{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "Deletes", @@ -1586,9 +1526,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Tuples inserts/updates/deletes", "tooltip": { "shared": true, @@ -1597,33 +1535,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1631,11 +1562,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "* blk_read_time: Time spent reading data file blocks by backends in this database, in milliseconds\n* blk_write_time: Time spent writing data file blocks by backends in this database, in milliseconds\n\ntrack_io_timings needs to be activated", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1663,9 +1597,10 @@ "linewidth": 1, "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1675,12 +1610,20 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum ((rate(pg_stat_database_blk_read_time{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "blk_read_time", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum ((rate(pg_stat_database_blk_write_time{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "blk_read_time", @@ -1688,9 +1631,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "I/O Read/Write time", "tooltip": { "shared": true, @@ -1699,33 +1640,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "ms", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1733,11 +1667,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_stat_database\n\n* tup_fetched: rows needed to satisfy queries\n* tup_returned: rows read/scanned", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1766,9 +1703,10 @@ "linewidth": 1, "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1778,12 +1716,20 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_tup_fetched{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "Fetched", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_tup_returned{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "Returned", @@ -1791,9 +1737,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Tuples fetched/returned", "tooltip": { "shared": true, @@ -1802,33 +1746,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": false } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1836,11 +1773,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_locks", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1874,9 +1814,10 @@ ], "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1886,15 +1827,17 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (mode) (pg_locks_count{instance=\"$Instance\"})", "legendFormat": "{{mode}}", "refId": "A" } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Locks by state", "tooltip": { "shared": true, @@ -1903,33 +1846,24 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -1937,11 +1871,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Should be 0 \n\nSource: pg_stat_database\n\nWith log_lock_waits turned on, deadlocks will be logged to the PostgreSQL Logfiles.", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -1977,9 +1914,10 @@ ], "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -1989,6 +1927,10 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (datname) ((rate(pg_stat_database_deadlocks{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "{{datname}}", @@ -1996,9 +1938,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Deadlocks by database", "tooltip": { "shared": true, @@ -2007,33 +1947,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -2041,11 +1974,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Should be 0. If temporary files are created, it can indicate insufficient work_mem. With log_temp_files the creation of temporary files are logged to the PostgreSQL Logfiles.", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -2081,9 +2017,10 @@ ], "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -2093,6 +2030,10 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (datname) ((rate(pg_stat_database_temp_files{instance=\"$Instance\"}[$Interval])))", "interval": "", "legendFormat": "{{datname}}", @@ -2100,9 +2041,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Temporary files by database", "tooltip": { "shared": true, @@ -2111,33 +2050,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -2145,11 +2077,14 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Lag behind master in seconds.\n\nOnly available on a standby System.", "fieldConfig": { "defaults": { - "custom": {} + "links": [] }, "overrides": [] }, @@ -2179,9 +2114,10 @@ "links": [], "nullPointMode": "null", "options": { - "dataLinks": [] + "alertThreshold": true }, "percentage": false, + "pluginVersion": "10.2.2", "pointradius": 2, "points": false, "renderer": "flot", @@ -2191,6 +2127,10 @@ "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "max(pg_replication_lag{instance=\"$Instance\"})", "instant": false, "intervalFactor": 1, @@ -2199,9 +2139,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Replication lag ", "tooltip": { "shared": true, @@ -2210,38 +2148,34 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "s", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { "collapsed": true, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "d9592e16-ddb7-4e6e-a53f-b31701d78c14" + }, "gridPos": { "h": 1, "w": 24, @@ -2251,29 +2185,44 @@ "id": 20, "panels": [ { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -2281,95 +2230,77 @@ "y": 2 }, "id": 46, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^pg_stat_activity_count{datname=\"grafana\", instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\", state=\"active\"}$/", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "pg_stat_activity_count{datname=\"grafana\", instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\", state=\"active\"}", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "pg_stat_activity_count{state=\"active\",instance=\"$Instance\",datname=~\"$Database\"}", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Active clients", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", - "decimals": 2, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "decimals": 2, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "bytes" }, "overrides": [] }, - "format": "bytes", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -2377,73 +2308,42 @@ "y": 2 }, "id": 43, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "/^pg_database_size{datname=\"grafana\", instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}$/", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "pg_database_size{datname=\"grafana\", instance=\"db01vp-nbg6a.obs.noris.net:9187\", job=\"postgres-exporter\", server=\"213.95.132.73:5432\"}", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "pg_database_size{instance=\"$Instance\",datname=\"$Database\"}", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Database size", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { - "cacheTimeout": null, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Shared buffer hits vs reads from disc", "fieldConfig": { "defaults": { @@ -2451,11 +2351,13 @@ "decimals": 2, "mappings": [ { - "id": 0, - "op": "=", - "text": "N/A", - "type": 1, - "value": "null" + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" } ], "nullValueMode": "connected", @@ -2463,8 +2365,7 @@ "mode": "absolute", "steps": [ { - "color": "semi-dark-red", - "value": null + "color": "semi-dark-red" }, { "color": "semi-dark-yellow", @@ -2501,27 +2402,24 @@ "showThresholdMarkers": true }, "pluginVersion": "7.0.3", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "pg_stat_database_blks_hit{instance=~\"$Instance\",datname=~\"$Database\"}/(pg_stat_database_blks_hit{instance=~\"$Instance\",datname=~\"$Database\"}+pg_stat_database_blks_read{instance=~\"$Instance\",datname=~\"$Database\"})", "refId": "A" } ], - "timeFrom": null, - "timeShift": null, "title": "Shared Buffer Hits", "type": "gauge" }, { - "cacheTimeout": null, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Transaction committed vs rollbacked", "fieldConfig": { "defaults": { @@ -2529,11 +2427,13 @@ "decimals": 2, "mappings": [ { - "id": 0, - "op": "=", - "text": "N/A", - "type": 1, - "value": "null" + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" } ], "nullValueMode": "connected", @@ -2541,8 +2441,7 @@ "mode": "absolute", "steps": [ { - "color": "semi-dark-red", - "value": null + "color": "semi-dark-red" }, { "color": "#EAB839", @@ -2579,49 +2478,59 @@ "showThresholdMarkers": true }, "pluginVersion": "7.0.3", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "pg_stat_database_xact_commit{instance=\"$Instance\",datname=~\"$Database\"}/(pg_stat_database_xact_commit{instance=\"$Instance\",datname=~\"$Database\"} + pg_stat_database_xact_rollback{instance=\"$Instance\",datname=~\"$Database\"})", "refId": "A" } ], - "timeFrom": null, - "timeShift": null, "title": "Commit Ratio", "type": "gauge" }, { - "cacheTimeout": null, - "colorBackground": false, - "colorValue": false, - "colors": [ - "#299c46", - "rgba(237, 129, 40, 0.89)", - "#d44a3a" - ], - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Transactions committed + roolback per minute\n\nSource: pg_stat_database,xact_commit + xact_rollback", "fieldConfig": { "defaults": { - "custom": {} + "color": { + "fixedColor": "rgb(31, 120, 193)", + "mode": "fixed" + }, + "mappings": [ + { + "options": { + "match": "null", + "result": { + "text": "N/A" + } + }, + "type": "special" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 80 + } + ] + }, + "unit": "none" }, "overrides": [] }, - "format": "none", - "gauge": { - "maxValue": 100, - "minValue": 0, - "show": false, - "thresholdLabels": false, - "thresholdMarkers": true - }, "gridPos": { "h": 3, "w": 4, @@ -2629,78 +2538,48 @@ "y": 5 }, "id": 122, - "interval": null, "links": [], - "mappingType": 1, - "mappingTypes": [ - { - "name": "value to text", - "value": 1 - }, - { - "name": "range to text", - "value": 2 - } - ], "maxDataPoints": 100, - "nullPointMode": "connected", - "nullText": null, - "postfix": "", - "postfixFontSize": "50%", - "prefix": "", - "prefixFontSize": "50%", - "rangeMaps": [ - { - "from": "null", - "text": "N/A", - "to": "null" - } - ], - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, - "sparkline": { - "fillColor": "rgba(31, 118, 189, 0.18)", - "full": false, - "lineColor": "rgb(31, 120, 193)", - "show": true, - "ymax": null, - "ymin": null + "options": { + "colorMode": "none", + "graphMode": "area", + "justifyMode": "auto", + "orientation": "horizontal", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "auto", + "wideLayout": true }, - "tableColumn": "", + "pluginVersion": "10.2.2", "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum((rate(pg_stat_database_xact_commit{instance=\"$Instance\",datname=\"$Database\"}[$Interval])))+sum((rate(pg_stat_database_xact_rollback{instance=\"$Instance\",datname=\"$Database\"}[$Interval])))", "interval": "", "legendFormat": "", "refId": "A" } ], - "thresholds": "", - "timeFrom": null, - "timeShift": null, "title": "Transaction rate", - "type": "singlestat", - "valueFontSize": "80%", - "valueMaps": [ - { - "op": "=", - "text": "N/A", - "value": "null" - } - ], - "valueName": "current" + "type": "stat" }, { "aliasColors": {}, "bars": true, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "decimals": 0, "description": "View: pg_stat_activity", "fieldConfig": { @@ -2750,28 +2629,23 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": true, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (state) (pg_stat_activity_count{instance=\"$Instance\",datname=\"$Database\"})", "legendFormat": "{{state}}", "refId": "A" } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Connections by state (stacked)", "tooltip": { "shared": true, @@ -2780,9 +2654,7 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, @@ -2790,25 +2662,18 @@ { "decimals": 0, "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true }, { "decimals": 0, "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -2816,7 +2681,10 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "decimals": 2, "description": "1 Minute rate of transactions committed or rollback.", "fieldConfig": { @@ -2859,25 +2727,26 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_xact_commit{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "committed", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_xact_rollback{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "hide": false, "interval": "", @@ -2886,9 +2755,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Transactions", "tooltip": { "shared": true, @@ -2897,33 +2764,24 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, - "min": null, "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -2931,7 +2789,10 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_stat_database", "fieldConfig": { "defaults": { @@ -2970,31 +2831,36 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_tup_inserted{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "Inserts", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_tup_updated{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "Updates", "refId": "B" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_tup_deleted{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "Deletes", @@ -3002,9 +2868,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Tuples inserts/updates/deletes", "tooltip": { "shared": true, @@ -3013,33 +2877,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -3047,7 +2904,10 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_stat_database\n\n* tup_fetched: rows needed to satisfy queries\n* tup_returned: rows read/scanned", "fieldConfig": { "defaults": { @@ -3086,25 +2946,26 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_tup_fetched{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "Fetched", "refId": "A" }, { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_tup_returned{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "Returned", @@ -3112,9 +2973,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Tuples fetched/returned", "tooltip": { "shared": true, @@ -3123,33 +2982,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -3157,7 +3009,10 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Source: pg_locks", "fieldConfig": { "defaults": { @@ -3201,28 +3056,23 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "sum by (mode) (pg_locks_count{instance=\"$Instance\",datname=\"$Database\"})", "legendFormat": "{{mode}}", "refId": "A" } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Locks by state", "tooltip": { "shared": true, @@ -3231,33 +3081,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -3265,7 +3108,10 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Should be 0 \n\nSource: pg_stat_database\n\nWith log_lock_waits turned on, deadlocks will be logged to the PostgreSQL Logfiles.", "fieldConfig": { "defaults": { @@ -3311,19 +3157,16 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_deadlocks{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "deadlocks", @@ -3331,9 +3174,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Deadlocks by database", "tooltip": { "shared": true, @@ -3342,33 +3183,26 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } }, { @@ -3376,7 +3210,10 @@ "bars": false, "dashLength": 10, "dashes": false, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "description": "Should be 0. If temporary files are created, it can indicate insufficient work_mem. With log_temp_files the creation of temporary files are logged to the PostgreSQL Logfiles.", "fieldConfig": { "defaults": { @@ -3422,19 +3259,16 @@ "pointradius": 2, "points": false, "renderer": "flot", - "scopedVars": { - "Database": { - "selected": false, - "text": "grafana", - "value": "grafana" - } - }, "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "expr": "rate(pg_stat_database_temp_files{instance=\"$Instance\",datname=\"$Database\"}[$Interval])", "interval": "", "legendFormat": "temp_files", @@ -3442,9 +3276,7 @@ } ], "thresholds": [], - "timeFrom": null, "timeRegions": [], - "timeShift": null, "title": "Temporary files by database", "tooltip": { "shared": true, @@ -3453,77 +3285,83 @@ }, "type": "graph", "xaxis": { - "buckets": null, "mode": "time", - "name": null, "show": true, "values": [] }, "yaxes": [ { "format": "none", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true }, { "format": "short", - "label": null, "logBase": 1, - "max": null, "min": "0", "show": true } ], "yaxis": { - "align": false, - "alignLevel": null + "align": false } } ], "repeat": "Database", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "d9592e16-ddb7-4e6e-a53f-b31701d78c14" + }, + "refId": "A" + } + ], "title": "Database: $Database", "type": "row" } ], "refresh": "", - "schemaVersion": 25, - "style": "dark", + "schemaVersion": 38, "tags": [], "templating": { "list": [ { - "allValue": null, "current": {}, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", - "definition": "label_values({job=\"postgres-exporter\"}, instance)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values({job=\"geospatial-exporter\"},instance)", "hide": 0, "includeAll": false, - "label": null, "multi": false, "name": "Instance", "options": [], - "query": "label_values({job=\"postgres-exporter\"}, instance)", + "query": { + "qryType": 1, + "query": "label_values({job=\"geospatial-exporter\"},instance)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, "refresh": 1, "regex": "", "skipUrlSync": false, "sort": 0, "tagValuesQuery": "", - "tags": [], "tagsQuery": "", "type": "query", "useTags": false }, { - "allValue": null, "current": {}, - "datasource": "${DS_OBSERVE-PROMETHEUS02}", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, "definition": "label_values(datname)", "hide": 0, "includeAll": true, - "label": null, "multi": true, "name": "Database", "options": [], @@ -3533,7 +3371,6 @@ "skipUrlSync": false, "sort": 1, "tagValuesQuery": "", - "tags": [], "tagsQuery": "", "type": "query", "useTags": false @@ -3544,11 +3381,10 @@ "auto_min": "10s", "current": { "selected": false, - "text": "10m", - "value": "10m" + "text": "1m", + "value": "1m" }, "hide": 0, - "label": null, "name": "Interval", "options": [ { @@ -3562,12 +3398,12 @@ "value": "30sec" }, { - "selected": false, + "selected": true, "text": "1m", "value": "1m" }, { - "selected": true, + "selected": false, "text": "10m", "value": "10m" }, @@ -3624,5 +3460,5 @@ "title": "PostgreSQL Exporter", "uid": "v5ciIbUZz", "version": 2, - "description": "postgres exporter Dashboard for display important postgreSQL metric" + "weekStart": "" } diff --git a/ingest_service/README.md b/ingest_service/README.md index 56a61b7..8ecdfa6 100644 --- a/ingest_service/README.md +++ b/ingest_service/README.md @@ -4,9 +4,14 @@ The Ingest Service exposes POST API endpoints for applications to send data to t designed to handle various types of data formats and can be extended to support additional formats as needed. -The service is written in FastAPI and uses Pydantic for data validation. Data is then forwarded to +The service is written with FastAPI and uses Pydantic for data validation. Data is then forwarded to the [processor_service](../processor_service/README.md) for further processing. +## Configuration + +The [constants.py](./constants.py) file contains configuration constants for the service. These +should be configured based on the deployment environment. + ## Usage To start the Ingest Service run `make run-local-dev` from this directory. This will create a Python @@ -20,25 +25,124 @@ Currently available endpoints include: - `/ttn`: For data in the format forwarded by the Things Network (TTN). - `/http`: For data in a generic HTTP format. +The service is also started with the `make dev/prod-up` commands from the root directory, which will +start all the services defined in the `docker-compose` files. + +## Documentation + +REST API documentation is available at the `/docs` endpoint when the server is running. In the case +of a local deployment, it can be accessed at `http://localhost:50005/docs`. + ## Data Format The service expects data in a predefined format, which is validated using Pydantic models. The data structure is defined in the `models` directory. -Even though the structure is predefined, what is variable is the actual data being sent, which can -differ based on the device and application. +Even though the structure is predefined, the sensor data can vary, depending on the device and +application. The data should be part of the `measurement` dictionary, which can contain any number of key-value pairs representing the actual data being sent. The keys in this dictionary are not fixed and can vary based on the device and application sending the data. -### TTN +### Example Data + +#### HTTP Data + +Below is an example of a valid data structure that can be sent to the `/http` endpoint: + +```json +{ + "sensor_name": "string", + "application": "string", + "measurement": { + "temperature": 23.5, + "humidity": 60, + "version": "v1" + }, + "location": { + "latitude": 45.0, + "longitude": 12.0 + }, + "received_at": "2023-01-01T00:00:00Z" +} +``` + +See the [ingest_service/models/http.py](../ingest_service/models/http.py) for the Pydantic model +that validates the data sent to this endpoint. + +#### TTN Data The data sent by any TTN formation generally follows the same structure, therefore a Pydantic validator is in place to ensure that the data conforms to the expected format. See the model in -`ingest_service/models/ttn.py` for details. - -### HTTP +[ingest_service/models/ttn.py](../ingest_service/models/ttn.py) for details. + +Example of data received from a TTN application: + +```json +{ + "end_device_ids": { + "device_id": "mock-dev-090", + "application_ids": { + "application_id": "irnas-mock-application" + } + }, + "received_at": "2023-01-01T00:00:00Z", + "uplink_message": { + "decoded_payload": { + "measurement": { + "temperature": 22.5, + "humidity": 45.0, + "version": "v1" + } + } + } +} +``` + +There are additional fields in the TTN data structure, but they are not relevant for the Ingest +Service. + +## TTN Integration + +To integrate a TTN application with the Ingest Service, you need to set up a webhook in the TTN +console. The webhook should point to the `/ttn` endpoint of the Ingest Service. + +1. To do so navigate to your TTN application, go to the `Webhooks` tab, and click on the + `Add webhook` button. +2. Select `Custom webhook` from the list of available webhook types. +3. Set the `Webhook ID`, the `Webhook format` to **JSON**, and the `Base URL` to the URL where your + `Ingest Service` is hosted, e.g. `http://localhost:50005`. In the case of a deployment, this + should be the public URL of the service. +4. Click on the `Add header entry` button and add a header with the key `Authorization` and the + value `Bearer your-secret-api-key`, replacing `your-secret-api-key` with the actual API key + defined in the `constants.py` file. +5. Select the `Uplink message` checkbox to enable uplink messages to be sent to the webhook. Set the + value to `/ttn`. +6. Confirm by clicking the `Add webhook` button at the bottom of the page. + +![ttn-webhook-setup](../docs/images/ttn_integration.png) + +> [!IMPORTANT] Network Configuration In order for the integration to work, the Ingest Service must +> be accessible from the internet if TTN is to reach it. This may involve setting up port forwarding +> on your router or using a service like [ngrok](https://ngrok.com/) to expose your local server to +> the internet. + +### Decoder + +For the integration to work, a decoder must be set up in the TTN console that decodes the raw +payload into a JSON object. + +The decoder should ensure that the decoded payload contains a `measurement` field, which is a +dictionary containing the actual sensor data. + +The decoder should also include a `version` field within the `measurement` dictionary to indicate +the version of the data format being used. + +This `version` field is crucial for the Processor Service to determine how to process and store the +incoming data correctly. + +## HTTP Integration The `/http` endpoint is meant to be used by any other application that wants to send data to the Ingest Service in a generic HTTP format. @@ -46,10 +150,13 @@ Ingest Service in a generic HTTP format. The sensor data structure is flexible and can be defined by the application sending the data. The only requirement is that the data should be part of the `measurement` dictionary. -See the `ingest_service/models/http.py` for the Pydantic model that validates the data sent to this -endpoint. +Similarly to the TTN integration, you need to set up your application to send HTTP POST requests to +the `/http` endpoint of the Ingest Service. + +> [!IMPORTANT] Authentication Make sure to include the `Authorization` header with the API key. Also +> make sure the data is in the [correct format](#data-format). ## Mock Data Generation -The `mock/ingest_service` directory contains scripts to generate mock data that can be ingested by -the service. This is useful for testing and development purposes. +The [mock/ingest_service](../mock/ingest_service/README.md) directory contains scripts to generate +mock data that can be ingested by the service. This is useful for testing and development purposes. diff --git a/init/postgres.sql b/init/postgres.sql index dfc713e..3972a1e 100644 --- a/init/postgres.sql +++ b/init/postgres.sql @@ -1,50 +1,16 @@ CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE; CREATE EXTENSION IF NOT EXISTS postgis; +-- Create static tables + CREATE TABLE IF NOT EXISTS sensors ( id SERIAL PRIMARY KEY, - sensor_uid UUID NOT NULL UNIQUE DEFAULT gen_random_uuid(), -- renamed to avoid confusion with PK - sensor_name VARCHAR(255) NOT NULL, -- Added for better identification + sensor_uid UUID NOT NULL UNIQUE DEFAULT gen_random_uuid(), application_id INTEGER NOT NULL, created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP ); --- This is generated programmatically first time sensor data is received - --- CREATE TABLE IF NOT EXISTS water_level_data ( --- id SERIAL PRIMARY KEY, --- sensor_id INTEGER NOT NULL, --- water_level DOUBLE PRECISION NOT NULL --- ); - --- CREATE TABLE IF NOT EXISTS fire_sensor_data ( --- id SERIAL PRIMARY KEY , --- sensor_id INTEGER NOT NULL, --- temperature FLOAT NOT NULL, --- smoke_level FLOAT NOT NULL --- ); - --- CREATE TABLE IF NOT EXISTS locations ( --- id SERIAL PRIMARY KEY, --- description TEXT NOT NULL, --- name TEXT NOT NULL, --- created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP --- ); - --- CREATE TABLE IF NOT EXISTS metadata ( --- id SERIAL, --- key TEXT NOT NULL, --- value TEXT NOT NULL, --- created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, --- updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, --- sensor_id INTEGER, --- application_id INTEGER, --- location_id INTEGER --- ); - --- Create metatable holding information about application and sensor - CREATE TABLE IF NOT EXISTS applications( id SERIAL PRIMARY KEY, app_name TEXT NOT NULL, @@ -61,38 +27,7 @@ CREATE TABLE IF NOT EXISTS application_measurements_registry( updated_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP ); - --- Create hypertables - --- CREATE TABLE IF NOT EXISTS positions ( --- id INTEGER NOT NULL, --- sensor_id INTEGER NOT NULL, --- latitude DOUBLE PRECISION NOT NULL, --- longitude DOUBLE PRECISION NOT NULL, --- altitude FLOAT, --- position_time TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, --- geom GEOGRAPHY(Point, 4326), -- Geospatial column for geographic operations --- PRIMARY KEY (id, sensor_id, position_time) --- ); - --- CREATE TABLE IF NOT EXISTS measurements ( --- id INTEGER NOT NULL, --- sensor_id INTEGER NOT NULL, --- application_id INTEGER NOT NULL, --- timestamp TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP, --- PRIMARY KEY (id, sensor_id, application_id, timestamp) --- ); - --- 2.1. Create a geospatial index to improve spatial query performance --- CREATE INDEX IF NOT EXISTS idx_positions_geom ON positions USING GIST (geom); --- 2.2. Add unique constraint on deviceId and date to support ON CONFLICT operations - --- SELECT create_hypertable('positions', 'position_time', 'sensor_id', if_not_exists => TRUE, number_partitions => 10); --- SELECT create_hypertable('measurements', 'timestamp', 'sensor_id', if_not_exists => TRUE, number_partitions => 10); - --- 3.2. Create additional indexes to optimize queries --- CREATE INDEX IF NOT EXISTS idx_positions_id ON positions (id); --- CREATE INDEX IF NOT EXISTS idx_measurements_timestamp ON measurements (timestamp); +-- Add constraints to ensure data integrity ALTER TABLE applications DROP CONSTRAINT IF EXISTS unique_application_name; ALTER TABLE applications ADD CONSTRAINT unique_application_name UNIQUE (app_name); @@ -103,24 +38,6 @@ ALTER TABLE application_measurements_registry ADD CONSTRAINT unique_application_ ALTER TABLE sensors DROP CONSTRAINT IF EXISTS unique_sensor_name; ALTER TABLE sensors ADD CONSTRAINT unique_sensor_name UNIQUE (sensor_name); --- ----------------------------------------------------------------------------------- --- 4. Establishing Referential Integrity --- ----------------------------------------------------------------------------------- - --- -- 4.5. Add a foreign key from `metadata.sensor_id` to `sensors.id` --- ALTER TABLE metadata --- ADD CONSTRAINT fk_metadata_sensors --- FOREIGN KEY (sensor_id) --- REFERENCES sensors(id) --- ON DELETE CASCADE; - --- -- 4.7. Add a foreign key from `metadata.location_id` to `location.id` --- ALTER TABLE metadata --- ADD CONSTRAINT fk_metadata_location --- FOREIGN KEY (location_id) --- REFERENCES locations(id) --- ON DELETE CASCADE; - ALTER TABLE application_measurements_registry ADD CONSTRAINT fk_application_measurements_registry_applications FOREIGN KEY (application_id) @@ -132,13 +49,6 @@ ADD CONSTRAINT fk_sensors_applications FOREIGN KEY (application_id) REFERENCES applications(id); --- -- 4.8. Add a foreign key from `metadata.application_id` to `applications.id` --- ALTER TABLE metadata --- ADD CONSTRAINT fk_metadata_applications --- FOREIGN KEY (application_id) --- REFERENCES applications(id) --- ON DELETE CASCADE; - -- Warning: This affects all transactions on the PostgreSQL instance. -- Required for high-volume FK operations. Requires PostgreSQL restart. ALTER SYSTEM SET max_locks_per_transaction = 512; diff --git a/mock/ingest_service/README.md b/mock/ingest_service/README.md index e311178..dbd4c6d 100644 --- a/mock/ingest_service/README.md +++ b/mock/ingest_service/README.md @@ -30,7 +30,7 @@ The arguments specified in the command line are: - `--devices`: Number of devices to generate. - `--applications`: Number of applications to generate. -- `--messages`: Number of messages to send. If set to 0 it will send messages indefinitely. +- `--messages`: Number of messages to send. If set to 0 data will be sent indefinitely. - `--url`: URL of the ingest service to send the data to. Defaults to `http://localhost:50005/`. - `--type`: Type of the device, either `ttn` or `http`. diff --git a/processor_service/README.md b/processor_service/README.md index f2d8cbd..14a3b81 100644 --- a/processor_service/README.md +++ b/processor_service/README.md @@ -17,6 +17,11 @@ directory: make run-local-dev ``` +## Configuration + +The [constants.py](./constants.py) file contains configuration constants for the service. These +should be configured based on the deployment environment. + ## Tables An `application` is the endpoint the data is receive from. In the case of a TTN application, it is @@ -75,6 +80,6 @@ The table is created to be as general as possible, therefore all numeric fields In the image below the ER model is shown. You can see the two dynamically generated `measurements_1` and `measurements_2` tables, which are logically linked to the `application` via their names (red -lines on the image). +connections). ![ER Model](../docs/images/er_model.png) diff --git a/processor_service/constants.py b/processor_service/constants.py index 3b848fa..e8f0907 100644 --- a/processor_service/constants.py +++ b/processor_service/constants.py @@ -1,4 +1,6 @@ import os localhost = os.environ.get("LOCALHOST") == "true" -KAFKA_HOST = "localhost:9992" if localhost else "kafka:9993" +KAFKA_HOST = ( + "localhost:9992" if localhost else "kafka:9993" +) # Kafka broker to connect to and consume messages from diff --git a/rest_api/README.md b/rest_api/README.md index 64aed1f..a52d3ae 100644 --- a/rest_api/README.md +++ b/rest_api/README.md @@ -12,7 +12,7 @@ make run-local-dev ## Endpoints -Currently the following endpoints are available: +The following endpoints are available: ### `GET /api/sensors` @@ -58,14 +58,17 @@ Output: { "applications": [ { + "id": 3, "app_name": "mock-app-4", "description": null }, { + "id": 4, "app_name": "mock-app-2", "description": null }, { + "id": 5, "app_name": "mock-app-10", "description": null } @@ -73,15 +76,20 @@ Output: } ``` -### `GET /api/measurements?sensor_name={sensor_name}` +### `GET /api/measurements?sensor_name={sensor_name}&limit={limit}&offset={offset}` Fetches measurements for a specific sensor. The response includes the sensor name, table name, and a list of measurements. +The measurements are sorted by timestamp in descending order (most recent first). + +A pagination mechanism is implemented using `limit` and `offset` query parameters. Pagination info +is returned in the `page_info` field. + Example: ```bash -curl "http://localhost:51005/api/measurements?sensor_name=mock-dev-004" -H "Authorization: Bearer your-api-token" | jq +curl "http://localhost:51005/api/measurements?sensor_name=mock-dev-004&limit=10&offset=0" -H "Authorization: Bearer your-api-token" | jq ``` Output: @@ -90,23 +98,189 @@ Output: { "measurements": [ { - "table_name": "mock-app-4_v4", - "timestamp": "2025-07-18 12:42:25.242002+00:00", + "table_name": "mock-app-5_v1", + "timestamp": "2025-09-11 12:47:24.985990+00:00", + "data": { + "temperature": -80.0, + "motion": -56.0, + "gps_latitude": 73.0, + "note": 19.0, + "gps_timestamp": 0.0 + } + }, + { + "table_name": "mock-app-5_v7", + "timestamp": "2025-09-10 14:53:12.984453+00:00", + "data": { + "motion": 14.0, + "gps_longitude": -28.0, + "gps_timestamp": 53.0, + "acceleration": 69.0, + "note": 52.0 + } + }, + { + "table_name": "mock-app-5_v0", + "timestamp": "2025-09-10 14:53:06.477678+00:00", "data": { - "current": 24.0, - "gps_heading": -96.0, - "gps_timestamp": 6.0, - "gps_longitude": 87.0, - "gps_speed": -40.0, - "speed": -4.0, - "acceleration": -84.0, - "battery": 99.0, - "pressure": 91.0, - "gyroscope": -49.0, - "temperature": 49.0, - "voltage": null + "gps_altitude": -7.0, + "voltage": -93.0, + "gps_timestamp": -85.0, + "gps_latitude": -52.0, + "acceleration": -95.0, + "battery": 44.0, + "temperature": -71.0, + "gps_speed": -22.0 } + }, + { + "table_name": "mock-app-5_v0", + "timestamp": "2025-09-10 14:53:04.465782+00:00", + "data": { + "gps_altitude": 71.0, + "gyroscope": -52.0, + "magnetometer": -19.0, + "gps_latitude": 55.0, + "distance": 43.0, + "acceleration": -80.0, + "temperature": 11.0 + } + }, + { + "table_name": "mock-app-5_v3", + "timestamp": "2025-09-10 14:53:04.365329+00:00", + "data": { + "battery": -27.0, + "acceleration": 16.0, + "gps_longitude": 52.0, + "current": -91.0, + "note": -12.0 + } + }, + { + "table_name": "mock-app-5_v1", + "timestamp": "2025-09-10 14:53:02.095786+00:00", + "data": { + "gps_altitude": 79.0, + "humidity": 74.0, + "gps_heading": -77.0, + "gyroscope": 55.0, + "light": 55.0, + "current": 39.0, + "gps_timestamp": -18.0, + "battery": -73.0, + "gps_longitude": -92.0, + "gps_speed": -37.0 + } + }, + { + "table_name": "mock-app-5_v6", + "timestamp": "2025-09-10 14:53:00.836799+00:00", + "data": { + "gyroscope": 3.0 + } + }, + { + "table_name": "mock-app-5_v0", + "timestamp": "2025-09-10 14:52:55.914773+00:00", + "data": { + "gps_latitude": -15.0, + "distance": 61.0, + "motion": -8.0, + "note": -58.0 + } + }, + { + "table_name": "mock-app-5_v4", + "timestamp": "2025-09-10 14:52:51.694956+00:00", + "data": { + "light": 27.0, + "humidity": -71.0, + "gps_speed": 55.0, + "current": -75.0, + "temperature": 21.0, + "acceleration": 98.0 + } + }, + { + "table_name": "mock-app-5_v5", + "timestamp": "2025-09-10 14:52:48.586263+00:00", + "data": { + "gps_altitude": -89.0, + "battery": 7.0 + } + } + ], + "page_info": { + "total": 93, + "count": 10, + "limit": 10, + "offset": 0, + "page": 1, + "total_pages": 10, + "has_next": true, + "has_previous": false + } +} +``` + +### `Get /api/applications/{application_name}/sensors` + +Lists all sensors for a specific application. The response includes sensor names and the application +they belong to. + +Example: + +```bash +curl http://localhost:51005/api/applications/mock-app-5/sensors -H "Authorization: Bearer your-api-token" | jq +``` + +Output: + +```json +{ + "sensors": [ + { + "sensor_name": "mock-dev-006", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-019", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-013", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-068", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-003", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-066", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-032", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-063", + "application_id": 17 + }, + { + "sensor_name": "mock-dev-055", + "application_id": 17 } ] } ``` + +## OpenAPI Documentation + +OpenAPI documentation is available at `/docs` endpoint when the server is running. In the case of a +local deployment, it can be accessed at `http://localhost:51005/docs`. diff --git a/rest_api/main.py b/rest_api/main.py index 99a0aa9..f24aa7c 100644 --- a/rest_api/main.py +++ b/rest_api/main.py @@ -55,7 +55,10 @@ async def lifespan(app): app.add_middleware( CORSMiddleware, - allow_origins=["http://localhost:3001", "application:3001"], + allow_origins=[ + "http://localhost:3001", + "http://localhost:8004", + ], # Configure these as needed allow_credentials=True, allow_methods=["*"], allow_headers=["*"],