Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions docs/src/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,12 @@ graph TB

Connections between entities forming a knowledge graph.

- 🌐 **[Webhooks](webhooks.md)**

---

Runtime-configurable connectors that authenticate external events and map payloads to your data model.

</div>

---
Expand Down Expand Up @@ -124,3 +130,4 @@ Dive deeper into each concept:
- **[Entity Templates](entity-templates.md)** - Learn how to design your data model
- **[Properties](properties.md)** - Understand property types and validation
- **[Relations](relations.md)** - Connect your entities into a graph
- **[Webhooks](webhooks.md)** - Configure inbound integrations and security strategies
216 changes: 216 additions & 0 deletions docs/src/concepts/webhooks.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
---
title: Webhooks
description: Understand webhook connectors, security strategies, and dynamic mappings in IDP-Core
---

Webhooks let external systems push JSON events to IDP-Core through a generic HTTP endpoint. You configure a webhook connector at runtime, choose a security strategy, and define mappings that translate incoming payloads into entity data.

## Overview

A webhook connector combines three concerns:

- **Connector metadata** - Identifier, title, description, and enabled flag
- **Security** - How IDP-Core authenticates incoming requests
- **Mappings** - How the payload maps to an Entity Template

```mermaid
flowchart LR
S[External system] --> E[POST /webhooks/{configurationId}]
E --> H[InboundWebhookHandler]
H --> D[Security dispatcher]
D --> C[WebhookConnector]
C --> M[Dynamic mappings]
M --> T[Entity Template]
```

## Webhook Connector

A webhook connector is the runtime configuration stored by IDP-Core for one inbound integration.

| Field | Type | Description |
| --- | --- | --- |
| `identifier` | String | Stable key used in the webhook URL and management APIs |
| `title` | String | Human-readable name |
| `description` | String | Optional explanation of the connector purpose |
| `enabled` | Boolean | Enables or disables request processing |
| `mappings` | Array | One or more dynamic mapping rules |
| `security` | Object | Authentication strategy and configuration |

### Example

```json
{
"identifier": "github-repositories",
"title": "GitHub repositories",
"description": "Receives repository events from GitHub",
"enabled": true,
"mappings": [
{
"template": "github_repository",
"filter": ".action == \"created\" or .action == \"edited\"",
"entity": {
"identifier": ".repository.full_name | gsub(\"/\"; \"_\")",
"title": ".repository.name",
"properties": {
"name": ".repository.name",
"url": ".repository.html_url",
"language": ".repository.language // \"Unknown\""
},
"relations": {
"owner": ".repository.owner.login"
}
}
}
],
"security": {
"type": "HMAC_SHA256",
"config": {
"header_name": "X-Hub-Signature-256",
"secret_alias": "GITHUB_WEBHOOK_SECRET",
"prefix": "sha256="
}
}
}
```

## Dynamic Mappings

Each connector contains at least one dynamic mapping. A mapping targets one Entity Template and describes how to derive entity fields from the incoming JSON payload.

| Field | Description |
| --- | --- |
| `template` | Target Entity Template identifier |
| `filter` | Expression that decides whether the mapping applies |
| `entity.identifier` | Expression that generates the entity identifier |
| `entity.title` | Expression that generates the entity title |
| `entity.properties` | Map of template property names to extraction expressions |
| `entity.relations` | Map of template relation names to extraction expressions |

### Validation Rules

When you create or update a connector, IDP-Core validates each mapping against the target Entity Template.

It checks that:

- The referenced template exists
- Every mapped property exists in the template
- Every required property is mapped
- Every mapped relation exists in the template
- Every required relation is mapped

This validation keeps the connector configuration aligned with the current data model.

## Security Strategies

Each connector declares one security type. IDP-Core validates the configuration at creation time and validates requests again at runtime.

| Type | Required configuration keys | Runtime behavior |
| --- | --- | --- |
| `HMAC_SHA256` | `header_name`, `secret_alias` | Computes the SHA-256 HMAC of the raw body and compares it with the request header |
| `STATIC_TOKEN` | `header_name`, `secret_alias` | Compares a header value with a secret loaded from the environment |
| `BASIC_AUTH` | `username`, `secret_alias` | Compares the `Authorization: Basic ...` header with the configured username and secret |
| `JWT_BEARER` | `jwks_uri` | Validates the bearer token against a JWKS endpoint |
| `NONE` | none | Skips authentication |

> [!IMPORTANT]
> Security configuration keys accept `snake_case` and `camelCase` variants for the supported fields.
> [!WARNING]
> `secret_alias` must reference an environment variable alias in `UPPER_SNAKE_CASE`. It does not store the raw secret value in the connector configuration.

### Example Security Configurations

=== "HMAC_SHA256"
```json
{
"type": "HMAC_SHA256",
"config": {
"header_name": "X-Hub-Signature-256",
"secret_alias": "GITHUB_WEBHOOK_SECRET",
"prefix": "sha256="
}
}
```

=== "STATIC_TOKEN"
```json
{
"type": "STATIC_TOKEN",
"config": {
"header_name": "X-Webhook-Token",
"secret_alias": "WEBHOOK_SHARED_TOKEN"
}
}
```

=== "BASIC_AUTH"
```json
{
"type": "BASIC_AUTH",
"config": {
"username": "webhook-user",
"secret_alias": "WEBHOOK_PASSWORD"
}
}
```

=== "JWT_BEARER"
```json
{
"type": "JWT_BEARER",
"config": {
"jwks_uri": "https://issuer.example.com/.well-known/jwks.json"
}
}
```

## Runtime Flow

The webhook runtime uses a single generic endpoint:

```text
POST /webhooks/{configurationId}
```

The request flow is:

1. IDP-Core receives the request on the generic webhook endpoint.
2. The `configurationId` resolves the stored `WebhookConnector`.
3. If the connector is disabled, IDP-Core ignores the event.
4. The security dispatcher selects the matching strategy for the connector security type.
5. The strategy validates the headers and, when needed, the raw request body.
6. After authentication, the event is accepted for downstream processing.

> [!IMPORTANT]
> The connector model, security validation, management APIs, and mapping validation are implemented now. The final payload-to-entity ingestion route is still marked as pending Camel routing in the current handler implementation.

## Management Lifecycle

You manage webhook connectors through the inbound webhook management API.

| Operation | Endpoint |
| --- | --- |
| Create connector | `POST /api/v1/inbound-webhooks` |
| List connectors | `GET /api/v1/inbound-webhooks` |
| Get connector | `GET /api/v1/inbound-webhooks/{identifier}` |
| Update connector | `PUT /api/v1/inbound-webhooks/{identifier}` |
| Delete connector | `DELETE /api/v1/inbound-webhooks/{identifier}` |

This separation keeps configuration management under versioned API routes while the event ingestion endpoint stays simple for external systems.

## When to Use Webhooks

Use webhooks when an external system can push JSON events over HTTP and you want to:

- Ingest updates without redeploying IDP-Core
- Reuse one generic endpoint for multiple providers
- Apply connector-specific authentication rules
- Map external payloads to your own Entity Templates at runtime

---

## Next Steps

- **[Entity Templates](entity-templates.md)** - Define the target structures that mappings reference
- **[Entities](entities.md)** - Understand the records produced by successful ingestion
- **[Relations](relations.md)** - Model links that webhook mappings can populate
- **[Data Integration](../features/data-integration.md)** - Explore the broader ingestion roadmap
3 changes: 2 additions & 1 deletion docs/zensical.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ nav = [
"concepts/entity-templates.md",
"concepts/entities.md",
"concepts/properties.md",
"concepts/relations.md"
"concepts/relations.md",
"concepts/webhooks.md"
]},
{ "Features" = [
"features/index.md",
Expand Down
8 changes: 8 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -226,6 +226,14 @@
<version>3.20.0</version>
</dependency>

<!-- JSLT: JVM-native JSON transformation language used as the DSL engine for dynamic mappings -->
<!-- ADR 0004: JSLT chosen over JQ for performance, robust syntax checking and JVM-native execution -->
<dependency>
<groupId>com.schibsted.spt.data</groupId>
<artifactId>jslt</artifactId>
<version>0.1.14</version>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ public class ValidationMessages {

public static final String PROPERTY_RULES_MUTUALLY_EXCLUSIVE = "{rule1} and {rule2} are mutually exclusive for STRING properties";


//Webhook connector validation messages
public static final String WEBHOOK_CONNECTOR_ALREADY_EXIST="Webhook Connector already exists with the same identifier";
public static final String WEBHOOK_CONNECTOR_IDENTIFIER_MANDATORY = "Webhook Connector identifier is mandatory and cannot be blank";
public static final String WEBHOOK_CONNECTOR_TITLE_ALREADY_EXIST ="Webhook Connector already exist with the same name";
public static final String WEBHOOK_CONNECTOR_MAPPINGS_MANDATORY = "Webhook mappings section is mandatory";
// Helper method to construct rules incompatibility message
public static String rulesAreIncompatible(String rule1, String rule2) {
return PROPERTY_RULES_MUTUALLY_EXCLUSIVE
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
package com.decathlon.idp_core.domain.exception.entity_mapping;

public class EntityDynamicMappingConfigurationException extends RuntimeException {

public EntityDynamicMappingConfigurationException(String message) {
super(message);
}

public EntityDynamicMappingConfigurationException(String message, Throwable cause) {
super(message, cause);
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
package com.decathlon.idp_core.domain.exception.entity_template;

public class PropertyNameNotFoundEntityTemplatePropertiesException extends RuntimeException {
public PropertyNameNotFoundEntityTemplatePropertiesException(String message) {
super(message);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
package com.decathlon.idp_core.domain.exception.entity_template;

public class RelationNameNotFoundEntityTemplateRelationsException extends RuntimeException {
public RelationNameNotFoundEntityTemplateRelationsException(String message) {
super(message);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
package com.decathlon.idp_core.domain.exception.webhook;

public class WebhookAuthenticationException extends RuntimeException {
public WebhookAuthenticationException(String message) {
super(message);
}

public WebhookAuthenticationException(String message, Throwable cause) {
super(message, cause);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
package com.decathlon.idp_core.domain.exception.webhook;

import static com.decathlon.idp_core.domain.constant.ValidationMessages.WEBHOOK_CONNECTOR_ALREADY_EXIST;

public class WebhookConnectorAlreadyExistException extends RuntimeException {

public WebhookConnectorAlreadyExistException(String identifier) {
super(String.format("%s:%s", WEBHOOK_CONNECTOR_ALREADY_EXIST, identifier));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package com.decathlon.idp_core.domain.exception.webhook;

public class WebhookConnectorNotFoundException extends RuntimeException {

public WebhookConnectorNotFoundException(String identifier) {
super(String.format("No webhook connector found for identifier: %s", identifier));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
package com.decathlon.idp_core.domain.exception.webhook;

import static com.decathlon.idp_core.domain.constant.ValidationMessages.WEBHOOK_CONNECTOR_TITLE_ALREADY_EXIST;

public class WebhookConnectorTitleAlreadyExistsException extends RuntimeException {
public WebhookConnectorTitleAlreadyExistsException(String webhookName) {
super(String.format("%s:%s", WEBHOOK_CONNECTOR_TITLE_ALREADY_EXIST, webhookName));
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package com.decathlon.idp_core.domain.exception.webhook;

public class WebhookSecurityConfigurationException extends RuntimeException {

public WebhookSecurityConfigurationException(String message) {
super(message);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
package com.decathlon.idp_core.domain.exception.webhook;

public class WebhookTemplateHasNoPropertiesException extends RuntimeException {

public WebhookTemplateHasNoPropertiesException(String message) {
super(message);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
package com.decathlon.idp_core.domain.model.entity_mapping;

import java.util.Map;

import com.decathlon.idp_core.domain.model.webhook.WebhookConnector;
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Delete unused import

import jakarta.validation.constraints.NotBlank;
import jakarta.validation.constraints.NotNull;

public record
EntityDynamicMapping(
@NotBlank
String templateIdentifier,
@NotBlank
String filter,
@NotBlank
String entityIdentifier,
@NotBlank
String entityTitle,
@NotNull
Map<String, String> properties,
@NotNull
Map<String, String> relations
) {
}
Loading