-
Notifications
You must be signed in to change notification settings - Fork 0
feat(core): add endpoint for creating inbound webhook connector #58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
foukou19
wants to merge
2
commits into
main
Choose a base branch
from
feat/idp-core-add-inbound-webhook-connector-endpoint
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,216 @@ | ||
| --- | ||
| title: Webhooks | ||
| description: Understand webhook connectors, security strategies, and dynamic mappings in IDP-Core | ||
| --- | ||
|
|
||
| Webhooks let external systems push JSON events to IDP-Core through a generic HTTP endpoint. You configure a webhook connector at runtime, choose a security strategy, and define mappings that translate incoming payloads into entity data. | ||
|
|
||
| ## Overview | ||
|
|
||
| A webhook connector combines three concerns: | ||
|
|
||
| - **Connector metadata** - Identifier, title, description, and enabled flag | ||
| - **Security** - How IDP-Core authenticates incoming requests | ||
| - **Mappings** - How the payload maps to an Entity Template | ||
|
|
||
| ```mermaid | ||
| flowchart LR | ||
| S[External system] --> E[POST /webhooks/{configurationId}] | ||
| E --> H[InboundWebhookHandler] | ||
| H --> D[Security dispatcher] | ||
| D --> C[WebhookConnector] | ||
| C --> M[Dynamic mappings] | ||
| M --> T[Entity Template] | ||
| ``` | ||
|
|
||
| ## Webhook Connector | ||
|
|
||
| A webhook connector is the runtime configuration stored by IDP-Core for one inbound integration. | ||
|
|
||
| | Field | Type | Description | | ||
| | --- | --- | --- | | ||
| | `identifier` | String | Stable key used in the webhook URL and management APIs | | ||
| | `title` | String | Human-readable name | | ||
| | `description` | String | Optional explanation of the connector purpose | | ||
| | `enabled` | Boolean | Enables or disables request processing | | ||
| | `mappings` | Array | One or more dynamic mapping rules | | ||
| | `security` | Object | Authentication strategy and configuration | | ||
|
|
||
| ### Example | ||
|
|
||
| ```json | ||
| { | ||
| "identifier": "github-repositories", | ||
| "title": "GitHub repositories", | ||
| "description": "Receives repository events from GitHub", | ||
| "enabled": true, | ||
| "mappings": [ | ||
| { | ||
| "template": "github_repository", | ||
| "filter": ".action == \"created\" or .action == \"edited\"", | ||
| "entity": { | ||
| "identifier": ".repository.full_name | gsub(\"/\"; \"_\")", | ||
| "title": ".repository.name", | ||
| "properties": { | ||
| "name": ".repository.name", | ||
| "url": ".repository.html_url", | ||
| "language": ".repository.language // \"Unknown\"" | ||
| }, | ||
| "relations": { | ||
| "owner": ".repository.owner.login" | ||
| } | ||
| } | ||
| } | ||
| ], | ||
| "security": { | ||
| "type": "HMAC_SHA256", | ||
| "config": { | ||
| "header_name": "X-Hub-Signature-256", | ||
| "secret_alias": "GITHUB_WEBHOOK_SECRET", | ||
| "prefix": "sha256=" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Dynamic Mappings | ||
|
|
||
| Each connector contains at least one dynamic mapping. A mapping targets one Entity Template and describes how to derive entity fields from the incoming JSON payload. | ||
|
|
||
| | Field | Description | | ||
| | --- | --- | | ||
| | `template` | Target Entity Template identifier | | ||
| | `filter` | Expression that decides whether the mapping applies | | ||
| | `entity.identifier` | Expression that generates the entity identifier | | ||
| | `entity.title` | Expression that generates the entity title | | ||
| | `entity.properties` | Map of template property names to extraction expressions | | ||
| | `entity.relations` | Map of template relation names to extraction expressions | | ||
|
|
||
| ### Validation Rules | ||
|
|
||
| When you create or update a connector, IDP-Core validates each mapping against the target Entity Template. | ||
|
|
||
| It checks that: | ||
|
|
||
| - The referenced template exists | ||
| - Every mapped property exists in the template | ||
| - Every required property is mapped | ||
| - Every mapped relation exists in the template | ||
| - Every required relation is mapped | ||
|
|
||
| This validation keeps the connector configuration aligned with the current data model. | ||
|
|
||
| ## Security Strategies | ||
|
|
||
| Each connector declares one security type. IDP-Core validates the configuration at creation time and validates requests again at runtime. | ||
|
|
||
| | Type | Required configuration keys | Runtime behavior | | ||
| | --- | --- | --- | | ||
| | `HMAC_SHA256` | `header_name`, `secret_alias` | Computes the SHA-256 HMAC of the raw body and compares it with the request header | | ||
| | `STATIC_TOKEN` | `header_name`, `secret_alias` | Compares a header value with a secret loaded from the environment | | ||
| | `BASIC_AUTH` | `username`, `secret_alias` | Compares the `Authorization: Basic ...` header with the configured username and secret | | ||
| | `JWT_BEARER` | `jwks_uri` | Validates the bearer token against a JWKS endpoint | | ||
| | `NONE` | none | Skips authentication | | ||
|
|
||
| > [!IMPORTANT] | ||
| > Security configuration keys accept `snake_case` and `camelCase` variants for the supported fields. | ||
| > [!WARNING] | ||
| > `secret_alias` must reference an environment variable alias in `UPPER_SNAKE_CASE`. It does not store the raw secret value in the connector configuration. | ||
|
|
||
| ### Example Security Configurations | ||
|
|
||
| === "HMAC_SHA256" | ||
| ```json | ||
| { | ||
| "type": "HMAC_SHA256", | ||
| "config": { | ||
| "header_name": "X-Hub-Signature-256", | ||
| "secret_alias": "GITHUB_WEBHOOK_SECRET", | ||
| "prefix": "sha256=" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| === "STATIC_TOKEN" | ||
| ```json | ||
| { | ||
| "type": "STATIC_TOKEN", | ||
| "config": { | ||
| "header_name": "X-Webhook-Token", | ||
| "secret_alias": "WEBHOOK_SHARED_TOKEN" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| === "BASIC_AUTH" | ||
| ```json | ||
| { | ||
| "type": "BASIC_AUTH", | ||
| "config": { | ||
| "username": "webhook-user", | ||
| "secret_alias": "WEBHOOK_PASSWORD" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| === "JWT_BEARER" | ||
| ```json | ||
| { | ||
| "type": "JWT_BEARER", | ||
| "config": { | ||
| "jwks_uri": "https://issuer.example.com/.well-known/jwks.json" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Runtime Flow | ||
|
|
||
| The webhook runtime uses a single generic endpoint: | ||
|
|
||
| ```text | ||
| POST /webhooks/{configurationId} | ||
| ``` | ||
|
|
||
| The request flow is: | ||
|
|
||
| 1. IDP-Core receives the request on the generic webhook endpoint. | ||
| 2. The `configurationId` resolves the stored `WebhookConnector`. | ||
| 3. If the connector is disabled, IDP-Core ignores the event. | ||
| 4. The security dispatcher selects the matching strategy for the connector security type. | ||
| 5. The strategy validates the headers and, when needed, the raw request body. | ||
| 6. After authentication, the event is accepted for downstream processing. | ||
|
|
||
| > [!IMPORTANT] | ||
| > The connector model, security validation, management APIs, and mapping validation are implemented now. The final payload-to-entity ingestion route is still marked as pending Camel routing in the current handler implementation. | ||
|
|
||
| ## Management Lifecycle | ||
|
|
||
| You manage webhook connectors through the inbound webhook management API. | ||
|
|
||
| | Operation | Endpoint | | ||
| | --- | --- | | ||
| | Create connector | `POST /api/v1/inbound-webhooks` | | ||
| | List connectors | `GET /api/v1/inbound-webhooks` | | ||
| | Get connector | `GET /api/v1/inbound-webhooks/{identifier}` | | ||
| | Update connector | `PUT /api/v1/inbound-webhooks/{identifier}` | | ||
| | Delete connector | `DELETE /api/v1/inbound-webhooks/{identifier}` | | ||
|
|
||
| This separation keeps configuration management under versioned API routes while the event ingestion endpoint stays simple for external systems. | ||
|
|
||
| ## When to Use Webhooks | ||
|
|
||
| Use webhooks when an external system can push JSON events over HTTP and you want to: | ||
|
|
||
| - Ingest updates without redeploying IDP-Core | ||
| - Reuse one generic endpoint for multiple providers | ||
| - Apply connector-specific authentication rules | ||
| - Map external payloads to your own Entity Templates at runtime | ||
|
|
||
| --- | ||
|
|
||
| ## Next Steps | ||
|
|
||
| - **[Entity Templates](entity-templates.md)** - Define the target structures that mappings reference | ||
| - **[Entities](entities.md)** - Understand the records produced by successful ingestion | ||
| - **[Relations](relations.md)** - Model links that webhook mappings can populate | ||
| - **[Data Integration](../features/data-integration.md)** - Explore the broader ingestion roadmap |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
13 changes: 13 additions & 0 deletions
13
.../idp_core/domain/exception/entity_mapping/EntityDynamicMappingConfigurationException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| package com.decathlon.idp_core.domain.exception.entity_mapping; | ||
|
|
||
| public class EntityDynamicMappingConfigurationException extends RuntimeException { | ||
|
|
||
| public EntityDynamicMappingConfigurationException(String message) { | ||
| super(message); | ||
| } | ||
|
|
||
| public EntityDynamicMappingConfigurationException(String message, Throwable cause) { | ||
| super(message, cause); | ||
| } | ||
|
|
||
| } |
7 changes: 7 additions & 0 deletions
7
...main/exception/entity_template/PropertyNameNotFoundEntityTemplatePropertiesException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| package com.decathlon.idp_core.domain.exception.entity_template; | ||
|
|
||
| public class PropertyNameNotFoundEntityTemplatePropertiesException extends RuntimeException { | ||
| public PropertyNameNotFoundEntityTemplatePropertiesException(String message) { | ||
| super(message); | ||
| } | ||
| } |
7 changes: 7 additions & 0 deletions
7
...omain/exception/entity_template/RelationNameNotFoundEntityTemplateRelationsException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| package com.decathlon.idp_core.domain.exception.entity_template; | ||
|
|
||
| public class RelationNameNotFoundEntityTemplateRelationsException extends RuntimeException { | ||
| public RelationNameNotFoundEntityTemplateRelationsException(String message) { | ||
| super(message); | ||
| } | ||
| } |
11 changes: 11 additions & 0 deletions
11
.../java/com/decathlon/idp_core/domain/exception/webhook/WebhookAuthenticationException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| package com.decathlon.idp_core.domain.exception.webhook; | ||
|
|
||
| public class WebhookAuthenticationException extends RuntimeException { | ||
| public WebhookAuthenticationException(String message) { | ||
| super(message); | ||
| } | ||
|
|
||
| public WebhookAuthenticationException(String message, Throwable cause) { | ||
| super(message, cause); | ||
| } | ||
| } |
10 changes: 10 additions & 0 deletions
10
...om/decathlon/idp_core/domain/exception/webhook/WebhookConnectorAlreadyExistException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| package com.decathlon.idp_core.domain.exception.webhook; | ||
|
|
||
| import static com.decathlon.idp_core.domain.constant.ValidationMessages.WEBHOOK_CONNECTOR_ALREADY_EXIST; | ||
|
|
||
| public class WebhookConnectorAlreadyExistException extends RuntimeException { | ||
|
|
||
| public WebhookConnectorAlreadyExistException(String identifier) { | ||
| super(String.format("%s:%s", WEBHOOK_CONNECTOR_ALREADY_EXIST, identifier)); | ||
| } | ||
| } |
8 changes: 8 additions & 0 deletions
8
...va/com/decathlon/idp_core/domain/exception/webhook/WebhookConnectorNotFoundException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| package com.decathlon.idp_core.domain.exception.webhook; | ||
|
|
||
| public class WebhookConnectorNotFoundException extends RuntimeException { | ||
|
|
||
| public WebhookConnectorNotFoundException(String identifier) { | ||
| super(String.format("No webhook connector found for identifier: %s", identifier)); | ||
| } | ||
| } |
9 changes: 9 additions & 0 deletions
9
...athlon/idp_core/domain/exception/webhook/WebhookConnectorTitleAlreadyExistsException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| package com.decathlon.idp_core.domain.exception.webhook; | ||
|
|
||
| import static com.decathlon.idp_core.domain.constant.ValidationMessages.WEBHOOK_CONNECTOR_TITLE_ALREADY_EXIST; | ||
|
|
||
| public class WebhookConnectorTitleAlreadyExistsException extends RuntimeException { | ||
| public WebhookConnectorTitleAlreadyExistsException(String webhookName) { | ||
| super(String.format("%s:%s", WEBHOOK_CONNECTOR_TITLE_ALREADY_EXIST, webhookName)); | ||
| } | ||
| } |
8 changes: 8 additions & 0 deletions
8
...om/decathlon/idp_core/domain/exception/webhook/WebhookSecurityConfigurationException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| package com.decathlon.idp_core.domain.exception.webhook; | ||
|
|
||
| public class WebhookSecurityConfigurationException extends RuntimeException { | ||
|
|
||
| public WebhookSecurityConfigurationException(String message) { | ||
| super(message); | ||
| } | ||
| } |
8 changes: 8 additions & 0 deletions
8
.../decathlon/idp_core/domain/exception/webhook/WebhookTemplateHasNoPropertiesException.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| package com.decathlon.idp_core.domain.exception.webhook; | ||
|
|
||
| public class WebhookTemplateHasNoPropertiesException extends RuntimeException { | ||
|
|
||
| public WebhookTemplateHasNoPropertiesException(String message) { | ||
| super(message); | ||
| } | ||
| } |
24 changes: 24 additions & 0 deletions
24
src/main/java/com/decathlon/idp_core/domain/model/entity_mapping/EntityDynamicMapping.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| package com.decathlon.idp_core.domain.model.entity_mapping; | ||
|
|
||
| import java.util.Map; | ||
|
|
||
| import com.decathlon.idp_core.domain.model.webhook.WebhookConnector; | ||
| import jakarta.validation.constraints.NotBlank; | ||
| import jakarta.validation.constraints.NotNull; | ||
|
|
||
| public record | ||
| EntityDynamicMapping( | ||
| @NotBlank | ||
| String templateIdentifier, | ||
| @NotBlank | ||
| String filter, | ||
| @NotBlank | ||
| String entityIdentifier, | ||
| @NotBlank | ||
| String entityTitle, | ||
| @NotNull | ||
| Map<String, String> properties, | ||
| @NotNull | ||
| Map<String, String> relations | ||
| ) { | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Delete unused import