diff --git a/docs/extensions.md b/docs/extensions.md index 040c4d8..a5ca1de 100644 --- a/docs/extensions.md +++ b/docs/extensions.md @@ -1,59 +1,61 @@ # Shuffle extensions + This is documentation for integrating and sending data from third-party services to Shuffle, along with being a brief overview of our Research & Development (R&D) activities. Not to be confused with [apps](/apps) and [workflows](/workflows) ## Table of contents -* [Introduction](#introduction) -* [Single Signon (OpenID/SAML)](#single-signon-sso) - * [Okta](#okta) - * [Google SSO - SAML](#google-saml-sso) - * [Auth0](#auth0) - * [PingIdentity](#ping-id) - * [Keycloak - OpenID](#keycloak) - * [Azure AD - OpenID](#azure-ad) - * [Other SSO providers](#other) - * [Testing SSO](#sso-testing) - * [SSO Required for Org](#sso-required-for-org) - * [Auto Provisioning](#auto-provisioning) - * [Role Base Restriction](#role-base-restriction) - * [Skip SSO for admin in suborgs](#skip-sso-for-admin-in-suborgs) -* [Singul](#singul) -* [AI Agents](#ai-agents) -* [CACAO](#cacao) -* [Detection Manager](#detection-manager) -* [KMS](#KMS) -* [Webhooks](#webhooks) - * [Wazuh Webhook](#wazuh) - * [TheHive Webhook](#thehive) - * [Logz.io Webhook](#logzio) - * [MISP forwarder](#misp) - * [AWS S3 forwarder](#aws-s3-forwarder) - * [QRadar Webhook](#qradar) - * [FortiSIEM Webhook](#fortisiem) - * [ELK Webhook](#elk) - * [Cortex Webhook](#cortex) - * [Splunk Webhook](#splunk-siem) - * [Eventlog Analyzer - SIEM](#eventlog-analyzer) - * [ServicePilot - SIEM](#servicepilot-siem) -## Introduction -From the start, Shuffle has been a platform about integrations. We've focused on making them as open and usable as possible, but were missing one part; inbound data. The general way Shuffle handles this has been through third-party API's, where we poll for data on a schedule. There are however some cases where this doesn't do the trick. That's what extensions are. +- [Introduction](#introduction) +- [Single Signon (OpenID/SAML)](#single-signon-sso) + - [OpenID Connect requirements](#openid-connect-requirements) + - [Okta](#okta) + - [Google SSO - SAML](#google-saml-sso) + - [Auth0](#auth0) + - [PingIdentity](#ping-id) + - [Keycloak - OpenID](#keycloak) + - [Azure AD - OpenID](#azure-ad) + - [Other SSO providers](#other) + - [Testing SSO](#sso-testing) + - [SSO Required for Org](#sso-required-for-org) + - [Auto Provisioning](#auto-provisioning) + - [Role Base Restriction](#role-base-restriction) + - [Skip SSO for admin in suborgs](#skip-sso-for-admin-in-suborgs) +- [Singul](#singul) +- [AI Agents](#ai-agents) +- [CACAO](#cacao) +- [Detection Manager](#detection-manager) +- [KMS](#KMS) +- [Webhooks](#webhooks) + - [Wazuh Webhook](#wazuh) + - [TheHive Webhook](#thehive) + - [Logz.io Webhook](#logzio) + - [MISP forwarder](#misp) + - [AWS S3 forwarder](#aws-s3-forwarder) + - [QRadar Webhook](#qradar) + - [FortiSIEM Webhook](#fortisiem) + - [ELK Webhook](#elk) + - [Cortex Webhook](#cortex) + - [Splunk Webhook](#splunk-siem) + - [Eventlog Analyzer - SIEM](#eventlog-analyzer) + - [ServicePilot - SIEM](#servicepilot-siem) -These integrations will typically entail third party services connecting to Shuffle with inbound Webhooks as triggers in a workflow. +## Introduction -## Single Signon -Shuffle added Single Signon (SAML v1.0) from Shuffle version 0.9.16 & OpenID in Shuffle version 1.0.0. This allows you to log into Shuffle from other identity platforms, entirely controlled by you. SSO is available for **onprem AND cloud**. It works by setting an Entrypoint (IdP) and X509 Certificate, both used to validate who the requester is. This can be added in [your admin panel](/admin). +From the start, Shuffle has been a platform about integrations. We've focused on making them as open and usable as possible, but were missing one part; inbound data. The general way Shuffle handles this has been through third-party API's, where we poll for data on a schedule. There are however some cases where this doesn't do the trick. That's what extensions are. -Note: This feature will be depreciated on cloud by 10th December 2025. Please migrate to OAuth2/OpenID Connect as soon as possible. +These integrations will typically entail third party services connecting to Shuffle with inbound Webhooks as triggers in a workflow. ### Org Swapping behavior + If an Organization requires SSO, it will FORCE you through the SSO login unless your session already has been through SSO for that organization. This may feel counter-intuitive at first, but is a required system as each organization is controlled for SSO individually, and there is no limit to how many organizations a user can have. -### Using ANY SSO platform -**ONPREM ONLY:** You will have to change the SSO_REDIRECT_URL variable in the .env file to match your front end server link i.e `SSO_REDIRECT_URL=http://:` +### Using ANY SSO platform + +**ONPREM ONLY:** You will have to change the SSO_REDIRECT_URL variable in the .env file to match your front end server link i.e `SSO_REDIRECT_URL=http://:` ![Single Signon button](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-3.png?raw=true) **How it works:** + - Every user gets the role "user" by default. - If the user DOESN'T exist, it will be created, based on an identifier provided from the SSO provider. - If the user DOES exist, it will simply log them in. @@ -62,32 +64,67 @@ If an Organization requires SSO, it will FORCE you through the SSO login unless **PS**: In some cases, the persons' username may be appear an ID in Shuffle. If so, an admin should login to Shuffle and change their username. SSO should still work. **PPS**: The callback URL/redirect URL/recipient URL is `https://:/api/v1/login_sso` +### OpenID Connect requirements + +Shuffle supports OpenID Connect (OIDC) login with the **Authorization Code Flow with PKCE**. The OIDC server must support: + +- `response_type=code` +- `code_challenge_method=S256` +- The `openid` and `email` scopes +- A signed `id_token` returned from the token endpoint +- Provider discovery and JWKS for token signature verification +- A stable issuer URL derived from the configured authorization or token endpoint + +Shuffle generates a high-entropy PKCE verifier, sends the SHA-256 based `S256` code challenge in the authorization URL, stores the verifier server-side, and sends the original verifier to the token endpoint during callback handling. Shuffle does **not** support implicit flow, hybrid flow, `response_type=id_token`, or `response_mode=form_post` for OIDC login. + +Configure the OIDC client with this redirect URI: + +- Cloud: `https://shuffler.io/api/v1/login_openid` +- On-prem: `https:///api/v1/login_openid` + +The client may be public or confidential. If no client secret is configured, Shuffle performs a public-client authorization code exchange protected by PKCE and does not send a `client_secret` parameter. If the OIDC server requires confidential-client authentication, configure the client secret in Shuffle; Shuffle uses it only server-to-server at the token endpoint and does not send it in the browser authorization URL. The ID token must contain a `sub` claim and should contain an `email` claim. Shuffle can read roles from `roles`, `groups`, or Keycloak-style `realm_access.roles`; supported Shuffle role names are `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader`. + +Supported OIDC servers include any standards-compliant provider with Authorization Code Flow and PKCE S256 support. Common examples: + +| Type | Examples | +| ----------- | --------------------------------------------------------------------------------------------------------------- | +| Self-hosted | Keycloak, Authentik, Dex, Zitadel, Ory Hydra, Authelia | +| Cloud | Okta, Microsoft Entra ID / Azure AD, Auth0, Google Workspace / Cloud Identity, PingOne, OneLogin, Zitadel Cloud | + +For multi-replica Shuffle backend deployments, configure `SHUFFLE_MEMCACHED` for every backend replica. OIDC login state is created before the browser redirect and consumed when the IdP calls back; without shared Memcached, the callback may reach a different backend replica and fail with invalid OpenID state. Single-replica deployments do not require Memcached for OIDC, and sticky sessions can avoid the symptom, but shared Memcached is the recommended configuration for multi-replica backends. + ![Single signon configuration](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-1.png?raw=true) ### Okta -To use Okta SSO with Shuffle, first make an app in Okta. Their guide for making an app [can be found here](https://developer.okta.com/docs/guides/build-sso-integration/saml2/create-your-app/). + +To use Okta SSO with Shuffle, first make an app in Okta. Their guide for making an app [can be found here](https://developer.okta.com/docs/guides/build-sso-integration/saml2/create-your-app/). Once an application is made, it's time to find the required information. Go to the app > Sign On > Click "View Setup Instructions" under SAML 2.0. This will open a new page with the credentials. ![Okta SSO setup](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-2.png?raw=true) **Move these fields over to Shuffle:** -* Identity Provider Single Sign-On URL -> SSO Entrypoint (IdP) -* X.509 -> SSO Certificate (X509) + +- Identity Provider Single Sign-On URL -> SSO Entrypoint (IdP) +- X.509 -> SSO Certificate (X509) After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization. ### Okta OpenID + To use Okta OpenID with Shuffle, first make an app in Okta. **Configure the following in Okta:** -* Sign-in redirect URIs -> `https://shuffler.io/api/v1/login_openid` -* Initiate login URI -> `https://shuffler.io/api/v1/login_openid` -* Under Client Credentials, enable "Require PKCE as additional verification" -* Under Grant type, enable "Implicit (hybrid)" and "Allow ID Token with implicit grant type" + +- Sign-in redirect URIs -> `https://shuffler.io/api/v1/login_openid` +- Initiate login URI -> `https://shuffler.io/api/v1/login_openid` +- Under Client Credentials, enable "Require PKCE as additional verification" with `S256` +- Under Grant type, enable "Authorization Code" +- Do not enable implicit or hybrid ID Token grant types for Shuffle OIDC login After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. ### Google SAML SSO + • Log in to your Google Workspace portal as admin -> Apps -> Web and mobile apps. • Click Add app -> Add custom SAML app. @@ -98,76 +135,82 @@ After adding them, click "Save", saving the configuration. After saving, log out • Next page will be the SP information, this is where you should provide the Single Sign On URL, and SP Entity ID. - - ACS URL : - - For Cloud : https://shuffler.io/api/v1/login_sso - - For On-prem: https://:3443/api/v1/login_sso - - Entity ID : shuffle-saml - - Name ID: Basic Information > Primary Email - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/7958a55b-c932-4b5c-8910-dfb42703b695) + - ACS URL : + +- For Cloud : https://shuffler.io/api/v1/login_sso +- For On-prem: https://:3443/api/v1/login_sso +- Entity ID : shuffle-saml +- Name ID: Basic Information > Primary Email + ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/7958a55b-c932-4b5c-8910-dfb42703b695) - • The next page is where you add the below attributes and click on Finish and the app is added to the Google Workspace. - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/2bd0c217-de1c-41a4-82b9-7672ed2a4984) +• The next page is where you add the below attributes and click on Finish and the app is added to the Google Workspace. +![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/2bd0c217-de1c-41a4-82b9-7672ed2a4984) - • After completing the app creation on Google Workspace. You can select the shuffle app under the "Web and mobile apps". Click on the shuffle app, and note - the SPID in the URL, that is the SPID that will be needed. +• After completing the app creation on Google Workspace. You can select the shuffle app under the "Web and mobile apps". Click on the shuffle app, and note +the SPID in the URL, that is the SPID that will be needed. - ![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/cfd1fc98-8efe-4aa2-a4af-4ea60d10a3d6) +![image](https://github.com/yogeshgurjar127/Shuffle-docs/assets/118437260/cfd1fc98-8efe-4aa2-a4af-4ea60d10a3d6) - • Now we configure the Google SAML SSO in shuffle portal. Log in to your Shuffle web portal. - - • Click on profile -> Admin --> Edit Details --> Now add the Entrypoint (IdP) and SSO Certificate under the SAML SSO and click on save. +• Now we configure the Google SAML SSO in shuffle portal. Log in to your Shuffle web portal. + +• Click on profile -> Admin --> Edit Details --> Now add the Entrypoint (IdP) and SSO Certificate under the SAML SSO and click on save. + +**Note : The IdP metadata contains the Idp ID and certificate. Use this link as your SSO entry point, substituting your SP ID for "YourSPID" and your IDP ID for "YourIDPId."** - **Note : The IdP metadata contains the Idp ID and certificate. Use this link as your SSO entry point, substituting your SP ID for "YourSPID" and your IDP ID for "YourIDPId."** - - SSO Entrypoint (IdP) : https://accounts.google.com/o/saml2/initsso?idpid=C0ivw&spid=634181550209&forceauthn=false -### Auth0 +### Auth0 + To use Auth0 SSO with Shuffle, first make an app on [https://manage.auth0.com](https://manage.auth0.com). Documentation can be found [here](https://auth0.com/docs/configure/saml-configuration/configure-auth0-saml-identity-provider). -After the app is made, click "Addons" > "SAML2 Web App". +After the app is made, click "Addons" > "SAML2 Web App". ![Auth0 SSO setup](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-4.png?raw=true) **In the popup, move the data of these fields to Shuffle:** -* Identity Provider Login URL -> SSO Entrypoint (IdP) -* Identity Provider Certificate -> SSO Certificate (X509) + +- Identity Provider Login URL -> SSO Entrypoint (IdP) +- Identity Provider Certificate -> SSO Certificate (X509) Open the Certificate file in a text editor, and copy it's contents. After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization. -### PingIdentity +### PingIdentity + To use PingID SSO with Shuffle, first make an app on [https://console.pingone.eu/](https://docs.pingidentity.com/bundle/pingoneforenterprise/page/xsh1564020480660-1.html). Documentation can be found [here](https://docs.pingidentity.com/bundle/pingoneforenterprise/page/xsh1564020480660-1.html). After the app is made, click the dropdown for it on the right side > Configuration > find these fields. ![PingID SSO setup Shuffle](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-5.png?raw=true) **In the view above, move the data of these fields to Shuffle:** -* INITIATE SINGLE SIGN-ON URL -> SSO Entrypoint (IdP) -* DOWNLOAD METADATA -> SSO Certificate (X509) + +- INITIATE SINGLE SIGN-ON URL -> SSO Entrypoint (IdP) +- DOWNLOAD METADATA -> SSO Certificate (X509) Open the Certificate file in a text editor, and copy it's contents in the field. After adding them, click "Save", saving the configuration. After saving, log out of your user to verify the SSO configuration. If you don't see a button for "Use SSO", you most likely configured the wrong organization. ### Keycloak -Open ID SSO setup with Keycloak: -- Click on clients -- Create new client -- Set client id -- Ensure client protocol is openid-connect -- Hit save - -You now have a client. Click on settings and configure them as follows -- Your client ID -- Set up a name -- Client Protocol set to openid-connect -- Access Type set to public -- Standard Flow Enabled toggled to ON -- Direct access grants toggled to ON + +OpenID Connect SSO setup with Keycloak: + +- Click **Clients** +- Create a new client +- Set **Client type** / **Client protocol** to `openid-connect` +- Set the client ID you want Shuffle to use +- Enable **Standard flow** +- Disable **Implicit flow**. Shuffle does not support implicit or `form_post` ID token login. +- Disable **Direct access grants** unless you need them for another application. +- Enable **Proof Key for Code Exchange (PKCE)** with `S256` when your Keycloak version exposes this option. +- If you use a confidential Keycloak client, enable client authentication and copy the client secret into Shuffle. Shuffle only sends the secret server-to-server during the token exchange. ![Valid redirect URI](https://user-images.githubusercontent.com/31187099/162692070-ccc54692-6793-4331-adc7-2356d0fd5397.jpg) -For the Valid Redirect URI https://shuffler.io/api/v1/login_openid +For the valid redirect URI, use your Shuffle callback URL: + +- Cloud: `https://shuffler.io/api/v1/login_openid` +- On-prem: `https:///api/v1/login_openid` - Backchannel Logout URL https://shuffler.io/* - Backchannel Logout Session toggled to ON @@ -179,16 +222,21 @@ Valid Request URIs https://shuffler.io/login?autologin=true ![Auto login = true](https://user-images.githubusercontent.com/31187099/162692186-6ac71b93-01be-4cb1-83f4-b7cb17758378.jpg?raw=true) Once this is done head over to your shuffle instance. + 1. Click on Admin button 2. Scroll down and click on the downward facing arrow beneath Organization overview 3. Scroll down again to OpenID connect 4. Fill in the client ID (It should be the same as what you entered in Keycloak) -5. Authorization URL http://:/auth/realms/openid/protocol/openid-connect/auth -6. Token URL http://:/auth/realms/openid/protocol/openid-connect/token +5. If the Keycloak client is confidential, fill in the client secret. +6. Authorization URL `https:///realms//protocol/openid-connect/auth` +7. Token URL `https:///realms//protocol/openid-connect/token` + +Older Keycloak deployments may include `/auth` before `/realms/...`; use the URLs shown in your realm's OpenID endpoint configuration. ![OIDC shuffle-side](https://user-images.githubusercontent.com/31187099/162689392-51fcb2e9-3d89-4066-8b99-6074065f9c2a.png?raw=true) If you keep getting redirected to your backend url, head on to your Shuffle folder on your server. + 1. Vim .env ![vim env](https://user-images.githubusercontent.com/31187099/164943744-981638cf-6149-42e1-a455-3d12927ec24c.png) @@ -199,67 +247,108 @@ If you keep getting redirected to your backend url, head on to your Shuffle fold Finally go back to shuffle and use SSO button to login. +If you run multiple Shuffle backend replicas, configure `SHUFFLE_MEMCACHED` so the OIDC login state created before the redirect is available to the backend replica that receives the callback. Without shared cache, use sticky sessions; otherwise callbacks may fail with invalid OpenID state. + ![shuffle SSO](https://user-images.githubusercontent.com/31187099/162689445-8db0766c-6f18-4463-8a92-f6ae62213918.png?raw=true) ### Assigning a Role from Keycloak to Shuffle for a New User -If you want to assign a Shuffle organization role (`admin`, `user`, `org-reader`) from your Keycloak client, you can achieve this using the following method: +If you want to assign a Shuffle organization role (`admin`, `user`, `org-reader`) from your Keycloak client, you can achieve this using **client roles** and a **dedicated protocol mapper** that includes the roles in the ID token. -Steps to Assign Roles +> **Important:** Shuffle reads roles exclusively from the **ID token**, not the access token. Without a mapper configured to add roles to the ID token, Shuffle will not see any role assignments. -1. In your Keycloak client, create three new roles with the names `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader`, as shown in the image below: +#### Step 1: Create Client Roles - ![image](https://github.com/user-attachments/assets/e8a1f344-73c9-453d-b119-aef4643610b4) +In Keycloak, navigate to your Shuffle client and create three roles: -2. After creating the roles, assign them to the users you want. The `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader` roles in Keycloak correspond to the `admin`, `user`, and `org-reader` roles in Shuffle. +**Clients** → *your-shuffle-client* → **Roles** → **Create role** -3. Once the roles are assigned to users, navigate to: +Create the following roles (names must match exactly): - **Client Scopes** → **Roles** → **Mapper** → **Client Roles** +| Keycloak Client Role | Shuffle Role | +|---------------------|--------------| +| `shuffle-admin` | admin | +| `shuffle-user` | user | +| `shuffle-org-reader` | org-reader | - On the **Client Roles** page: +![image](https://github.com/user-attachments/assets/e8a1f344-73c9-453d-b119-aef4643610b4) - - Update the `Token Claim Name` to `roles` - - Enable the option **"Add to ID Token"** so that the roles are included in the response. +#### Step 2: Assign Roles to Users - ![image](https://github.com/user-attachments/assets/09582542-efa1-429b-9d45-e0a3796c6fbd) +Assign one of the client roles to each user who should access Shuffle: - Important Notes: +**Users** → *select user* → **Role mapping** → **Assign role** → Filter by *clients* → select the appropriate `shuffle-*` role -- After completing these settings, if valide role is present in roles claim during sso login then that user will be assign role from sso provider. If no role is assigned, the default role will be `user`. -- You can achieve this behavior starting from Shuffle version 2.0.1 or later. Make sure you are using this version or a newer one +#### Step 3: Add a Protocol Mapper to the Dedicated Client Scope - +Each Keycloak client has a dedicated scope (named `-dedicated`) that is automatically assigned to the client. Add a mapper here to include client roles in the ID token: + +1. Navigate to: **Client Scopes** → **`-dedicated`** → **Mappers** → **Configure a new mapper** (or **Add mapper** → **By configuration**) +2. Select mapper type: **User Client Role** +3. Configure the mapper: + - **Name**: `shuffle-roles` (or any descriptive name) + - **Client ID**: Select your Shuffle client + - **Token Claim Name**: `roles` + - **Claim JSON Type**: `String` + - **Add to ID token**: ✅ **ON** (required) + - **Add to access token**: Optional (not used by Shuffle) + - **Add to userinfo**: Optional (not used by Shuffle) + - **Multivalued**: ✅ **ON** +4. Click **Save** + +![image](https://github.com/user-attachments/assets/09582542-efa1-429b-9d45-e0a3796c6fbd) + +#### Verifying the Configuration + +You can verify that roles appear in the ID token by using Keycloak's built-in token evaluation: + +**Clients** → *your-shuffle-client* → **Client Scopes** → **Evaluate** → select a user → **Generated ID token** + +The token should contain a `roles` claim with the assigned client role, for example: + +```json +{ + "roles": ["shuffle-admin"] +} +``` + +#### Important Notes + +- Shuffle reads roles from three possible claims in the ID token: `roles`, `groups`, and `realm_access.roles`. The recommended approach above uses the `roles` claim via client roles. +- Shuffle updates the user's organization role on every OpenID login based on the verified ID token. If a user previously had `shuffle-admin` but the provider later sends `shuffle-user`, `shuffle-org-reader`, or no Shuffle role, Shuffle revokes the old admin role and applies the current provider role. If no Shuffle role is assigned and role-required mode is off, the default role is `user`. +- If **"Restrict login to users with a valid SSO role"** is enabled in Shuffle's SSO settings, users without one of the three `shuffle-*` roles in their ID token will be denied login entirely. +- You can achieve this behavior starting from Shuffle version 2.0.1 or later. Make sure you are using this version or a newer one ### Azure AD -To use OpenID with Azure AD, Shuffle supports OpenID connect with the use of Client IDs and Client secrets. To set up OpenID Connect with Azure, we use "ID_token" authentication. This entails a few normal steps regarding app creation in Azure App Registration. -1. Set up an app in Azure AD with ID tokens enabled -Go to [app registrations and create a new app](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade). Make it use "Web" for redirect URI's and direct it to your Shuffle instance at /api/v1/login_openid. From here, make sure to go to "authentication" and enable "ID Tokens" -![image](https://user-images.githubusercontent.com/5719530/169712651-24f481bc-7d90-4b09-b25f-cd1ec6c52c20.png) +To use OpenID with Azure AD, Shuffle supports OpenID Connect with authorization code flow, PKCE `S256`, Client IDs and Client secrets. This entails a few normal steps regarding app creation in Azure App Registration. + +1. Set up an app in Azure AD with authorization code flow + Go to [app registrations and create a new app](https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade). Make it use "Web" for redirect URI's and direct it to your Shuffle instance at `/api/v1/login_openid`. Do not enable implicit grant ID tokens for Shuffle OIDC login. + ![image](https://user-images.githubusercontent.com/5719530/169712651-24f481bc-7d90-4b09-b25f-cd1ec6c52c20.png) 2. Get the Client ID, Client Secret and your Tenant -Start by generating a client secret. Keep it safe, as we'll use it later. -![image](https://user-images.githubusercontent.com/5719530/169712756-a07ae08f-75a6-4174-8c8a-26700fd48654.png) + Start by generating a client secret. Keep it safe, as we'll use it later. + ![image](https://user-images.githubusercontent.com/5719530/169712756-a07ae08f-75a6-4174-8c8a-26700fd48654.png) The client secret and tenant ID can be found in the "Overview" tab: ![image](https://user-images.githubusercontent.com/5719530/169712793-7173a2c5-3450-4a79-8f95-9363b940c293.png) 3. Go to the admin panel in Shuffle and put in the Client ID and Secret -![image](https://user-images.githubusercontent.com/5719530/169712842-83db0618-d1f2-4bcd-ad78-5276f0c0696a.png) + ![image](https://user-images.githubusercontent.com/5719530/169712842-83db0618-d1f2-4bcd-ad78-5276f0c0696a.png) 4. Put in your Tenant ID in the authorization URL -The URL is as such: `https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize`. The Token URL is not strictly required for ID Token auth. -![image](https://user-images.githubusercontent.com/5719530/169712857-fe49855e-da3a-4dca-9fde-59bdc60eaa4d.png) + The Authorization URL is `https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/authorize`. The Token URL is required for authorization code exchange: `https://login.microsoftonline.com/TENANT_ID/oauth2/v2.0/token`. + ![image](https://user-images.githubusercontent.com/5719530/169712857-fe49855e-da3a-4dca-9fde-59bdc60eaa4d.png) 5. In the "Token Configuration" tab on the left, click "Add Optional Claim." Select the token type as "ID" and choose "email" as the claim, then click "Add." -![token_configuration](https://github.com/user-attachments/assets/2f3b62b9-23a6-4404-a6c6-2dfd796a0b2d) - -7. Done! Click save and log out. Try your new login based on your Azure AD configuration. - + ![token_configuration](https://github.com/user-attachments/assets/2f3b62b9-23a6-4404-a6c6-2dfd796a0b2d) +6. Done! Click save and log out. Try your new login based on your Azure AD configuration. + PS: When a user signs in, they are granted the access rights of a "user" within the designated organization. The username will be derived from the email address listed in the Azure users list. Therefore, when creating or adding a new user in Azure, ensure that the email field in their profile is populated. If this email field is empty, the user will not be able to log in to the organization. - + ### Other Platforms + As long as you can create an identity and acquire an Entrypoint (IdP) and X509, paste them into the Shuffle fields, and it should work with any SAML/SSO provider. ### Testing SSO @@ -276,111 +365,123 @@ https://github.com/user-attachments/assets/0a927283-d39e-4200-8ba3-654ef6f1b9c1 **Note**: Ensure that the "email" field is included in the SSO response from your SSO provider. If this field is empty, you may encounter errors. The email from your SSO provider will be assigned as the username in Shuffle. +### How to Assign a Role to a New User from an SSO Provider (OpenID Connect) in Shuffle +If you want to assign a role to a user from an SSO provider, you can do so using the following method: -### How to Assign a Role to a New User from an SSO Provider (OpenID Connect) in Shuffle - -If you want to assign a role to a user from an SSO provider, you can do so using the following method: - -1. Create three roles in your SSO application/client: - - `shuffle-admin` - - `shuffle-user` - - `shuffle-org-reader` +1. Create three roles in your SSO application/client: + - `shuffle-admin` + - `shuffle-user` + - `shuffle-org-reader` -2. Assign one of these roles to new users when granting them access to the application/client. - - The `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader` roles in your SSO provider correspond to the `admin`, `user`, and `org-reader` roles in Shuffle. +2. Assign one of these roles to new users when granting them access to the application/client. + - The `shuffle-admin`, `shuffle-user`, and `shuffle-org-reader` roles in your SSO provider correspond to the `admin`, `user`, and `org-reader` roles in Shuffle. -3. Ensure that the `roles` claim is included in the **ID token** response from your SSO provider. +3. Ensure that the `roles` claim is included in the **ID token** response from your SSO provider. -- If none of the specified roles are assigned, or if the `roles` claim is missing, the user will be assigned the default role: **`user`** in Shuffle. +- If none of the specified roles are assigned, or if the `roles` claim is missing, the user will be assigned the default role: **`user`** in Shuffle. -Important Notes +Important Notes -1. Currently, this feature is only available for **OpenID Connect**. Support for **SAML SSO** may be added in the future. -2. After completing these settings, if valide role is present in roles claim during sso login then that user will be assign role from sso provider. If no role is assigned, the default role will be `user` for new users. +1. Currently, this feature is only available for **OpenID Connect**. Support for **SAML SSO** may be added in the future. +2. After completing these settings, if valide role is present in roles claim during sso login then that user will be assign role from sso provider. If no role is assigned, the default role will be `user` for new users. 3. You can achieve this behavior starting from Shuffle version 2.0.1 or later. Make sure you are using this version or a newer one. If you have any questions or need further assistance, please feel free to reach out to us at **[support@shuffler.io](mailto:support@shuffler.io)**. - ### SSO Required for Org + In Shuffle, user can configure whether Single Sign-On (SSO) is **required** or **optional** for an organization. By default, SSO is optional. However, when the **"Require SSO"** option is enabled from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel, all users within that organization must log in through SSO. If SSO is required for an organization and a user logs in using their username and password (not SSO), then switches to a non-SSO organization, they can access it without SSO. However, if they later switch back to the SSO-required organization and the session has changed or expired, they will be required to authenticate via SSO again. ### Auto Provisioning + In Shuffle, users can configure whether **auto-provisioning** is enabled or disabled for an organization. By default, auto-provisioning is **enabled**, meaning that if SSO is configured for the organization, a new user account will be automatically created using the email address provided by the SSO provider—even if the user does not already exist in the organization. However, when the **"Disable Auto Provisioning"** option is enabled from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel, new user accounts will **not** be created automatically. In this case, only users who already exist in the organization will be allowed to log in through SSO. ### Role-Based Restriction -In Shuffle, you can now [assign roles to users](https://shuffler.io/docs/extensions#how-to-assign-a-role-to-a-new-user-from-an-sso-provider-(openid-connect)-in-shuffle) directly from the SSO provider. This functionality is currently supported **only** for SSO providers using **OpenID Connect**. + +In Shuffle, you can now [assign roles to users]() directly from the SSO provider. This functionality is currently supported **only** for SSO providers using **OpenID Connect**. By default, if no valid role is passed from the SSO provider during login, the user is automatically assigned the **user** role. -If you want to restrict login for users who do not receive a valid role from the SSO provider, you can enable the **"Restrict user login if no valid role is assigned"** option from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel. When this option is enabled, only users with a valid role received from the SSO provider will be allowed to log in. +If you want to restrict login for users who do not receive a valid role from the SSO provider, you can enable the **"Restrict user login if no valid role is assigned"** option from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel. When this option is enabled, only users with a valid role received from the SSO provider will be allowed to log in. ### Skip SSO for Admins in Suborganizations + In Shuffle, you can now allow parent organization admins to **skip SSO login** when switching to suborganizations. By default, all users are required to authenticate via SSO when switching to a suborganization if SSO is required for that suborg and the user's session has changed or expired. However, by enabling the **"Skip SSO for Admin"** option from the [SSO tab](https://shuffler.io/admin?admin_tab=sso) in the admin panel, users with an **admin role in the parent organization** can bypass the SSO login when switching to a suborganization, even if SSO is required there and user session is changed or expired. ## Singul -Singul are a new way Shuffle interacts with data, built brick by brick since introducing Shuffle's Integration Layer API in early 2023. The goal of Singul is to enable ourselves and others to be able to perform actions towards a specific API, without necessarily know how to do it specifically for that system. + +Singul are a new way Shuffle interacts with data, built brick by brick since introducing Shuffle's Integration Layer API in early 2023. The goal of Singul is to enable ourselves and others to be able to perform actions towards a specific API, without necessarily know how to do it specifically for that system. As of early 2024, this system is in active development, and we will implement features with it and help third party platforms do the same throughout the next few years. **Example usecases:** + - Listing assets from your asset management system/CMDB to make a list of assets, without needing to know the Assets' API - Blocking an Endpoint without knowing how to use the EDR API. Add this as a button to the list from the previous usecase - Shuffle Notification Workflow: Get notifications directly to your ticketing system with minimal configuration image -The Singul system is based on [generative AI for automatic mapping of fields (Schemaless)](https://github.com/frikky/schemaless), uses Github to [store configurations (Standards)](https://github.com/shuffle/standards), and uses [Shuffle's Integration Layer API](https://shuffler.io/docs/API#integration-layer) to run the actions. +The Singul system is based on [generative AI for automatic mapping of fields (Schemaless)](https://github.com/frikky/schemaless), uses Github to [store configurations (Standards)](https://github.com/shuffle/standards), and uses [Shuffle's Integration Layer API](https://shuffler.io/docs/API#integration-layer) to run the actions. ## AI Agents + AI Agents are in active development as of April 2025. They take input(s) and automatically do the tasks for you, running in unison with [Singul](#singul) to perform tasks. This will be published to everyone as soon as it: + 1. Has an interface allowing interactivity 2. Works with local models + Shuffle AI inference (for open sourcing) -3. Has good reasoning capabilities to handle decisionmaking +3. Has good reasoning capabilities to handle decisionmaking image ## CACAO -CACAO is a standardization framework for playbooks. It is still lacking significantly in capabilities as compared to Shuffle's own workflow system, but may be a good way to handle interoperability. + +CACAO is a standardization framework for playbooks. It is still lacking significantly in capabilities as compared to Shuffle's own workflow system, but may be a good way to handle interoperability. The goals are as follows: + 1. ✅ Ensure [CACAO playbook imports work](https://github.com/shuffle/cacao) -3. Ensure Shuffle Workflow -> CACAO playbook exports work well +2. Ensure Shuffle Workflow -> CACAO playbook exports work well -Shuffle may in the future be based on CACAO playbooks to make workflow documentation work better. +Shuffle may in the future be based on CACAO playbooks to make workflow documentation work better. ## Detection Manager -The Shuffle Detection Manager is a system introduced in beta in December 2024, allowing Shuffle to work with platforms like Tenzir and other systems to help with Detection Engineering. The goal of the system is not to replace actual detection systems themselves, but to offer a centralized way to control Detection rules across tenants and different tools. As an example, **below is a focus on Sigma rules with Tenzir**. The system is tested with Yara rules, Email detection rules and custom rule systems. + +The Shuffle Detection Manager is a system introduced in beta in December 2024, allowing Shuffle to work with platforms like Tenzir and other systems to help with Detection Engineering. The goal of the system is not to replace actual detection systems themselves, but to offer a centralized way to control Detection rules across tenants and different tools. As an example, **below is a focus on Sigma rules with Tenzir**. The system is tested with Yara rules, Email detection rules and custom rule systems. ### Testing Tenzir + Sigma -1. **Rule Manager:** At least One Shuffle org -2. **Job Handler:** An Orborus instance running + +1. **Rule Manager:** At least One Shuffle org +2. **Job Handler:** An Orborus instance running 3. **Detection Handler:** A Tenzir instance running on the same server as Orborus **(no setup needed)** -4. **Log Forwarder:** Any system that can forward logs to Tenzir +4. **Log Forwarder:** Any system that can forward logs to Tenzir To test the Tenzir detection system, it is first important to ensure that your Orborus instance is attached to Shuffle, which can be found on the /admin?tab=Locations path. Below is a BAD instance, where Orborus both says "Stopped" AND Pipelines is crossed out. The first goal is to re-enable these. A bad instance that is not running ### Fixing the pipeline setup + To solve the pipeline issue shown in the previous image, we have to do two things: -1. Start Orborus and get it to the "Running" state. To do this, click the Location in question and copy the command to one of your servers. After this has been done, you should see a the Red "Stopped" part change to a Green "Running" box as in the image below. If this does not occur, reach out to support@shuffler.io. +1. Start Orborus and get it to the "Running" state. To do this, click the Location in question and copy the command to one of your servers. After this has been done, you should see a the Red "Stopped" part change to a Green "Running" box as in the image below. If this does not occur, reach out to support@shuffler.io. 2. Go to /detections/Sigma in the UI, and click "Connect" in the top-right corner. Refresh the page after a minute or so, and the Pipeline system should be showing as green on the [Location page](https://shuffler.io/admin?tab=Locations) and in the top-right corner of the [Detection page](https://shuffler.io/detections/Sigma). If it does not, please reach out to support@shuffler.io. image Tenzir setup configuration: -- **Adding a custom storage folder for Sigma rules:** Mount in the folder you want to control into the Orborus command. Then add the environment variable `SHUFFLE_STORAGE_FOLDER=/tmp/foldername` to Orborus. The default is `/tmp/`. + +- **Adding a custom storage folder for Sigma rules:** Mount in the folder you want to control into the Orborus command. Then add the environment variable `SHUFFLE_STORAGE_FOLDER=/tmp/foldername` to Orborus. The default is `/tmp/`. - **Connecting to an EXTERNAL Tenzir node:** Add the following environment variables to the Orborus command: `SHUFFLE_PIPELINE_URL=http://:5160`. Change the `` to your values (localhost does NOT work). This requires that [the web API is enabled](https://docs.tenzir.com/rest-api) on the node. ### Connecting to a remote Tenzir -If Tenzir is ran separately, follow these steps to set connect to it from Shuffle. The first step is to enable webserver mode. This is for systemd. + +If Tenzir is ran separately, follow these steps to set connect to it from Shuffle. The first step is to enable webserver mode. This is for systemd. 1. `sudo nano /etc/systemd/system/tenzir-node.service` 2. Find the line that says "ExecStart" and add `"--commands=web server --mode=dev --bind=0.0.0.0"` to the end of it @@ -391,10 +492,10 @@ It should look like this: `ExecStart=/opt/tenzir/bin/tenzir-node "--commands=web 4. `systemctl daemon-reload & systemctl restart tenzir-node` 5. Try to connect to it: `curl -XPOST http://localhost:5160/api/v0/ping` - - **Control the Shuffle Tenzir node from Tenzir Cloud**: Go to [Tenzir Cloud](https://app.tenzir.com) and create a node configuration. Download the configuration file, then add the variables found in it to the following environment variables to Orborus: `TENZIR_PLUGINS__PLATFORM__API_KEY=`, `TENZIR_PLUGINS__PLATFORM__CONTROL_ENDPOINT=`, `TENZIR_PLUGINS__PLATFORM__TENANT_ID=` ### Mounting the Tenzir Sigma location into Orborus + 1. You need to mount in the folder that is going to store the sigma rules, controlled from within Shuffle 2. The default location is /tmp/sigma_rules, so to mount it in, use `--volume "/tmp:/tmp"` in the Dockerfile. 3. If you end up changing the storage location, use the `SHUFFLE_STORAGE_FOLDER` environment variable with the full path. @@ -402,19 +503,21 @@ It should look like this: `ExecStart=/opt/tenzir/bin/tenzir-node "--commands=web image -5. Verify if the files actually exist on the server. By default, you can find the rules in the `/tmp/sigma_rules` folder (ls /tmp/sigma_rules). If the folder exists, then it worked. +5. Verify if the files actually exist on the server. By default, you can find the rules in the `/tmp/sigma_rules` folder (ls /tmp/sigma_rules). If the folder exists, then it worked. image **PS: This folder is deleted if rules are globally disabled in the UI.** ### Running the Tenzir Detection pipeline + With detection rules in the `/tmp/sigma_rules` folder, you can now test the Sigma rules directly. To do this, there are two parts: -1. Ingest some kind of logs. We suggest using [Syslog](https://docs.tenzir.com/integrations/syslog/), but almost data format you want is supported. You can enable this by going to the "Triggers" page in Shuffle: [/admin?tab=triggers](/admin?tab=triggers), then clicking `Deploy New Pipeline`, and running the `Syslog listener` pipeline. + +1. Ingest some kind of logs. We suggest using [Syslog](https://docs.tenzir.com/integrations/syslog/), but almost data format you want is supported. You can enable this by going to the "Triggers" page in Shuffle: [/admin?tab=triggers](/admin?tab=triggers), then clicking `Deploy New Pipeline`, and running the `Syslog listener` pipeline. image -It can take up to 60 seconds to deploy, and should look like this in the UI when ready to receive logs. +It can take up to 60 seconds to deploy, and should look like this in the UI when ready to receive logs. image @@ -425,40 +528,45 @@ It can take up to 60 seconds to deploy, and should look like this in the UI when [Sigma Pipelines details](https://docs.tenzir.com/tql2/operators/sigma) ### Running a sample Detection + To run a sample detection requires that the correct rules are in place in the /tmp/sigma_rules folder. 1. [Ensure that the detection pipeline is running](/admin?tab=triggers): -image + image 2. One of the test rules is called `notepad_test.yaml` and works as follows. This should already exist in your sigma_rules folder. + ``` detection: selection: - EventID: 4688 + EventID: 4688 NewProcessName: '*notepad.exe*' Context: Testing condition: selection ``` 3. This detection requires three events to match: EventID, NewProcessName & Context. To trigger this, you may run the following from any machine with connectivity to the Tenzir machine on port 1514. + ``` printf '<165>1 2025-10-06T12:34:56.789Z myhost.example.com myapp 1234 ID47 [huh eventSource="App" EventID="4688" NewProcessName="notepad.exe"] This is a test log message' | nc IP 1514` ``` -4. Find your detection workflow [here](http://localhost:3002/workflows?tab=background_processes). +4. Find your detection workflow [here](http://localhost:3002/workflows?tab=background_processes). image -PS: If you can't see whether it is triggering in the workflow, run this locally to see if it is being ingested: `tenzir 'export live=true'` +PS: If you can't see whether it is triggering in the workflow, run this locally to see if it is being ingested: `tenzir 'export live=true'` 5. Detections are live! Feel free to add or change them on [/detections/sigma](/detections/sigma). ### Storing Tenzir logs in Opensearch + Since we are already ingesting logs using the `import` mechanism, it means we in theory can just route those logs forward. [Tenzir -> Opensearch documentation](https://docs.tenzir.com/integrations/opensearch) Modify the following: + ``` export live=true | to_opensearch "localhost:9200", action="create", index="shuffle_logs", user="admin", passwd="PASSWORD"` ``` @@ -466,22 +574,26 @@ export live=true | to_opensearch "localhost:9200", action="create", index="shuff Additional [to_opensearch docs here](https://docs.tenzir.com/reference/operators/to_opensearch/). - ### Debugging -Running a pipeline manually. This example dumps database content (`export`) into the `/tmp/events.ndjson` file. + Running a pipeline manually. This example dumps database content (`export`) into the `/tmp/events.ndjson` file. + ``` curl http://localhost:5160/api/v0/pipeline/launch -H "Content-type: application/json" -d '{"definition": "export | write_ndjson | save_file \"/tmp/events.ndjson\""}' ``` Listing all pipelines + ``` curl -XPOST http://localhost:5160/api/v0/pipeline/list -H "Content-Type: application/json" ``` Delete a pipeline + ``` curl -XPOST http://localhost:5160/api/v0/pipeline/delete -H "Content-Type: application/json" -d '{"id":"ID"}' -v ``` #### Manually testing pipelines Tenzir: + Ensures pipelines act as they should, and you get a reasonable output ``` @@ -503,6 +615,7 @@ tenzir 'export | to "/var/lib/tenzir/output.json"' ``` #### Network Tests + In case the exports aren't working and you need to figure out what is going wrong. ``` @@ -523,30 +636,32 @@ tenzir '' ``` ## KMS -Shuffle by default allows you to store authentication tokens within Shuffle itself, which are encrypted in the database. Since February 2024, we additionally support the use of external KMS systems to handle authentication, which is based on [Native Actions](https://shuffler.io/docs/extensions#native-actions) and [Schemaless](https://github.com/frikky/schemaless). Native Actions run in the background to perform the "Get KMS key" action, and the run of the app is NOT stored. + +Shuffle by default allows you to store authentication tokens within Shuffle itself, which are encrypted in the database. Since February 2024, we additionally support the use of external KMS systems to handle authentication, which is based on [Native Actions](https://shuffler.io/docs/extensions#native-actions) and [Schemaless](https://github.com/frikky/schemaless). Native Actions run in the background to perform the "Get KMS key" action, and the run of the app is NOT stored. The Shuffle KMS system is built as a third party Key:Value provider. You can reference keys from the KMS in any field marked as "Authentication" in the UI, or from within a Shuffle authentication itself (meaning you can authenticate the authentication..). The way you reference the keys is path-based, starting with `kms/`. Requirements: -* Have an Authentication called **kms shuffle storage** in Shuffle. On the [Auth page](/admin?tab=app_auth), it should clearly stand out as a different type of authentication. -* The Authentication needs to be associated with an App in the IAM category -* The App needs to have an action labeled as "Get KMS key" -* If it's the FIRST translation, it may fail out without internet access to github.com +- Have an Authentication called **kms shuffle storage** in Shuffle. On the [Auth page](/admin?tab=app_auth), it should clearly stand out as a different type of authentication. +- The Authentication needs to be associated with an App in the IAM category +- The App needs to have an action labeled as "Get KMS key" +- If it's the FIRST translation, it may fail out without internet access to github.com When these requirements are fullfilled, you can do the following to use the KMS system: -* Find the required parameters for the action. The first image below shows the parameters IN ORDER for Hashicorp Cloud Platform Vault. -* Use the following format: `kms/field1/field2/field3/field4/...`. This NEEDS to start with `kms/` -* Make sure the Environment is correct. It uses your default environment to connect to the KMS if not otherwise specified on the [Auth page](/admin?tab=app_auth). -image +- Find the required parameters for the action. The first image below shows the parameters IN ORDER for Hashicorp Cloud Platform Vault. +- Use the following format: `kms/field1/field2/field3/field4/...`. This NEEDS to start with `kms/` +- Make sure the Environment is correct. It uses your default environment to connect to the KMS if not otherwise specified on the [Auth page](/admin?tab=app_auth). + image -* Example referencing the "username" in the app name "Jira": `kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username` -image +- Example referencing the "username" in the app name "Jira": `kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username` + image If all of this is fulfilled, you can run the workflow, and Shuffle will automatically reference the KMS correctly. **If it fails to authenticate**, you should see a Notification show up like in the following image. image ### Failure handling and Debugging + During the initial configuration of KMS, there are many things that CAN go wrong. After the setup is initially completed and it works, it should not fail again, meaning spending time to set it up properly the first time is well worth it. If you are using Shuffle onprem and have trouble with KMS translations, we suggest setting the environment variable `SHUFFLE_KMS_DEBUG=true` on the shuffle-backend container. This will allow you to test individual keys in ANY field, meaning you can e.g. print out the KMS value that it should translate to. This further means we will not be deleting the KMS execution, and you can see the execution on the [Workflow Runtime Debugger page.](/workflows/debug). **KMS debug mode is NOT enabled on shuffler.io**. @@ -554,13 +669,15 @@ If you are using Shuffle onprem and have trouble with KMS translations, we sugge image If your KMS translation fails, it is most likely due to network connectivity OR translation errors [for the standard](https://github.com/Shuffle/standards/blob/main/translation_standards/get_kms_key.json) in use. After running a translation, you should see the following three file categories in the admin panel: -- translation standards - The available translation standards -- translation input - The data FROM the kms used for translation. Does NOT contain values, only keys. -- translation output - The translation OUTPUT. Includes JSON paths to the correct keys + +- translation standards - The available translation standards +- translation input - The data FROM the kms used for translation. Does NOT contain values, only keys. +- translation output - The translation OUTPUT. Includes JSON paths to the correct keys IF your translation fails, the first area to look at is the "[translation output](/admin?tab=files&category=translation_output)" file category, as this is where the translation for the schema happens. The hash will match the data you have sent in, which can be found in the "translation input" folder. Documentation about schemaless explains [how translations happen](https://github.com/frikky/schemaless?tab=readme-ov-file#example), and how they can be fixed. If you need further help, contact support@shuffler.io A valid KMS translation file should be in the following format: + ``` { "kms_value": "path.in.kms.json" @@ -570,9 +687,10 @@ A valid KMS translation file should be in the following format: **The "kms_key" field is NOT relevant - the ONLY thing that is important is that the kms_value path is correct.** ### Using any KMS + KMS is supported for any system as long as the sections above are covered. It has been tested for the following: -- Microsoft Azure KMS +- Microsoft Azure KMS - Hashicorp Vault - Google Cloud Key Management - AWS Key Management Service @@ -582,16 +700,19 @@ KMS is supported for any system as long as the sections above are covered. It ha - ... and more! Ask if you need help. ## Inbound Webhooks + This section describes inbound webhooks to Shuffle, and how to set them up in many commonly used third-party systems. If your system support outbound Webhooks, it can also forward to Shuffle as a GET or POST request. [More about webhook triggers](/triggers/#webhooks) ### Wazuh -Wazuh is a SIEM platform for security operations. We've used it through their API multiple ways, but were missing an important component; alerting. That's why we've developed a simple alert forwarder from Wazuh to Shuffle. + +Wazuh is a SIEM platform for security operations. We've used it through their API multiple ways, but were missing an important component; alerting. That's why we've developed a simple alert forwarder from Wazuh to Shuffle. [Wazuh extension documentation](https://documentation.wazuh.com/current/user-manual/manager/manual-integration.html) **PS: If you expect more than 10 alerts per second, you should add multiple workflows AND webhook forwarders to Wazuh** These are the steps to set it up: + 1. Create a Workflow which will receive alerts 2. Add a Webhook to the Workflow 3. Configure Wazuh with the Webhook URL @@ -609,7 +730,7 @@ Copy the URL and keep it for the next steps ![Extend Shuffle with Wazuh 2](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_2.png?raw=true) **3. Configure Wazuh with the Webhook URL** -Start by logging into your Wazuh management console with access to edit the ossec.conf file. We'll first start by adding the Shuffle webhook forwarder. +Start by logging into your Wazuh management console with access to edit the ossec.conf file. We'll first start by adding the Shuffle webhook forwarder. 1. Transfer [these integration files](https://github.com/frikky/Shuffle/tree/master/functions/extensions/wazuh) to the server. Ignore the file named "ossec.conf" for now. 2. Move the custom-\* files to the location /var/ossec/integrations @@ -631,6 +752,7 @@ drwxr-x---. 19 root ossec 242 Dec 14 09:40 .. ``` 3. Change the files' ownership and access rights. This MAY be necessary if Wazuh doesn't have root privileges (it shouldn't) + ``` $ chown root:ossec custom-shuffle $ chown root:ossec custom-shuffle.py @@ -638,9 +760,9 @@ $ chmod 750 custom-shuffle $ chmod 750 custom-shuffle.py ``` -4. Configure ossec.conf to forward to Shuffle +4. Configure ossec.conf to forward to Shuffle -Go to the location /var/ossec/etc/ossec.conf (or where the ossec.conf file is). Take the information below (including ) and paste it in. Find the webhook URL from earlier, copy it and add it to the section. +Go to the location /var/ossec/etc/ossec.conf (or where the ossec.conf file is). Take the information below (including ) and paste it in. Find the webhook URL from earlier, copy it and add it to the section. Find more fields like [levels, groups and rule_id here](https://documentation.wazuh.com/current/user-manual/manager/manual-integration.html). @@ -654,11 +776,13 @@ Find more fields like [levels, groups and rule_id here](https://documentation.wa ``` 5. Restart ossec manager. Every time you change ossec.conf, you need to restart the ossec manager. + ``` systemctl restart wazuh-manager.service ``` You should now start seeing data sent from Wazuh into Shuffle which can be used. If data is NOT sent, make sure of a few things: + - If https in the Shuffle URL: is the certificate of Shuffle signed? Default: no. - Check /var/ossec/logs for logs: grep -R custom-shuffle @@ -668,6 +792,7 @@ There are many ways to test the integration, but you can simplify it by setting ![Extend Shuffle with Wazuh 3](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_3.png?raw=true) #### Running custom commands with Wazuh + Active response can be used with the command "Run Command" in the Wazuh app in Shuffle. This requires an Agent (Agent list) and Command ([Active response](https://documentation.wazuh.com/current/getting-started/use-cases/active-response.html) command). Below, we will show how to set up the custom command "reboot". - Wazuh API logs will be on the manager in: /var/ossec/logs/api.log @@ -677,6 +802,7 @@ Active response can be used with the command "Run Command" in the Wazuh app in S The goal with this section is to set up a bash script that can run custom commands from within Shuffle. 1. Log into the Wazuh agent of choice (Linux), and add the following script to the active-response folder. Give it the name "shuffle.sh". Full path: /var/ossec/active-response/bin/shuffle.sh PS: Once done, make sure it's executable: chmod +x /var/ossec/active-response/bin/shuffle.sh + ``` #!/bin/bash # Extra arguments @@ -685,8 +811,8 @@ The goal with this section is to set up a bash script that can run custom comman if ! command -v jq &> /dev/null then echo "jq could not be found - installing" - sudo apt install jq -y - sudo yum install jq -y + sudo apt install jq -y + sudo yum install jq -y fi @@ -695,19 +821,22 @@ CMD=$(echo $INPUT_JSON | jq -r .parameters.alert.cmd) CALLBACK=$(echo $INPUT_JSON | jq -r .parameters.alert.callback) OUTPUT=$($CMD) curl -XPOST $CALLBACK -d """$OUTPUT""" -k -``` +``` 2. Open /var/ossec/etc/shared/ar.conf and ADD the following line to the bottom of the file: + ``` shuffle - shuffle.sh - 0 ``` 3. Restart the wazuh agent + ``` -/var/ossec/bin/wazuh-control restart +/var/ossec/bin/wazuh-control restart ``` 4. Log into Shuffle and import [this public workflow](https://shuffler.io/workflows/44bc4e93-ba6e-4895-8294-424fd2a1d169). Make sure to change the following: + - Activate the Wazuh app if it's not already - Add a Wazuh username & password to the HTTP node - Add the Wazuh URL to the HTTP node, as well as Get_agents and other Wazuh nodes. @@ -716,11 +845,12 @@ shuffle - shuffle.sh - 0 The default from this workflow is that it will reboot the server. ### TheHive + TheHive is a case management platform for and by security professionals. One of their key capabilities is webhooks, which can send realtime updates to a third party system whenever ANYTHING is changed within TheHive (e.g. a new alert or a case task is written). Shuffle has an ideal way of handling this, [outlined in this blogpost (TheHive4)](https://medium.com/shuffle-automation/indicators-and-webhooks-with-thehive-cortex-and-misp-open-source-soar-part-4-f70cde942e59). **PS: There is a difference between TheHive3 and TheHive 4 on how to set this up. We are referring to TheHive4 in this section.** -1. Create a Workflow in Shuffle +1. Create a Workflow in Shuffle 2. Add a Webhook to the Workflow 3. Configure TheHive with the Webhook URL 4. Test the integration @@ -739,10 +869,12 @@ Copy the URL and keep it for the next steps **3. Configure TheHive with the Webhook URL** Configuring TheHive is the only unique step here, and is different between versions. Full documentation about [TheHive4 webhooks can be found here](https://github.com/TheHive-Project/TheHiveDocs/blob/master/TheHive4/Administration/Webhook.md). -As can be seen in the documentation, there are three steps to setting up TheHive webhooks: +As can be seen in the documentation, there are three steps to setting up TheHive webhooks: + 1. Define the webhook forwarder: Find the [application.conf](https://docs.thehive-project.org/thehive/installation-and-configuration/configuration/manage-configuration/) file and scroll to the webhook section. If it doesn't exist, add the following: + ``` notification.webhook.endpoints = [ { @@ -759,13 +891,15 @@ notification.webhook.endpoints = [ Modify it to have the URL you found making the webhook in Shuffle. You can also use https by adding certificate references to wsConfig{} 2. Restart TheHive + ``` systemctl restart THeHive ``` 3. Activate the webhook -Run this curl command (change the URL, username and password), which activates TheHive forwarding to Shuffle. -**PS: Make sure you have access to the organization you want the Webhook for** + Run this curl command (change the URL, username and password), which activates TheHive forwarding to Shuffle. + **PS: Make sure you have access to the organization you want the Webhook for** + ``` curl -XPUT -u thehive_user:thehive_password -H 'Content-type: application/json' thehive_url/api/config/organisation/notification -d ' { @@ -784,7 +918,8 @@ In TheHive UI (NOT CLI), create a new case, or add a comment to an existing case ![Extend Shuffle with TheHive 3](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_3.png?raw=true) -### Logzio +### Logzio + **1. Create a Workflow which will receive alerts** This one is pretty easily explained. Go to Shuffle an make a new Workflow. @@ -802,11 +937,13 @@ After logging into app.logz.io, hover Settings Icon > Settings > click Notificat ![Extend Shuffle with Logz.io](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_4.png?raw=true) Once in the notification endpoint view, click "Add Endpoint". Configure the following elements: -* Type: Custom -* Name: Shuffle (or some other identifier) -* URL: The Webhook URL from step 2 -* Method: POST -* Data (should be autofilled): + +- Type: Custom +- Name: Shuffle (or some other identifier) +- URL: The Webhook URL from step 2 +- Method: POST +- Data (should be autofilled): + ``` { "alert_title": "{{alert_title}}", @@ -824,6 +961,7 @@ Custom data will be parsed into the field "alert_event_samples" Click "Run the test" at the bottom before saving. This allows for it to send a sample payload to Shuffle. If the data payload is configured as in step 3, the custom data will be available as such: + ``` $exec.alert_event_samples ``` @@ -831,6 +969,7 @@ $exec.alert_event_samples ![Extend Shuffle with Logz.io 3](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_6.png?raw=true) ### MISP + MISP, short for Malware Information Sharing Platform, is one of the best Open Source alternatives for Threat Intelligence. For that reason, a lot of our users have wanted a way to handle data in realtime from MISP. What kind of data? Event updates, indicator updates, IDS flag edits, Organization edit etc. That's why we released an extension for Shuffle which can read ZMQ messages from MISP in realtime and send them to a webhook. @@ -838,14 +977,18 @@ That's why we released an extension for Shuffle which can read ZMQ messages from [More information here](https://www.circl.lu/doc/misp/misp-zmq/) **Steps to set it up:** + 1. Enable ZMQ in MISP by going to Server Settings -> Plugins in MISP. Make sure to enable the options for forwarding Events and Attributes. -2. Install pyzmq and redis on the MISP server: +2. Install pyzmq and redis on the MISP server: + ``` pip3 install pyzmq redis ``` + 3. Go to Shuffle and create a new Webhook 4. [Install Docker](https://docs.docker.com/get-docker/) and [docker-compose](https://docs.docker.com/compose/install/) on a host with access to ZMQ (localhost:50000 by default). This can be on the MISP server, but at scale, it preferably shouldn't be. 5. Copy the docker-compose [found here](https://github.com/frikky/Shuffle/tree/master/functions/extensions/misp/docker-compose.yml) and edit the environment variables to point to your ZMQ instance (PS: Don't use localhost, even if it's on the MISP server) + ``` version: '3' services: @@ -859,19 +1002,24 @@ services: - ZMQ_FORWARD_URL= restart: unless-stopped ``` + 6. Run the docker-compose! + ``` docker-compose up -d ``` + 7. Test it by going to your MISP instance and changing an event or attribute ### AWS S3 forwarder + Ever wanted to run an action as soon as something is uploaded to your S3 Bucket? Maybe you just want to build an S3 honeypot? If so, this part is for you. Herein we describe how you can get information about files being uploaded or changed from an S3 bucket into Shuffle. **Basic overview:** User -> Upload to S3 -> S3 triggers Lambda function -> Lambda function triggers Webhook -> File is downloaded from within Shuffle Steps to set it up: + 1. Create a Workflow with Webhook inside Shuffle 2. Create a Lambda function that triggers on S3 changes 3. Configure the workflow to download the file @@ -888,19 +1036,21 @@ Copy the URL and keep it for the next steps ![Extend Shuffle with webhook2](https://github.com/frikky/shuffle-docs/blob/master/assets/extensions_example_2.png?raw=true) **2. Create the lambda function** -- After logging into your AWS, go to [Lambda functions](https://console.aws.amazon.com/lambda/home) and click "Create function" in the top right corner. + +- After logging into your AWS, go to [Lambda functions](https://console.aws.amazon.com/lambda/home) and click "Create function" in the top right corner. - Use "Author from scratch", and type in a name like "Shuffle-forwarding" and make sure to choose Runtime as Python 3.8. Click "Create function" in the bottom right corner. -![Author S3 from scratch](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-2.png?raw=true) + ![Author S3 from scratch](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-2.png?raw=true) - Click "Add trigger" in the window, left of the function. In the next menu find "S3", before choosing the bucket you want and the "Event type". Click "Add trigger" -**PS: The bucket and cloud function have to be in the same location** -![Add trigger](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-3.png?raw=true) + **PS: The bucket and cloud function have to be in the same location** + ![Add trigger](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-3.png?raw=true) - Under Configuration > Environment variables, click "Edit". Add variable with key "SHUFFLE_WEBHOOK", and the value from step 1. Click "Save". -![Configure s3 environment](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-4.png?raw=true) + ![Configure s3 environment](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-4.png?raw=true) + +- Time to add some code. Go to the "code" tab and paste in the code below. Click "deploy". This should now forward the request to Shuffle. -- Time to add some code. Go to the "code" tab and paste in the code below. Click "deploy". This should now forward the request to Shuffle. ``` import json import urllib.parse @@ -910,15 +1060,15 @@ import os def lambda_handler(event, context): # Get the object from the event and show its content type #bucket = event['Records'][0]['s3']['bucket']['name'] - + webhook = os.environ.get("SHUFFLE_WEBHOOK") if not webhook: return "No webhook environment defined: SHUFFLE_WEBHOOK" - + ret = requests.post(webhook, json=event["Records"][0]) if ret.status_code != 200: return "Bad status code for webhook: %d" % ret.status_code - + print("Status code: %d\nData: %s" % (ret.status_code, ret.text)) ``` @@ -931,7 +1081,6 @@ Now all we need is to do is actually download the file. This requires access rig And that's it! All file updates should now come into Shuffle, including a way to download the file. - **4. Extra: Scan the file with Yara** In a lot of cases you would want to analyze the files somehow. But how..? Shuffle has a built in way to run Yara on files, and get results based on built-in rules (or your own). Most of the time, you would also want a response to that search, hence we've created a way to also delete the file if it matches too many Yara rules. ![Basic s3 analysis](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-7.png?raw=true) @@ -939,13 +1088,16 @@ In a lot of cases you would want to analyze the files somehow. But how..? Shuffl To do this, add in the "Yara" node with the action "Analyze file", and put "$get_s3_file.file_id" in the "File Id" field. This should match the file downloaded from the bucket. After this, add another S3 node, and choose the action "Delete file from bucket" with these parameters: + - Bucket Name: $exec.s3.bucket.name - Bucket Path: {{ "$exec.s3.object.key" | url_decode | replace: "+", " " }} -Last but not least, we need a way [to see IF](/docs/workflows#conditions) the file should be deleted. This can be done by clicking the branch between Yara and the S3 delete action > "New Condition" > with the following data: +Last but not least, we need a way [to see IF](/docs/workflows#conditions) the file should be deleted. This can be done by clicking the branch between Yara and the S3 delete action > "New Condition" > with the following data: + ``` {{ $run_yara.matches | size }} > 3 ``` + ![S3 condition yara](https://github.com/frikky/shuffle-docs/blob/master/assets/s3_function-8.png?raw=true) Done! Whenever a file is downloaded, it will be analyzed by Yara, checking matches, then removing the file if it's more than 3. @@ -955,7 +1107,9 @@ Done! Whenever a file is downloaded, it will be analyzed by Yara, checking match **QRadar send offense webhook to Shuffle.** #### STEP 1: Add the Script to QRadar & Config + > **Bash Code:** + ``` #!/bin/bash # Version 1.0.0 @@ -976,48 +1130,48 @@ EOF What you should do to use it: -1. Log In to QRadar; -2. Go to Admin > Custom Actions > **Define Actions**; +1. Log In to QRadar; +2. Go to Admin > Custom Actions > **Define Actions**; 3. Click **Add**; 4. Fill the Basic Information (name & description); -5. In the Script Configuration set **bash** as interpreter, and import the bash script attached to this message; -6.** For the Script Parameters, add the parameters in the following order:** +5. In the Script Configuration set **bash** as interpreter, and import the bash script attached to this message; 6.** For the Script Parameters, add the parameters in the following order:** ``` -1. [Fixed Property] **shuffle_url** - As value insert the Shuffle Webhook URI, for example https://shuffle.local/api/v1/hooks/webhook_15acc.... -2. [Fixed Property] **authorization** - Insert the Required headers (**SAuthorization**), for example in Shuffle webhook if you set **SAuthorization=s873hn872n_s298ns2-98ns2ns** in the Required headers you authorization value will be "s873hn872n_s298ns2-98ns2ns". *If you change the token name, don't forget to change it in the bash code too*. +1. [Fixed Property] **shuffle_url** - As value insert the Shuffle Webhook URI, for example https://shuffle.local/api/v1/hooks/webhook_15acc.... +2. [Fixed Property] **authorization** - Insert the Required headers (**SAuthorization**), for example in Shuffle webhook if you set **SAuthorization=s873hn872n_s298ns2-98ns2ns** in the Required headers you authorization value will be "s873hn872n_s298ns2-98ns2ns". *If you change the token name, don't forget to change it in the bash code too*. 3. [Network Event Property] **offense_id** - Insert **offense_id** as value ``` -7. Save +7. Save 8. Deploy Changes ![image](https://user-images.githubusercontent.com/21691729/152444978-d360680a-a0b1-40b0-bd04-fa7395cf4d85.png) ![image](https://user-images.githubusercontent.com/21691729/152445000-0d2a1828-ee1d-41b1-a549-5417c9a48c75.png) - #### STEP 2: Create new offense alert rule - -1. Go to Offenses > **Rules**; + +1. Go to Offenses > **Rules**; 2. Click Actions > **New Event Rule**: + ``` Rule Description Apply Shuffle new offense alert on events which are detected by the Local system and when the event QID is one of the following (28250369) Offense Created - + Rule Responses Execute Custom Action QRadar to Shuffle (previous added script) This Rule will be: Enabled ``` -#### QRadar Overview +#### QRadar Overview + - Easy, this way each time a new offense dispatches in QRadar it will send a webhook to Shuffle containing the offense_id, then the webhook node will receive this info and pass it to the next node (QRadar App) that will perform a **get offense** ~~data~~ action using the received offense id as key. - + This way you won't need to execute an api call every x time and save the last offense id. This is a better solution and improves the SLA, coz if you set 1 minute as you x time, you may have 1 minute delay or less, besides you are getting all "ungotten" offenses at once, when you can use shuffle to handle with multiple at the same time if it happen to dispatch more than one offense in QRadar at the same time or with just some seconds of difference. - ### FortiSIEM + FortiSiEM is the SIEM of Fortigate. It has the possibility of notifying Shuffle through a webhook when a rule triggers, which is exatly what this documentation section is for. The main caveat: all data is XML and needs to be transformed with the Shuffle Tools "XML to JSON" formatter. **1. Create a Workflow which will receive alerts** @@ -1033,19 +1187,15 @@ Copy the URL and keep it for the next steps **3. Configure FortiSIEM forwarding** Log into the UI. When inside, go to ADMIN > Settings > Incident Notification. In the "Incident HTTP Notification", paste in the webhook from the previous step. Click "save", then "Test". After the test, check the workflow in Shuffle whether it triggered. If it didn't, your FortiSIEM can't access the Shuffle instance. - ![image](https://user-images.githubusercontent.com/5719530/162085936-58a44de4-0289-4cf9-8f72-7b3229857448.png) - With the Notification Endpoint specified in the previous step, we need to decide what rules to add. By default, we add all of them. To do this, go to ADMIN > Settings > Notification Policy, and add a new policy. In here, select the "Send XML file over HTTP(S) to the destination set in...". This will make sure all alerts are sent to Shuffle. - ![image](https://user-images.githubusercontent.com/5719530/162086144-8bbfb6fa-d512-4c36-81db-f830bcf9c204.png) That's it! It's now time to wait for an alert to actually trigger. When it has, make sure to send Execution Argument from the webhook ($exec) straight into an XML to JSON parser. That way you can use it easily in Shuffle. - - -### Splunk SIEM -Splunk is a SIEM tool for security operations. There are multiple ways to forward the Splunk alerts to external systems. Simplest way to forward splunk alerts to Shuffle is with using webhook. +### Splunk SIEM + +Splunk is a SIEM tool for security operations. There are multiple ways to forward the Splunk alerts to external systems. Simplest way to forward splunk alerts to Shuffle is with using webhook. **Step 1:** First we'll have to create a Shuffle workflow which will recieve alerts from Splunk. Go to /workflows in Shuffle and create a new worfklow. Then, inside workflow editor drag in the webhook from the trigger section in left pane. @@ -1056,7 +1206,7 @@ Splunk is a SIEM tool for security operations. There are multiple ways to forwar **Now that we have webhook running in the Shuffle, Go to your Splunk deployment server and log in.** -**Step 3:** Now we'll have to configure Splunk with the webhook URL. Once logged in, go to the **search and reporting** app and type in the query you want to create an alert for. +**Step 3:** Now we'll have to configure Splunk with the webhook URL. Once logged in, go to the **search and reporting** app and type in the query you want to create an alert for. ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/splunk-alerts-3.png) **Step 4:** Save the search query as an alert. @@ -1070,70 +1220,66 @@ Splunk is a SIEM tool for security operations. There are multiple ways to forwar **You should now start seeing data sent from Splunk into Shuffle which can be used inside workflow for further actions.** - ### Eventlog Analyzer + Note: API integration is unfortunately not supported with EventLog Analyzer. However, If you would like to forward logs from EventLog Analyzer to Shuffle. For more information please follow this [guide](https://www.manageengine.com/products/eventlog/help/StandaloneManagedServer-UserGuide/Configurations/log-forwarder.html) - -### ServicePilot SIEM +### ServicePilot SIEM + ServicePilot is a high-performance analytics platform that supports observability and full-stack monitoring: metrics, traces and logs. You can collect data from many services and sources across your entire IT stack (ITIM, NPM, APM, DEM, SIEM) as well as view details of historical data stored by ServicePilot. Webhook integration service is not provided by the ServicePilot platform. Here am gonna mentioned the steps for how to use ServicePilot app workflow. #### - What this workflow do? + This workflow is help to get all the alerts, objects, events and logs. In this workflow there are two nodes of ServicePilot app. one is for getting all the records from the ServicePilot SIEM app and another one is searching for alerts, objects, events and logs for specific entry. As a result this workflow first excecute the Get_All_Data node and after that it excecute the Get_Specific_Record node and at the last return the result. #### - How the seaching happens? + For searching records from the ServicePilot app we need to write SQL query to search the data. we need to write query as a query parameter in the request url. #### - Steps to create a workflow + **Step 1:** First we'll have to create a Shuffle workflow which will search for events, objects, alerts, and logs from ServicePilot. Go to /workflows in Shuffle and create a new worfklow. Then, inside workflow editor drag in the node for the ServicePilot. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-0.0.png) **Step 2:** Click on the ServicePilot node. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-1.0.png) **Step 3:** Change the name of the node as Get_All_Data. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-1.1.png) **Step 4:** Write Query for searching all the datas from the ServicePilot. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-1.2.png) **Step 5:** Inside workflow editor drag in the another node for the ServicePilot and connect it with the existing node. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-2.0.png) **Step 6:** Change the name of the node as Get_Specific_Record. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-2.1.png) **Step 7:** Write Query for searching specific record from the ServicePilot. - ![image](https://github.com/shuffle/Shuffle-docs/blob/master/assets/servicepilot-2.2.png) **Finally, Click the excecution button and you should now start seeing data sent from ServicePilot into Shuffle which can be used inside workflow for further actions.** - - + ### ELK/Elastic + Intergrating Elastic with Shuffle will first require you to set up a webhook in Elastic. Below are the steps you will need to follow to ensure your alerts are forwaded into Shuffle immediately. 1. Create a new workflow in Shuffle, bottom left on your screen head to the triggers tab and drag in the webhook into your workflow. Click on it and ensure that the webhook is started. Copy the provided webhook URI and head over to Elastic. - - ![image](https://github.com/user-attachments/assets/e708d1fd-8222-4c47-ac29-7a2d5f62024e) - ![image](https://github.com/user-attachments/assets/98799857-7640-4cf7-aec4-31224a186320) +![image](https://github.com/user-attachments/assets/e708d1fd-8222-4c47-ac29-7a2d5f62024e) + +![image](https://github.com/user-attachments/assets/98799857-7640-4cf7-aec4-31224a186320) 3. Follow Elastic's documentation on setting up the webhook [here](https://www.elastic.co/guide/en/kibana/current/webhook-action-type.html). - When setting up the webhook in Elastic you will have to provide a url to send the info to. Provide the Shuffle webhook URI you copied in step 1 above - You do not need to provide authentication in Elastic while setting up, but just incase you do, remember to do the same in your webhook in Shuffle 4. To finish setting up your connector in Elastic you will be required to provide the body of the webhook connector, we will need to provide a JSON body for the connector. Copy and paste the below in the connector body - ```{"raw": {{context.alerts}}}``` + `{"raw": {{context.alerts}}}` 5. Go ahead and test your connector and then check your workflow executions in Shuffle if you have an incoming execution. - -### Cortex +### Cortex + TBD: Responder executions