|
| 1 | +--- |
| 2 | +title: Kestra integration |
| 3 | +description: Connect Apify with Kestra to orchestrate workflows — run flows, extract structured data, and react to Actor or task events. |
| 4 | +sidebar_label: Kestra |
| 5 | +sidebar_position: 7 |
| 6 | +slug: /integrations/kestra |
| 7 | +--- |
| 8 | + |
| 9 | +**Connect Apify with Kestra to orchestrate workflows — run flows, extract structured data, and react to Actor or task events.** |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +[Kestra](https://kestra.io/) is an open-source, event-driven orchestration platform. The [Apify plugin for Kestra](https://github.com/kestra-io/plugin-kestra) connects Apify Actors and storage to your workflows. Run scrapers, extract structured data — all defined declaratively in YAML and orchestrated directly from the UI. |
| 14 | + |
| 15 | +This guide shows you how to set up the integration, configure authentication, and create a workflow that runs an Actor and processes its results. |
| 16 | + |
| 17 | +## Prerequisites |
| 18 | + |
| 19 | +Before you begin, make sure you have: |
| 20 | + |
| 21 | +- An [Apify account](https://console.apify.com/) |
| 22 | +- A [Kestra instance](https://kestra.io/docs/getting-started/quickstart) (self‑hosted or cloud) |
| 23 | + |
| 24 | +## Authentication |
| 25 | + |
| 26 | +The Apify plugin uses API key authentication. Store your API key in [Kestra Secrets](https://kestra.io/docs/concepts/secret) through the UI or environment variables. In the open-source version, manage Secrets using base64-encoded environment variables. You can also use [Kestra's KV Store](https://kestra.io/docs/concepts/kv-store) to persist API keys across executions and workflows. |
| 27 | + |
| 28 | +To add your Apify API token, go to the Secrets section in the Kestra UI and create a new secret with the key `APIFY_API_KEY` and your token as the value. |
| 29 | + |
| 30 | +## Use Apify Tasks as an action |
| 31 | + |
| 32 | +Tasks allow you to perform operations like running an Actor within a workflow. |
| 33 | + |
| 34 | +1. Create a new flow. |
| 35 | +1. Inside the **Flow code** tab change the hello task's type to be **io.kestra.plugin.apify.actor.Run**. |
| 36 | +1. Change the task's id to be **run_apify_actor** |
| 37 | +1. Remove the message property. |
| 38 | +1. Configure the **run_apify_actor** task by adding your required values for the properties listed below: |
| 39 | + - **actorId**: Actor ID or a tilde-separated owner's username and Actor name. |
| 40 | + - **apiToken**: A reference to the secret value you set up earlier. For example "\{\{secret(namespace=flow.namespace, key='APIFY_API_KEY')\}\}" |
| 41 | +1. Add a new task below the **run_apify_actor** with an ID of **get_dataset** and a type of **io.kestra.plugin.apify.dataset.Get**.: |
| 42 | +1. Configure the **get_dataset** to fetch the dataset generated by the **run_apify_actor** task by configuring the following values: |
| 43 | + - **datasetId**: The ID of the dataset to fetch. You can use the value from the previous task using the following syntax: "\{\{secret(namespace=flow.namespace, key='APIFY_API_KEY')\}\}" |
| 44 | + - **input**: Input for the Actor run. The input is optional and can be used to pass data to the Actor. For our example we will add 'hashtags: ["fyp"]' |
| 45 | + - **maxItems**: The maximum number of items to fetch from the dataset. For our example we will set this to 5. |
| 46 | +1. Now add the final task to log the output of the dataset. Add a new task below the **log_output** with an ID of **log_output** and a type of **io.kestra.plugin.core.log.Log**. |
| 47 | +1. Configure the **log_output** task to log the output of the dataset by configuring the following values: |
| 48 | + - **message**: The message to log. You can use the value from the previous task using the following syntax: '\{\{outputs.get_dataset.dataset\}\}' |
| 49 | +1. Now save and run your flow. |
| 50 | + |
| 51 | +Your completed template should match the template below. |
| 52 | + |
| 53 | +```yaml |
| 54 | +id: run_actor_and_fetch_dataset |
| 55 | +namespace: company.team |
| 56 | + |
| 57 | +tasks: |
| 58 | + - id: run_actor |
| 59 | + type: io.kestra.plugin.apify.actor.Run |
| 60 | + actorId: GdWCkxBtKWOsKjdch |
| 61 | + maxItems: 5 |
| 62 | + input: |
| 63 | + hashtags: ["fyp"] |
| 64 | + apiToken: "{{secret(namespace=flow.namespace, key='APIFY_API_KEY')}}" |
| 65 | + - id: log_get_last_run_results |
| 66 | + type: io.kestra.plugin.core.log.Log |
| 67 | + message: '{{outputs.run_actor}}' |
| 68 | + - id: get_data_set_raw |
| 69 | + type: io.kestra.plugin.apify.dataset.Get |
| 70 | + datasetId: '{{outputs.run_actor.defaultDatasetId}}' |
| 71 | + apiToken: "{{secret(namespace=flow.namespace, key='APIFY_API_KEY')}}" |
| 72 | + - id: log_get_data_set_raw_results |
| 73 | + type: io.kestra.plugin.core.log.Log |
| 74 | + message: '{{outputs.get_data_set_raw}}' |
| 75 | +``` |
| 76 | +
|
| 77 | +## Resources |
| 78 | +
|
| 79 | +- [Kestra Apify Plugin](https://kestra.io/plugins/plugin-apify) |
| 80 | +- [Apify API Documentation](https://docs.apify.com) |
| 81 | +- [Kestra Documentation](https://kestra.io/docs) |
| 82 | +
|
| 83 | +## Troubleshooting |
| 84 | +
|
| 85 | +If you encounter issues, start by double-checking basics. |
| 86 | +
|
| 87 | +- **Authentication errors**: Verify your API token in [Secrets](https://kestra.io/docs/concepts/secret). |
| 88 | +- **Operation failures**: Check input parameters, YAML syntax, and resource IDs in your Apify account. |
| 89 | +
|
| 90 | +Feel free to explore other resources and contribute to the integration on [GitHub](https://github.com/kestra-io/plugin-apify). |
0 commit comments