In today's world, when cloud-native development is crucial, developers and businesses confront substantial challenges with infrastructure installation, maintenance, and monitoring. Managing these challenges is crucial to ensuring effective operations and innovation. This blog post will discuss Kestra, a tool that offers essential solutions for modern infrastructure orchestration.

Prerequisites

• Basic understanding of Docker.

• Knowledge of YAML configurations.

• AWS account and basic understanding of AWS S3 Buckets.

Challenges in Infrastructure Orchestration

There are various challenges faced while orchestrating an infrastructure. Some of the common ones are listed below:

• Complex Workflow Orchestration: Orchestrating infrastructure processes required coordinating several jobs with dependencies, which was time-consuming and more exposed to errors.

• Scalability Issues: As workloads increased, scaling the orchestration system proved ot be more and more difficult.

• Monitoring and Logging: Extensive monitoring and logging were necessary to provide insights into workflow execution and identify errors rapidly.

• Inefficient Use of Developer Time: Developers spend excessive time maintaining workflows and infrastructure rather than creating and innovating.

Kestra as a Solution:

Kestra is a Unified Orchestration Platform for scheduling and orchestration that simplifies complicated data pipelines and processes. Kestra gives developers the freedom to concentrate on creativity and execution rather than the complexities of infrastructure administration by providing a complete platform. It offers a user-friendly visual interface for process design, which both technical and non-technical people may efficiently utilize.

Kestra successfully handles typical difficulties in infrastructure orchestration with a range of additional features:

• Visual Workflow Designer: Kestra's offers a topology feature which provides visual aid to the developers of the Kestra flow implemented. It furthermore helps the other team members to edit the configuration as required using the topology feature.

• Scalable Architecture: Kestra was designed to handle large-scale processes easily. Because it supports horizontal scalability, It may expand with your demands, delivering stable performance even under high load.

• Comprehensive Monitoring Tools: Kestra provides access to thorough records and monitoring capabilities. These technologies give insights on process progress and performance, allowing for faster identification and resolution of issues, hence ensuring system dependability.

• Simplified Management: Kestra combines several management duties into a single platform, easing infrastructure deployment and management. This connection lowers the need for various external tools, simplifying the entire management process.

• Realtime Triggers: One of Kestra's most notable features is its support for realtime triggers. This feature enables workflows to be started automatically depending on specific events or situations, guaranteeing fast reactions to significant changes and improving the efficiency of automated procedures.

Key Features of Kestra

• Declarative Orchestration: Kestra enables you to define workflows and configurations clearly and declaratively, streamlining the automation process.

• Automation Platform: Kestra simplifies your scheduling and automation tasks through declarative language, making workflow management straightforward.

• API First: Kestra is designed in an API-first way, enabling programmatic access to all actions from managing workflows to user administration.

• Language Agnostic: Kestra supports language-agnostic workflows, allowing users to implement tasks using the language best suited to their needs.

• Kestra's Terraform Provider: Manage and deploy Kestra workflows directly within your existing Terraform environment, eliminating the need for separate configurations.

• Configuration without Extensive Code: Reduces the need to write long and complex configuration files.

• Quick Deployment: Deploy workflows with just a few clicks, streamlining the process.

• Comprehensive Dashboard: Monitor all workflows and statuses at a glance.

• Detailed Insights: Provides in-depth information on each task and workflow, including logs and metrics.

Automate Tasks Using Kestra

In this section we will see one of many examples of how Kestra can help you in automating daily tasks in your workflow.

In the demo documented below I have created a flow that uses the in-built trigger functionality provided by Kestra. This flow will execute whenever new files with a given prefix are detected in the specified S3 bucket and will iterate over the objects and print their URIs..

Step 1: Install Kestra in your Local Machine

To install Kestra using docker, run the below command in your terminal:

docker run --pull=always --rm -it -p 8080:8080 --user=root \
 -v /var/run/docker.sock:/var/run/docker.sock \
 -v /tmp:/tmp kestra/kestra:latest server local

To explore other installing options you can visit installation guide.

Open http://localhost:8080 in your browser to launch the UI and start building your first flow.

Step 2: Create a New Flow

Click on create new flow button and you can edit the default configuration to setup your own Kestra flow.

id: s3Trigger
namespace: company.team

tasks:
  - id: each
    type: io.kestra.plugin.core.flow.EachParallel
    tasks:
      - id: s3object
        type: io.kestra.plugin.core.debug.Return
        format: "{{ taskrun.value }}"
    value: "{{ trigger.objects | jq('.[].uri') }}"

triggers:
  - id: waitForS3object
    type: io.kestra.plugin.aws.s3.Trigger
    bucket: <bucket-name>
    prefix: <file_name-prefix>
    interval: PT1S
    filter: FILES
    action: MOVE
    moveTo:
      key: archive/demo/ #Structuring of files inside the S3 bucket
    region: "{{ secret('AWS_DEFAULT_REGION') }}"
    accessKeyId: "{{ secret('AWS_ACCESS_KEY_ID') }}"
    secretKeyId: "{{ secret('AWS_SECRET_ACCESS_KEY') }}"

The above flow will be triggered whenever new files with a given prefix are detected in the specified S3 bucket.

Definitions:

id and namespace: The workflow is identified by id s3Trigger and it resides within the namespace company.team. This structure helps in organizing and referencing workflows within the Kestra environment.

Tasks:

• each: This is a control flow task designed to execute its contained tasks in parallel with type io.kestra.plugin.core.flow.EachParallel. The task iterates over a collection provided in its value attribute.

• s3object: Within the each task, there is a sub-task of type io.kestra.plugin.core.debug.Return. This task simply returns the value it receives, formatted as specified. In this case, it formats the input to display the URI of each S3 object triggered.

• value Expression: The value for the each task is dynamically obtained from the trigger's output, specifically extracting the URI of each object from the S3 bucket using the jq expression .[].uri. This ensures that each parallel execution in the each task deals with a unique S3 object URI.

Triggers:

• waitForS3object: This component listens for events in a specified AWS S3 bucket.

• bucket and prefix: The trigger monitors a specific S3 bucket and objects that begin with a certain prefix.

• interval: It polls the bucket every second PT1S, which means it checks for changes at this frequency.

• filter and action: The trigger is set to respond only to files, and the action specified is MOVE, which moves processed files to a new location within the same bucket moveTo: archive/demo/.

• region, accessKeyId, and secretKeyId: It uses region and AWS credentials (access key ID and secret access key) stored securely in secrets. These credentials are necessary for the trigger to access and operate within the AWS environment. You can also directly use the value without storing them as secrets.

Step 3: Create a S3 Bucket

To complete the above head over to AWS Management Console and create a S3 bucket.

Click on Create Bucket button to create a new bucket. After the bucket creation is complete you can see the bucket under Buckets section

Copy the below information from AWS Management Console and update the value of configuration given in Step 2:

• S3 Bucket Name: Update bucket value.

• Access Key Value: Update accessKeyId value.

• Secret Key Value: Update secretKeyId value.

Step 4: Test the flow

Before we test the above flow, this is how your configuration on Kestra UI should look like:

Save the above configuration and head over to AWS S3 Bucket. Click on the bucket created, go to upload files options and click Add files button to upload the files.

Wait for few seconds after uploading the files and head over to execution tab on the Kestra UI page. You will see the overview of the flow just executed:

Click on Logs section in the navbar which will verify if all the tasks of the flow were executed and it should return the URI of the files uploaded as shown below:

To double verify you can head over to AWS S3 Bucket, after clicking on the S3 Bucket you will be able to see the structuring of the files under the folder name given in moveTo section which in this scenario was archive/demo.

Kestra vs Other Tools

Kestra distinguishes itself from competitors through the following:

• Realtime Alerts and Triggers: Notifies teams of workflow issues, ensuring quick resolution and minimal downtime while providing functionality to trigger a process to handle such situations simultaneously or in parallel, bringing in real-time automation.

• Unified Platform Experience: Unlike other systems that need several tools, Kestra provides a unified, smooth experience for orchestrating processes.

• Developer-Centric Design: The platform prioritizes usability, allowing developers to focus on generating business logic and code without being distracted by infrastructural concerns.

Ideal Market

Kestra is well-suited for a broad audience, including:

• Enterprises and SMEs seeking robust orchestration solutions.

• Startups need scalable, cost-effective tools for infrastructure management.

• Data Engineers looking to streamline complex workflows across various industries and domains.

• DevOps/Platform Teams looking for an accessible platform to streamline workflows.

Getting Involved

For those interested in exploring Kestra further:

• Contribute to the Project: Visit Kestra's GitHub repository to contribute to its development.

• Try Kestra: Download the tool from the official website and start orchestrating your infrastructure with ease.

• Join the Community: Interact with other users and the development team to share insights and get support on Slack.

Conclusion

Kestra's innovative workflow and task orchestration approach revolutionizes how organizations design, execute, and monitor complex workflows, ensuring more efficient operations and better integration across technological stacks. For any developer struggling with the complexities of infrastructure management, Kestra offers a streamlined and efficient solution that is worth exploring.

Shoutout to Kestra for collaborating with me on this blog.