Workflow on Wanzi's Blog – Notes on life, tech, and DevOps.

Getting Started with Argo Events

wzdushu@gmail.com (wanzi) — Wed, 07 Jul 2021 16:22:42 +0800

Previously, we introduced how to install Argo Workflow and trigger tasks. In this article, we focus on a new tool:

What is ArgoEvents?

Argo Events is an event-driven Kubernetes workflow automation framework. It supports over 20 different event sources (e.g., webhooks, S3 drops, cronjobs, message queues such as Kafka, GCP PubSub, SNS, SQS, etc.).

Features:

Supports events from over 20 event sources and more than 10 trigger types.
Enables customization of business-level constraints for workflow automation.
Manages everything from simple, linear, real-time workflows to complex, multi-source event scenarios.
Complies with the CloudEvents specification.

Components:

EventSource (similar to a gateway; sends messages to the event bus)
EventBus (event message queue; implemented using high-performance distributed messaging middleware NATS — note that NATS has ceased maintenance after 2023, so architectural changes may be expected in the future)
EventSensor (subscribes to the message queue, parameterizes events, and filters them)

Deploying ArgoEvents

Deploy argo-events:

1
2

kubectl create ns argo-events
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/v1.2.3/manifests/install.yaml

Deploy argo-eventbus:

`1`	`kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml`

RBAC Account Authorization

Create `operate-workflow-sa` account

Grant operate-workflow-sa permission to create Argo Workflows within the argo-events namespace — required for EventSensor to automatically create workflows later.

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: argo-events
  name: operate-workflow-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: operate-workflow-role
  namespace: argo-events
rules:
  - apiGroups:
      - argoproj.io
    verbs:
      - "*"
    resources:
      - workflows
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: operate-workflow-role-binding
  namespace: argo-events
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: operate-workflow-role
subjects:
  - kind: ServiceAccount
    name: operate-workflow-sa
    namespace: argo-events

Create `workflow-pods-sa` account

Grant workflow-pods-sa permission to create pods via workflows within the argo-events namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: argo-events
  name: workflow-pods-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: workflow-pods-role
rules:
  - apiGroups:
      - ""
    verbs:
      - "*"
    resources:
      - pods
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: workflow-pods-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: workflow-pods-role
subjects:
  - kind: ServiceAccount
    name: workflow-pods-sa
    namespace: argo-events

Automating Task Triggers with ArgoEvents

Start an event source to receive requests:

`1`	`kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/event-sources/webhook.yaml`

Note: The name field in the event source is example. You must specify this exact name when creating a sensor in production environments.

Create a webhook-based sensor to consume requests:

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: webhook
spec:
  template:
    serviceAccountName: operate-workflow-sa
  dependencies:
    - name: test-dep
      eventSourceName: webhook
      eventName: example
  triggers:
    - template:
        name: webhook-workflow-trigger
        k8s:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: webhook-
              spec:
                serviceAccountName: workflow-pods-sa
                entrypoint: whalesay
                arguments:
                  parameters:
                  - name: message
                    # Value will be overridden by payload from test-dep
                    value: hello world
                templates:
                - name: whalesay
                  inputs:
                    parameters:
                    - name: message
                  container:
                    image: docker/whalesay:latest
                    command: [cowsay]
                    args: ["{{inputs.parameters.message}}"]
          parameters:
            - src:
                dependencyName: test-dep
              dest: spec.arguments.parameters.0.value

Reference: https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/sensors/webhook.yaml

Forward local requests to remote:

`1`	`kubectl -n argo-events port-forward <name-of-event-source-pod> <local port>:12000`

Send data to the event source:

`1`	`curl -d '{"message":"ok"}' -H "Content-Type: application/json" -X POST http://localhost:12000/example`

Now, you can automate workflow task creation using ArgoEvents.

Argo Workflow Practice I: Installation and Deployment

wzdushu@gmail.com (wanzi) — Fri, 07 May 2021 16:22:42 +0800

Introduction & Architecture

Argo Workflows is an open-source, container-level workflow engine designed to orchestrate parallel jobs on Kubernetes. It leverages Kubernetes Custom Resource Definitions (CRDs) to implement its full architecture, including Workflow, Workflow Template, and Cron Workflow.

What Can Argo Workflows Do?

Define workflows where each step is a container.
Model multi-step workflows as a sequence of tasks or capture task dependencies using Directed Acyclic Graphs (DAGs).
Easily run compute-intensive jobs—such as machine learning or data processing—on Kubernetes within a short time frame.
Run CI/CD pipelines natively on Kubernetes without configuring complex software development tooling.

Key Features of Argo Workflows:

Workflow: Orchestrates multiple workflow templates with customizable execution order.
Workflow Template: A reusable template definition for workflows; can be invoked by other workflows or templates within the same namespace or cluster.
Cluster Workflow Template: A cluster-scoped workflow template accessible across all namespaces via ClusterRole permissions.
Cron Workflow: Scheduled workflow type, equivalent to an advanced version of Kubernetes CronJob.

Installation & Configuration

Install Argo Workflows

We are installing the stable version 2.12.10. The installation process sets up ServiceAccount, Role, ClusterRole, Deployment, and other necessary components.

1
2

kubectl create ns argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/v2.12.10/manifests/install.yaml

Configure Ingress for External Access

Since our cluster uses Traefik as the ingress controller, and Argo Workflows defaults to internal HTTPS access, we need to add specific annotations to route external requests to argo-server.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/redirect-entry-point: https
  name: argo-server
  namespace: argo
spec:
  rules:
  - host: argo.test.cn
    http:
      paths:
      - backend:
          serviceName: argo-server
          servicePort: web
        path: /
status:
  loadBalancer: {}

External access URL: https://argo.test.cn

Simple Workflow Test

Let’s test with a “Hello World” example:

# argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/master/examples/hello-world.yaml
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Pending
Created:             Fri May 07 16:11:25 +0800 (now)
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Fri May 07 16:11:25 +0800 (now)
Started:             Fri May 07 16:11:25 +0800 (now)
Duration:            0 seconds

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-4mffd  whalesay  hello-world-4mffd  0s
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Running
Created:             Fri May 07 16:11:25 +0800 (10 seconds ago)
Started:             Fri May 07 16:11:25 +0800 (10 seconds ago)
Duration:            10 seconds

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ◷ hello-world-4mffd  whalesay  hello-world-4mffd  10s       ContainerCreating
Name:                hello-world-4mffd
Namespace:           argo
ServiceAccount:      default
Status:              Succeeded
Conditions:
 Completed           True
Created:             Fri May 07 16:11:25 +0800 (2 minutes ago)
Started:             Fri May 07 16:11:25 +0800 (2 minutes ago)
Finished:            Fri May 07 16:14:18 +0800 (now)
Duration:            2 minutes 53 seconds
ResourcesDuration:   1m18s*cpu,1m18s*memory

STEP                  TEMPLATE  PODNAME            DURATION  MESSAGE
 ✔ hello-world-4mffd  whalesay  hello-world-4mffd  2m

# argo list -n argo
NAME                STATUS      AGE   DURATION   PRIORITY
hello-world-4mffd   Succeeded   3m    2m         0

# argo logs -f hello-world-4mffd -n argo
hello-world-4mffd:  _____________
hello-world-4mffd: < hello world >
hello-world-4mffd:  -------------
hello-world-4mffd:     \
hello-world-4mffd:      \
hello-world-4mffd:       \
hello-world-4mffd:                     ##        .
hello-world-4mffd:               ## ## ##       ==
hello-world-4mffd:            ## ## ## ##      ===
hello-world-4mffd:        /""""""""""""""""___/ ===
hello-world-4mffd:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
hello-world-4mffd:        \______ o          __/
hello-world-4mffd:         \    \        __/
hello-world-4mffd:           \____\______/