Amber

A declarative workflow orchestrator written in Go.

Features

• Declarative workflows using Nickel
• Automatic handling of task dependencies
• Task input/output system

Use Cases

• Backend infrastructure management (This very server relies on an Amber workflow made up of idempotent bash tasks for its deployment!)
• Data migrations
• CI/CD pipelines
• Monorepo scripting & management

Usage

CLI Tool

The CLI tool takes one or more paths to a workflow entry point. Note that multiple entry points will not run in sequence - instead they will effectively be combined into a single workflow graph.

cd amber
go run . /path/to/workflow.ncl

Workflows

Workflows are written in Nickel. Each entry point must evaluate to an array of type Task as defined in amber.ncl.

The following sub-sections will gradually explain features and go over some patterns you can use to compose manageable workflows. They build upon concepts found in The Nickel User Manual which will not be reiterated here.

Basics

Property	Description
id	A unique identifier for the task. If conflicting definitions for the same id are detected, the workflow will fail.
run	A bash script to run.

# workflow.ncl
[
  {
    id = "my-task",
    run = m%"
      echo "Hello, world!"
    "%
  }
]

$ amber workflow.ncl
> Building DAG...
> DAG ready
> Starting task: my-task
> Hello, world!
> Finished task: my-task

Dependencies

Property	Description
requires	An optional array of Task entries that must complete before this task runs.

Tasks added to requires are automatically included in the graph. It is not necessary to also include them as a top-level task in any entry point (although you can).

# workflow.ncl
let
  taskOne = {
    id = "task-one",
    run = m%"
      echo "I'm running first!"
    "%
  }
in
[
  {
    id = "task-two",
    requires = [taskOne],
    run = m%"
      echo "I'm running after!"
    "%
  }
]

$ amber workflow.ncl
> Building DAG...
> DAG ready
> Starting task: task-one
> I'm running first!
> Finished task: task-one
> Starting task: task-two
> I'm running after!
> Finished task: task-two

Outputs

• Providing task: A task that provides output value(s).
• Consuming task: A task that uses other task(s) output value(s).

Tasks can be both providers and consumers and multiple tasks can consume a single task's outputs.

Property	Description
inputs	An optional map of named outputs from other tasks to provide to this task as a runtime environment variable.
outputs	An optional array of required named outputs.

Providing Outputs

Outputs are provided by using a .env style syntax in stdout by a providing task:

#!/usr/bin/env bash

SOME_VALUE=1234

# Send it to stdout
echo "my_output=$SOME_OUTPUT"

If a providing task fails to provide a named output declared in its outputs array, it will fail.

A providing task can provide other outputs in addition to those declared in outputs, but they are optional and the providing task will not fail if such outputs are not set.

Consuming Outputs

A task will fail if any of its declared inputs are not available. The task providing the output must be added to the consuming task's requires array.

In a future version this dependency may be handled automatically provided the full task definition is included somewhere in the graph, however it will remain a good practice to include it explicitly.

Example

# workflow.ncl
let
  providerTask = {
    id = "provider-task",
    outputs = ["my_output"],
    run = m%"
      SOME_VALUE=1234

      # Send it to stdout
      echo "my_output=$SOME_OUTPUT"
    "%
  }
in
[
  {
    id = "consumer-task",
    requires = [providerTask],
    inputs = {
      OTHER_TASK_OUTPUT = "provider-task.my_output",
    },
    run = m%"
      echo "That task's my_output is $OTHER_TASK_OUTPUT"
    "%
  }
]

$ amber workflow.ncl
> Building DAG...
> DAG ready
> Starting task: provider-task
> my_output=1234
> Finished task: provider-task
> Starting task: consumer-task
> That task's my_output is 1234
> Finished task: consumer-task

Type Enforcement

You can import Task and enforce the type. This means you can use the nickel CLI to check your files. This example assumes Amber was cloned to a sub-directory:

let
  { Task } = import "amber/amber.ncl"
in
[
  {
    id = "my-task",
    run = m%"
      echo "Hello, world!"
    "%
  }
] | Array Task

Modular Configuration

import can be used to break tasks into their own files:

# task-1.ncl
{
  id = "task-1",
  run = m%"
    echo "Hello from task 1"
  "%
}

# task-2.ncl
{
  id = "task-2",
  run = m%"
    echo "Hello from task 2"
  "%
}

# workflow.ncl
[
  import "task-1.ncl",
  import "task-2.ncl",
]

$ amber workflow.ncl
> Building DAG...
> DAG ready
> Starting task: task-1
> Hello from task 1
> Finished task: task-1
> Starting task: task-2
> Hello from task 2
> Finished task: task-2

Bash File Imports

import can be used to read a bash script from its own file:

# my-script.sh

#!/usr/bin/env bash

set -euo pipefail

echo "Doing stuff..."

# workflow.ncl
[
  {
    id = "external-bash-script",
    run = import "my-script.sh" as 'Text,
  }
]

$ amber workflow.ncl
> Building DAG...
> DAG ready
> Starting task: external-bash-script
> Doing stuff...
> Finished task: external-bash-script