Microsoft is doing a lot of work to ensure that containers are a first-class citizen, not only on Azure but on any cloud. Part of that work is its growing family of tools to help manage workloads running on the open source Kubernetes container orchestration platform, work accelerated by its acquisition of Deis and its Helm and Draft tools. They’ve recently been joined by Brigade, a JavaScript-based tool for building and managing workflows running on Kubernetes.
Kubernetes via code: a distributed pipeline
People often think of Kubernetes as a tool for hosting and managing large-scale distributed applications, marshalling container-hosted microservices, handling resources, and managing scaling. But it’s more flexible than that; Kubernetes is also ideal for instantiating and running small applications or one-time functions. Perhaps you want to use it to host test scaffolding for a continuous integration/continuous delivery (CI/CD) pipeline, or to respond to a specific message from a line-of-business application or from a machine learning system.
Putting together a small application on Kubernetes isn’t particularly easy. It requires the same amount of work as building the framework for running a much larger application, including writing the YAML needed to manage your code and containers. But not all code needs that work, some because it’s a quick hack designed to try out a new idea, some because it’s intended to run only rarely, or some because it’s to host a set of tests that are needed only when you’re running a build.
Most of us prefer to write code than configuration files. There’s something about imperative programming that makes it easier to understand and to debug than using declarative statements. Brigade is designed to bring that imperative approach to building Kubernetes applications, pulling together containers and passing data between them.
Brigade applications are pipelines, a common distributed computing pattern that treats containers as separate jobs that can run in serial or in parallel. The Brigade JavaScript handles how the pipeline operates, passing data between jobs and marshalling both parallel and pipelined tasks. At heart it’s a workflow engine, with a basic grammar that uses Docker containers as actions, all scripted using familiar JavaScript.
While people often think of distributed application development as a way of building and managing large scalable applications, the underlying concepts work well for any application that can be broken down into small components—especially if elements can be run in parallel. Brigade pipelines can be used to handle important parts of any cloud application platform, batch-processing large amounts of data, handling ETL tasks, and even managing your devops environment and services.
Using Brigade to go serverless
If you’re running a Kubernetes cluster, you can use Brigade to build and run your own serverless applications. By triggering a Brigade pipeline, you launch a series of containers that you’ve defined and set up in advance, passing results from one container-hosted job to the next.
Serverless applications are more than the familiar function-as-a-service approach you know from Azure Functions or from AWS Lambdas. With Brigade, they can be much more powerful, hosting complete containers running complex applications that fire up as and when they’re needed, idling until required and thus using little or no compute and only a small amount of storage. It’s a model that fits well with the economics of the public cloud, taking advantage of per-second billing for compute and pennies-per-gigabyte storage costs.
Get started with Brigade
Getting started with Brigade is easy enough, because it’s just a single line command in Kubernetes to install the native Brigade service. It runs on any Kubernetes system: a local development minikube, a private cloud, or a hyperscale public clouds. And because it’s native Kubernetes code, there’s no lockin to Azure, giving you the option to use Brigade in multicloud deployments. Brigade-scripted jobs and containers will run on any Kubernetes install, pulling from Docker repositories as necessary.
Brigade projects can contain multiple pipelines, using a JavaScript config file to define your pipelines and tasks. They don’t need to only address containers; you can use them to work with any RESTful API, letting your code interact with services like Slack or Teams, quickly going from devops to chatops. Requests can come in from any service, so you can work with known APIs or use webhooks via the built-in Brigadier gateway. Alternatively, you can write your own gateways to, for example, handle events (with the appropriate libraries) that are sourced using CloudEvents.
New projects are created using a Helm chart. Although one of the aims of Brigade is to avoid using YAML, you do need to use it to set up basic project configuration to manage an install on your cluster, for delivering secrets and tokens. The development team also offers a command-line tool called Brig that helps you test your scripts and configure your server.
Brigade pipelines are conceptually very simple. They start by importing key objects from the underlying Brigadier service, which provides the event listener and links into Kubernetes. The main element of your cod is an event handler, which wraps your pipeline code. Once the even fires, the code runs, and your pipeline reports that is has completed. The code itself consists of a series of jobs, each of which is a Docker image. You can pull from the stock gallery or use your own custom images. Each job can then be given tasks, a series of commands that run inside the image.
Jobs can be grouped to run in parallel using Brigade’s group command, or left to run one after the other in series. With the basic group function, there’s no flow control, so the pipeline continues to run, launching the subsequent jobs before the grouped jobs complete.
You can use JavaScript features to add control to your code. Each job or group of jobs can use promise constructs to ensure sequential operations, taking advantage of JavaScript asynchronous operations. Once all the tasks have completed and the jobs in the pipeline are marked as succeeded, the Kubernetes host tears down the Brigade pipeline and discards the job containers until its next run.
Make friends with the Brigadier
Although much of Brigade is focused on working with containers, you still need a way to see how it’s operating. That’s where Kashti comes in, offering a dashboard for Brigade operations. With Kashti, you can configure and manage logs, using it as a console for all your pipelines. Outputs can be tracked and visualized, adding not only a reporting layer but also the deep insights into your Brigade scripts that you’ll need to test and debug your code.
You can do a lot with Kubernetes, especially if you’re using the managed Kubernetes services on Azure and competing services. Applications like Helm, Draft, and now Brigade extend Kubernetes in ways that make an operations platform much more developer-friendly, and they are well worth adding to your Kubernetes toolbox.