Build a devops pipeline for your Windows servers

Windows shops transitioning to devops may not yet know the power of build pipelines when it comes to standing up and maintaing infrastructure. A concept familiar to software developers, the build/release pipeline may sound foreign to those from the operations side of the aisle. But don’t sweat it; InfoWorld is here to help.

A build pipeline consists of many moving parts, the orchestration of which can be hard to grasp. In a nutshell, your build pipeline’s structured set of processes to set your scripts in motion on your servers automatically. You write code -- say, some configuration scripts -- and commit that code to a source control repository, which kicks off one or more rounds of tests before deploying your code to one or more environments. The key here is automation.

In this article, I break down the process of building a simple continuous deployment pipeline as a way of explaining build pipelines by doing. Follow along to create a build pipeline from scratch, going over each piece of this orchestration of tools. When it comes to devops automation, often the most valuable knowledge comes from focusing on the tactical components.

Note: From now on, I refer to this build/release pipeline as a build pipeline. Although the concept can be referred to a different ways, in this article, it means the process of writing code, committing code to source control, and automatically taking that code to a finished state as a “build pipeline.”

Setting the stage for the build pipeline

A build pipeline -- more specifically, a continuous deployment build pipeline -- automates the process of writing and deploying code. It ensures you deliver quality code in an consistent, automated fashion. In the devops world, infrastructure as code is the goal, and the more your infrastructure is provisioned, configured, and managed via code, the more important build pipelines become.

The tools required to orchestrate a build pipeline are many and varied. The point here is not to focus on the tools per se. It does not matter whether you choose to use GitHub, Team Foundation Server (TFS), or SVN for source control, or TFS, AppVeyor, or Jenkins to run builds. Each organization's tool set is different, as well as how those tools are used.

For this example build pipeline, you'll focus on PowerShell and Desired State Configuration (DSC), which provides the means for configuring and managing Windows servers. The primary focus of this pipeline is to automatically configure a server via DSC configuration scripts and automatically test the resulting configurations to ensure the correct configurations have been applied.

To accomplish the task you'll use the following tools:

• Git/GitHub (source control)
• AppVeyor (build orchestration)
• Pester (testing)
• Azure Automation DSC (deployment)

A build pipeline's devops prerequisites

To ensure you have a complete picture of what it takes to create a build pipeline, this example begins as early as possible in its development. However, to not bog down this article into a how-to for each tool, a few prerequisites should be met. You will need:

• An Azure virtual machine that's been onboarded to Azure Automation DSC
• A free GitHub account and repository (ours will be called TestDomainCreator)
• A free AppVeyor account
• A service principal created in Azure Active Directory for authentication

Everything else that you need will be created as the build pipeline is put together.

Setting up the PowerShell script

1. Create the DSC configuration

When this example DSC configuration script is applied to your virtual machine, it will create a new domain. To create this new domain, the script will perform four tasks:

• Promote a server to a domain controller.
• Ensure common Active Directory groups are created.
• Ensure common Active Directory organizational units are created.
• Ensure common Active Directory users are created.

The DSC configuration script can be downloaded via the TestDomainCreator Github repo.

You may notice in the TestDomainCreator Github repo that there is a separate ConfigurationData.psd1 file. In that file, you define all of the static configuration items necessary to bring up your test domain. This includes values like domain name, the names of various groups, users, and organizational units (OUs). If you're familiar with DSC, you know that when building DSC configurations you can define configuration data directly in the script itself, as a hash table that is passed to the configuration, or as a hash table stored in a separate file.

So why are you storing the configuration data in a separate PSD1 file? Modularizing code as much as possible makes it easier to manage, and it lets you reuse pieces of your code. By breaking configuration data out this way, you get the benefit of gathering in one file the expected state of our machine after DSC has run. That way, you can use the PSD1 file as instructions for DSC and for your Pester tests to verify that the machine is configured the way you want when DSC is done.

2. Create the Pester tests

Once you've created your DSC configuration script, you need to build tests for validating your script. In this case, you’ll use Pester for testing, and you’ll build integration or infrastructure tests, not to be confused with unit tests. The goal is to run your Pester tests to ensure that DSC did its job setting up your server — and more important, that your script did its job in telling DSC what you'd like it to do.

Download the Pester test script here. As you can see from the file, you're authenticating to Azure to communicate with the Azure virtual machine and running various tests on it to confirm that once the DSC configuration is applied, it matches the changes you expect DSC to perform.

The tasks you’re performing with DSC in this example are simple, but imagine if you were using DSC to provision entire environments. With a test file already built and a framework already set up, as you have here, you can add as many tests as you need to scale to a larger environment.

Setting up AppVeyor

1. Link your GitHub repo to AppVeyor

With your DSC script and test script complete, you still need to create the pipeline that glues your pieces together. Here's where a build system comes into play.

For this demonstration, you’ll use a cloud-based build system called AppVeyor. But the techniques used here can just as easily be implemented in build systems like TravisCI, Jenkins, and Visual Studio Team Services.

First, you’ll need to point your AppVeyor account to your GitHub account. This will let AppVeyor participate in your continuous integration pipeline by automatically detecting changes committed to your GitHub repository. If you have multiple GitHub repositories, don’t worry: AppVeyor connects to your GitHub account itself, so you'll only have to create one link.

To link your GitHub account to AppVeyor, log into AppVeyor, click on your name in the upper right corner, and then click Profile.

IDG

Once in your profile, click Link Github Account and enter your GitHub username and password. Congrats. You've successfully linked your GitHub account to AppVeyor.

2. Create the AppVeyor project

With AppVeyor linked to GitHub, it's time to create a project in AppVeyor for your build process. If you have more than one build pipeline, you'll need to create multiple projects in AppVeyor, one per pipeline. To create a project, click Projects at the top of the AppVeyor dashboard and then click New Project. If this is your first project, you'll then need to authorize AppVeyor to access your GitHub repositories.

Once authenticated to GitHub, AppVeyor will show a list of the GitHub repositories it sees. You'll need to add the GitHub repository (in this example, it’s called TestDomainCreator); browse to it and click Add. Once AppVeyor adds the repository, your project is created and you can continue to the next step.

3. Encrypt secrets in AppVeyor build scripts

You never want to store sensitive information like usernames and passwords in clear text. But encryption can be a pain. Luckily, AppVeyor provides a tool to encrypt a plaintext string, which is then automatically decrypted when the build begins. To use this handy feature, click your name in the upper right corner of the AppVeyor dashboard and then choose Encrypt Data from the dropdown. You will be presented with a form for entering the string you want encrypted, and AppVeyor will return a piece of encrypted text.

IDG

The instructions you send to AppVeyor will be wrapped in a YAML file, and AppVeyor gives you an example of how this encrypted text might look in the appveyor.yml file you will be using, below. For now, it’s worth encrypting any values you know will be in your build scripts. For this example, you will need the following values encrypted:

• Azure Service principal values
• Tenant ID
• Username
• Password
• Subscription ID
• Azure virtual machine local administrator password
• Azure application ID

Keep the encrypted values in a temporary text file to plug into your build scripts when created.

4. Create the AppVeyor build scripts

AppVeyor breaks its build process into a variety of phases. Because our example is relatively simple, you’ll use just three: install, build, and test. With these three AppVeyor steps, you will execute a PowerShell script to ensure your build goes off without a hitch.

Install: Because your build requires third-party PowerShell modules and, by default, those modules are not installed on the build agent server, you'll need to perform some tasks before the build script runs.

This is where the install script comes in. This script downloads and installs the Pester PowerShell module (for your tests) and the Azure module, given that you'll eventually be communicating with your Azure virtual machine and the Azure Automation DSC service. Rather than package those modules with your project, you'll just pull those down when you need them from the PowerShell Gallery with the install script.

Build: The build script is where the heavy lifting is done. In this step, invoked after the install script, you will have AppVeyor do the following:

• Connect to Azure.
• Start the Azure test virtual machine.
• Send the newly modified DSC script to Azure Automation.
• Instruct Azure Automation DSC to compile your script.
• Instruct the Azure test virtual machine to check for the new DSC configuration.

Test: Once the build script has completed, the test script is invoked. This script triggers the Pester test script you placed at the root of your Github repository, and it is configured to send the results of the tests to AppVeyor. Pester offers the option to output an nUnit style XML file, which AppVeyor natively understands, so if you're used to the red/green console output that Pester provides, you're not doing this here. Instead, the test script will send the results file to AppVeyor, letting you see graphical test results right in AppVeyor itself. You'll view these tests at the end to see how your build went.

5. Create the AppVeyor YAML instructions

With your build scripts in a single folder inside of your GitHub repository, it's time to tell AppVeyor how to use them. AppVeyor provides a couple of ways to configure builds.

Under the Settings heading in all projects, you'll find a ton of configurable options. However, AppVeyor also provides the ability to group your build configuration in a single YAML file called appveyor.yml. Upon any change to your GitHub repository, AppVeyor will look in the root of your GitHub repo for this file and, if found, will use it to provide configuration information to your build, such as what scripts to run when and a whole lot more. This is where you'll add references to your build scripts and a few other things.

You can see that your appveyor.yml file is simple, largely because you're using just three AppVeyor build tasks and most of the logic can be found in your PowerShell scripts. At the top of the file, you can see the various IDs and passwords, encrypted, for your environment. And because all instructions are wrapped in PowerShell scripts, you simply need to reference them in the YAML file to invoke them during the appropriate build task. You can see this in your YAML file with the lines starting install, build_script, and test_script.

Invoking the build and checking tests

You now have your entire build pipeline created! So it's time to test it out. To run a test, as soon as you perform a commit to your GitHub repository, AppVeyor will be triggered and each of your build scripts will be executed in the correct order. Once complete, the build will either succeed or fail depending on the results of the Pester tests invoked in your test script. Once all tests have been executed, you'll check the results in AppVeyor to see if this orchestration went to plan.

IDG

You can find this page in AppVeyor by going into my AppVeyor projects and clicking on your current project. By default, you'll see the Latest Build, which will be updated to reflect the current state of the build. Depending on how busy the build server is at the time, your build can sit in this state anywhere from a couple minutes to about 10 minutes. In my experience, it remains queued for around five minutes before it runs.