Connect(); 2017

Volume 32 Number 13

VSTS - The Road to Continuous Delivery with Visual Studio Team Services

By Willy-Peter Schaub | Connect(); 2017

In today’s fast-paced, feature- and value-driven markets, it’s imperative that developers deliver value quickly and continuously, and react to feedback faster than the competition. Key to that effort is the practice of continuous delivery (CD), which uses continuous integration (CI) practices and automates build, test, configure and deploy activities in the DevOps cycle. CD demands that dev shops update both tooling and practices, so it’s not surprising to see many organizations asking common questions, such as:

• What is CD and how can it help deliver changes and value faster?
• What tools can you use to provision CD?
• How do you integrate various tools and cloud services products in a CD effort?
• How do you mitigate the risk in a CD/CI pipeline?

This article aims to answer these questions, sharing insight into the lessons learned so far here at Microsoft as we practice DevOps, and explore our guidelines around CD.

It’s worth noting that there’s a subtle difference between CD and continuous deployment. Continuous deployment is suitable for teams with a high level of confidence in quality gates and are deploying known-good releases directly to a production environment. With CD, release moves through several environments, which may include functional, performance and staging tests and approvals along the way, before going to production. This article focuses on CD.

Continuous Delivery

CD is a lean practice that aims to eliminate diverging code, duplication of effort and, most important, merge conflicts. It starts with CI—the process of automating the build and testing of code every time anyone commits a change to version control. When a developer implements a new feature or bug fix in a feature branch, she typically submits a pull request (bit.ly/2ha4c2G) when the feature is complete. Upon approval of the pull request, changes are committed and merged into the master branch.

CD is the process to build, test, configure, deploy and confirm a feature change to production. To deliver value to users, it’s important to achieve the shortest path to deploy a new feature (known as time to mitigate, or TTM) and to minimize reaction time to feedback (known as time to remediate, or TTR). The goal is to achieve an automated and reliable deployment process that consistently confirms through integration, load and user acceptance testing, and continuously releases value to your end users. Figure 1 depicts the cycle.

Figure 1 The Loop of Continuous Delivery

CD promises to banish manual processes, time-consuming and error-prone handoffs, and unreliable releases that produce delays, cost overruns and avoidable end-user dissatisfaction.

Provisioning the Release Pipeline

A release pipeline is a collection of environments and workflows to support the activities of CI and CD. There are many products that can help you create a pipeline that supports rapid release cycles. I’ll start by shedding some light on a few tools from the Microsoft stack that create and execute a CI/CD pipeline on Visual Studio Team Services (VSTS).

VSTS is a Microsoft Azure Software-as-a-Service (SaaS) that enables teams to plan better, code together and ship faster. Program in any language, including the Microsoft .NET Framework, Java, Python, Ruby and Node.js. Develop on any OS, such as Linux, macOS and Windows. And deploy to any platform, such as Azure, Amazon Web Services (AWS), Linux, Mac, Docker, Android, iOS and Windows Phone.

The Visual Studio IDE can simplify the configuration of CI and CD using the Continuous Developer Tools (bit.ly/2ha7MtE) built into Visual Studio 2017 Update 3. Figure 2 shows how you can configure, provision and deploy your solution without leaving Visual Studio.

Figure 2 Continuous Delivery Tools for Visual Studio

Azure Portal allows you to seamlessly manage the Azure envi­ronment and set up your release pipeline, as depicted in Figure 3.

Figure 3 Deploy with Confidence from Azure

VSTS enables you to ship more quickly, more frequently and with more confidence, while extending and customizing your release pipelines, and controlling your deployments with automatic and manual gates for approval workflows. I’ll explore a release pipeline created using VSTS in the “practical continuous delivery showcase” section. You can learn more about the CD capabilities in VSTS at the VSTS Release Management Web site (bit.ly/2zcSrNE). Also see how Bing uses CD (bit.ly/2A4m8Q5) and explore the experiences of the ALM | DevOps Rangers as they implement release pipelines for their community projects at bit.ly/2zckUWc).

Generator-team, written by Donovan Brown (bit.ly/2zclmDS) is a powerful open source Yeoman generator that demonstrates the use of the VSTS REST APIs. It creates a complete CI/CD pipeline in Team Foundation Server (TFS) or VSTS for Java, Node.js or ASP.NET, and allows you to deploy to Azure App Service, Docker to private host, Docker images in Azure App Service on Linux, and Azure Container Instances.

The Team PowerShell module, written by Donovan Brown (bit.ly/2gZfzHg) exposes portions of the REST API for VSTS and TFS. (You can learn more about the REST API at bit.ly/2h1CWjf.) It’s an interesting option from the Ops perspective. It’s written in pure PowerShell and can be used on macOS, Linux or Windows to connect to TFS or VSTS.

By the way, a preview of a new Common Language Infrastructure (CLI) for working with VSTS on Windows, Linux or macOS was recently shipped. It’s currently geared toward developer scenarios like creating pull requests and updating work items, but will support a richer set of capabilities over time. The new CLI will also make it easier to automate interactions with VSTS using scripts. To learn more, visit Get Started with VSTS and TFS at docs.microsoft.com/vsts.

A Practical CD and Tooling Showcase

Let’s review a typical release pipeline created using VSTS, as illustrated in Figure 4. If you prefer to create your own instance of such a pipeline, read the in-depth paper, “CICD Pipeline for Your Extensions with Visual Studio Team Services,” before continuing. You can download the PDF from bit.ly/2hCAZKX.

Figure 4 View of a Typical Continuous Integration/Continuous Delivery Release Pipeline

Our journey starts with the team committing its code to a Git repository or service provider such as VSTS (Git and Team Foundation Version Control), GitHub, BitBucket or Subversion. When a developer commits changes to the team’s repository (1), continuous integration is optionally triggered (but strongly recommended) to check the associated pull request before you commit changes and after you successfully merge changes to a specified branch. VSTS not only automates your build, testing, and other validation tasks, it gives you an insight and traceability into everything in the build, including code changes, review comments, test results, and code coverage.

CD, started by the build, is the process of validating, configuring and deploying your changes to one or more environments. You can choose to have a predefined schedule and conditions to trigger the deployment for each environment in your release pipeline. In addition, you can define approvers for each environment before and after deploying an environment. Approvals can be set to be automatic or can be executed by one of many specific users, all users in any order, or all users in sequential order. When a group is specified as an approver, only one of the users in that group needs to act.

In the example for this article, we’re automatically processing the Canaries environment (2), which privately deploys the extension to the Visual Studio Marketplace (3). We’re using private extensions initially, so that they’re not discoverable on the public marketplace. Private extensions are in production for the customer, but only visible to the customer’s nominated VSTS accounts. See the previously mentioned white paper for details on the publishing process. It’s important to emphasize that your release pipeline could deploy to a variety of destinations, but we’re targeting the Marketplace as we are deploying a VSTS extension in this example.

CD proceeds with a parallel (forked) deployment to the On-Prem Validation (4) and the Early Adopters (5) environments. The former proceeds automatically, as indicated by the pre-deployment approvers icon () in the pre-­deployment condition. The latter introduces the pre-deployment manual approvers icon (), which shows that third-party approval is needed before the deployment can begin. In addition, the example specifies that the environment trigger fires only if changes apply to the master branch (6), effectively shielding production from other branches.

You can implement several control goals to progressively expose your changes to subsequent envi­ronments, keeping an existing (blue) version live, while deploying and validating a new (green) version. If an issue is discovered, the pipeline allows you to stop the deployment to minimize impact to end users.

CD continues with a joined deployment pipeline, where deployment to the Users environment (7) occurs only after successful deployments to both the Early Adopters and On-Prem Validation environments. See “Phases in Build and Release Management” (bit.ly/2zLrL71) for details on running tasks on different agents, manual interventions, and conditions under which tasks will process or stop your release pipeline.

We’ve used deployment rings to limit impact on end users, while gradually deploying and confirming change in the production environments. We evaluate the impact, sometimes called the “blast radius,” through observation, testing, diagnosis of telemetry and, most important, user feedback. The Canaries environment is the first stage in the ring deployment model we’re using to expose the private extension to a controlled user group to test the new version in production before a broader rollout. Again, the article “Phases in Build and Release Management” offers insight (bit.ly/2zLrL71).

As discussed, you can orchestrate deployments across multiple environments, using manual or automated approval gates, while supporting high availability for your application. Whether you’re targeting the cloud, the hybrid cloud, or an on-premises environment, VSTS is the recommended Azure DevOps solution. It allows you to plan, monitor, build and deploy to Windows, Unix, Android or iOS, from .NET, Java, PHP, Python, Ruby, C++, as well as from many other products such as Ant, Maven, Gradle, msbuild, Junit, NUnit, xUnit, MSTest and Jasmine. Figure 5 shows the range of available solutions.

Figure 5 Visual Studio Team Services Is Our Backbone Azure for DevOps Solution

Both TFS and VSTS support a rich integration model using the same marketplace extension. If you cannot find a task needed for your release pipeline, you can build custom tasks that integrate with your CI and CD. For example, we’re using the open source VSTS Developer Tools Build Tasks extension (bit.ly/2hcqopf) to package and publish our extensions to the Visual Studio Marketplace. Another example, the AWS Tools for Microsoft VSTS extension (bit.ly/2ybJYZZ), delivers tasks for Amazon S3, Beanstalk, CodeDeploy, Lambda and more. It enables you to orchestrate deployments that span both the Azure and AWS clouds.

Mitigating Risk in a CI/CD Pipeline

So far, I’ve explored how VSTS is a pipeline system with a simple, real-world example. The risk of failure grows rapidly as you increase the number and frequency of releases, and add dependencies and complexity to pipelines. Let’s look at a few techniques for gaining visibility into real-time status and mitigating the risk in your CI/CD pipelines.

Garbage in garbage out. The old saying applies to your release pipeline! A reliable and quality release pipeline depends on a healthy source to feed its CI and CD. You can leverage proven branching strategies, like those described in “Adopt a Git Branching Strategy” (bit.ly/2AoQQEA) and in “Branching Strategies with TFVC” (bit.ly/2yaqEMo). You can also employ branch policies, and pull requests to maintain a healthy and shippable master branch.

Figure 6 summarizes a few lessons from our DevOps transformation, documented by Sam Guckenheimer at bit.ly/2j7x0JY. For example, we work in three-week sprints (1), using this cadence as our common heartbeat to plan and deploy continuously. We manage debt continuously (2), to avoid the accumulation (or spike) of debt and resultant merge complexity (3) that we saw in the past.

Figure 6 Keep Your Master in a Good and Shippable State

The master branch (4) is our single source of truth, always in a healthy and shippable state. Teams use short-lived feature branches to isolate their work and submit a pull request when the feature is complete (5). On approval of the pull request, we merge the feature into the master branch. Teams repeat the process continuously for added work. The release isolation strategy (6) introduces one or more release branches from the main, enabling concurrent release management, multiple and parallel releases, and exact snapshots of our existing (blue) version at release time.

Pre-merge validation is important! Establish branch policies to ensure that the master branch meets your desired quality criteria. Always encourage developers to submit code changes via pull requests. You can selectively enforce policies, for instance that pull requests must have approval from a specified number of reviewers, or that work items are linked to improve traceability. Resolution of comments and a specific merge strategy upon completion are two more examples. Beyond the pre-merge validations, you can add build policies to automatically trigger a build that must succeed to complete pull requests, and invalidate builds when the master branch changes.

Security is paramount.It’s vital that security be confirmed continuously throughout the process, that applications are safe, and that data stays secure and private. Here are just a few of the ways that DevOps and an automated pipeline can work with CD to improve security:

• Simulate attacks and system stress during the build to detect vulnerabilities and failures. Take remedial action or block releases as necessary.
• Create a dynamic infrastructure to avoid a persistent place for vulnerabilities to hide. Quickly recreate the infrastructure and re-deploy new packages when vulnerabilities are detected.
• Automate tests for security and continuously monitor in production to ensure your solution is secure.

Check out the article “Adding Continuous Security Validation to Your CI/CD Pipeline” for an in-depth exploration of how security should be implemented in a CI/CD scenario at bit.ly/2j6GHIs.

Vet your open source components. Tools such as WhiteSource (whitesourcesoftware.com) can be immensely useful. It integrates seamlessly into the release pipeline and continuously checks the security, licensing and quality of open source components. The ALM | DevOps Rangers have been dogfooding this tool with all its VSTS extension release pipelines and shared its outcomes at bit.ly/2yD8yqU.

Use feature flags.Feature flags help keep feature branches shorter and let you integrate them into the master more often. Feature flags and the ring deployment model are symbiotic. Rings limit the exposure (blast radius) to predefined user groups, while feature flags provide a granular way to expose or hide a specific feature to all or selected users. Feature flags are invaluable for delivering value and receiving feedback on features easily, quickly and continuously.

You roll out code across the entire ring topology before you enable the feature flag. This allows you to release and test features while still in development, or to switch on verbose telemetry in production with a flip of a flag. See “Phase the Features of Your Application with Feature Flags” (bit.ly/2yDSepA) for a discussion of feature flags and A|B testing with our release pipelines, using the LaunchDarkly (launch­darkly.com) Software as a Service.

Pipeline health visualization. Visualize the health of your CI builds using the Team Project Health extension (bit.ly/2hgFk9A), which adds a visual cue to your dashboards informing you about the status of your builds and releases. Codify/build-light (github.com/Codify/build-light) is a Raspberry Pi-based solution that visualizes the health of your CI builds.

Greenlighting feature. New to VSTS is greenlighting, which intelligently automates approvals. It lets you define a set of gates in your pre- and post-deployment options that integrate signals from monitoring systems and other external services. Greenlighting not only visualizes your pipeline health, it uses gates to determine RED/GREEN/BLUE status based on real-time signal sampling.

Greenlighting can trigger an Azure function to ensure a successful completion, match work item query results to thresholds, and define a time and schedule, among a host of other things.  By greenlighting your pipeline, you enable automation and faster feedback loops. You’re able to focus on your deployment speed, supporting faster time to mitigate issues and faster time to value.

Wrapping Up

CD is an ongoing and endless process of innovation. Now that we’ve covered the concepts of CD, you should feel confident exploring ways to improve your release pipelines. It’s all about taking incremental steps to improve your process. Reducing time to test and time to deployment, minimizing integration debt, and raising release quality may feel like a daunting challenge. By taking small, early steps, it all becomes achievable. DevOps isn’t a destination, it’s a journey of continuous, rapid improvement.

Willy-Peter Schaub is a program manager in VSTS, working at Microsoft Vancouver in beautiful British Columbia. Since the mid-’80s, he’s been striving for simplicity and maintainability in software engineering. You can follow him on LinkedIn aka.ms/willysli or Twitter at @wpschaub.

Thanks to the following Microsoft technical experts for reviewing this article: Ryan Britton, Rui Carrilho de Melo, Edward Fry, Vijay Machiraju, Tiago Pascoal and Martin Woodward

Discuss this article in the MSDN Magazine forum