Howard Dierking

Our Development Pipeline and Process

I’ve been going around to different teams inside of my company recently and talking about how we’ve gone about the task of selecting different tools and technologies for our projects. I figured that if this many people inside the company were curious, there may be some value in sharing it with you good folks as well. So here, I’ll attempt to cover the principles, strategies, and technologies for both our development process/pipeline and our projects themselves. And as always, I would love to hear your experiences - especially if your experience provides insight that can help us further improve the ways that we do things.

Our Projects

My team has projects that go in a few different directions, but at a high level, we have 2 buckets of work. The first is developer experience. This includes everything from the developer portal and documentation Web site to client libraries in a variety of different languages. The second bucket cloud services. This particular work stream is not exclusive to us - there are a variety of teams around the company that are working on various Web services and even some that are positioning those services to run in the cloud. However, our team is unique in that we have taken a “public cloud-first” approach to everything that we build and as a result have been one of the first teams in the company to run a production service in the public cloud (we use AWS).

As I’m sure you experience on an hourly basis, the number of technology options for any and every endeavor are almost innumerable and expanding rapidly. As a result, we first came up with a handful of principles for technology selection to put some rigor around our decision making process. These principles have thus far been a helpful guide as we’ve evaluated technology both for our projects as well as our internal development pipeline tools.

One example of these principles in practice is our developer portal and API documentation site. By focusing on selecting the right tool for the right job and investing wisely, we realized that continuing to run our static documentation Web site on a highly customized content management system running inside of our data center was effectively waste. Firstly, it was technically wasteful to run a static Web site off of a CPU-bound content management system where every request to the origin server required dynamic calculation of HTML. On top of that, the content was stored in a relational database, which is an additional CPU-bound resource that provided no tangible benefit to the problem that the system was trying to address. Secondly, all of the content itself was trapped inside this relational database and reachable only using the CMS’s proprietary user interface. This went against our principle of substitutability. The solution? We extracted from the relational database, converted it to markdown and are about to launch the next version of the Web site as a completely static Web site using Jekyll and GitHub pages. This technology choice helped us to achieve 2 additional benefits regarding the documentation Web site. First, having the site hosted on GitHub enables us to more deeply engage our developer community. Find a bug in the documentation? File an issue on the GitHub site - or better yet, send us a pull request! Second, using Git as our backing store for the site gave us something that very few content management systems actually offer (despite the name content management): version management of content! The power and flexibility that Git provides enable management scenarios from roll back to working branches for larger-scale changes. And by using Jekyll for generating the static site, we can easily provision staging and test versions of the site - whether on a local machine, an S3 bucket, or anywhere in between where files can be stored.

One other example of the principles in context of a technology choice that is very active at the moment relates to Docker. We believe that containers are certainly the path forward for many of our projects. However, our current developer environments are OSX and that operating system does not many capabilities that are required to run Docker development workflows natively. For example, I need to be able to, at a minimum, launch my container and mount the source code directory from my host as a volume in the container so that as I edit my source code, the changes are reflected in the container. On OSX, which with applications such as boot2docker, the interaction between the host and the container goes across a hypervisor boundary, adding a non-trivial amount of friction to the process. As a result, while we are keeping a very close eye on what is happening in the Docker (and Rocket) space, we have not yet moved forward with using Docker for our production software. When we can have a symmetric experience across development and deployment environments, we’ll make that move.

Our Pipeline

While the architecture differs, most of our projects follow a similar development pipeline, which looks like the following.

Developer workflow and tools

Here’s a high level description of the moving parts.

Shippable pull request status

To give you a better sense of a Shippable CI configuration, here’s the shippable.yml files from one of our projects.

# only trigger ci for dev branch
    - dev

# language setting
language: node_js

# version numbers, testing against two versions of node
  - 0.10.35

# The path for Xunit to output test reports
    - XUNIT_FILE=shippable/testresults/result.xml
    - SLACK_ORG=ConcurCTO
    - PROJECT=my-project

# Update NPM
  - npm install -g npm@2.5.1
  - ./build/ $NPME_AUTH_TOKEN

# Create directories for test and coverage reports
  - mkdir -p shippable/testresults
  - mkdir -p shippable/codecoverage

  - grunt ci-shippable
  - ./build/

# Tell istanbul to generate a coverage report
  - ./node_modules/.bin/istanbul cover grunt -- -u tdd
  - ./node_modules/.bin/istanbul report cobertura --dir  shippable/codecoverage/

  - python ./build/ --project $PROJECT --org $SLACK_ORG --token $SLACK_TOKEN

  - python ./build/ --project $PROJECT --org $SLACK_ORG --token $SLACK_TOKEN -s
  - ./build/

Finally, the CI environment is just UNIX - so you can script out whatever you want as a part of your process. There’s no waiting for someone else to implement some hook or feature.

mochaTest: {
  'ci-shippable': {
    options: {
      reporter: 'xunit',
      captureFile: 'shippable/testresults/result.xml', // Optionally capture the reporter output to a file
      require: 'should'
    src: ['test/**/*.js', '!test/agentTests-perf.js']

Like I have probably mentioned a few times, there are still some areas where we are investing in to continue improving our process and tooling.

The thing I like about what we have at present when I think about some of these future aspirations is that the principles above actually give us the ability to move forward quickly on any of these without really having to wait for some big, vendor-specific piece of software to add capabilities. When we find something that will move our workflow forward, we’ll add it. If we later decide it’s not helping, we’ll remove it. If we find something better, we’ll replace it.