An Overview of Loggregator CI Pipelines

August 20, 2014 Johannes Petzold

Cloud Foundry at Pivotal recently set up new continuous integration pipelines for loggregator, using Thoughtworks’ Go (not to be confused with Go, the programming language). This is the first in a planned series of related blog posts; its intention is to provide an initial overview of our pipelines.

Our code lives in several git repositories, linked together via git submodules (see the following illustration). Also illustrated are our main pipelines and their sequence of execution:

loggregator

Loggregatorlib & Consumer Pipelines

These two pipelines are triggered whenever there is a change in their respective repo (lib or consumer), and they each execute the following steps:

  1. Run unit tests for their respective repo
  2. On success, bump submodule reference in loggregator to the tested lib/consumer version.

This will implicitly trigger a run of the loggregator unit tests pipeline.

Loggregator Unit Tests Pipeline

This pipeline runs whenever there is a change in the loggregator repo (on the develop branch, see below). It executes these steps:

  1. Run unit tests for the loggregator repo
  2. On success, promote tested loggregator version into the master branch. The loggregator repo’s default branch is develop; as the name implies, all development happens on this branch.

This will implicitly trigger a run of the integration tests pipeline.

Loggregator Integration Tests Pipeline

This pipeline runs whenever there is a change in loggregator (master), or in cf-release.

  1. Deploy loggregator (from master branch) to a dedicated bosh-lite VM in AWS. Since loggregator is part of cf-release, this also deploys the latest version of cf-release.
  2. Run loggregator end-to-end integration tests.
  3. Store tested versions (loggregator and cf-release) as Go-internal artifacts, for consumption by the acceptance pipeline.

Loggregator Acceptance Pipeline

Each of this pipeline’s two steps is triggered manually, typically by our PM during story acceptance:

  1. Deploy loggregator + cf-release to our acceptance environment in AWS. Use the versions from the most recent successful integration test run. Once the deployment completes, manual acceptance steps can be performed in the acceptance environment, e.g. confirmation that new features work as expected.
  2. If everything looks good, bump the loggregator submodule reference in cf-release. This will send the changes in our repos through other teams’ pipelines and eventually into production.

What’s Next?

In future blog posts, we plan to write in more detail about Cloud Foundry-related topics such as:

  • Docker: why and how we’re using it
  • Reducing time for bosh-lite deployments
  • How our pipelines pass versions along and use git to merge and rebase

About the Author

Biography

More Content by Johannes Petzold
Previous
Do the Google Hangout Hop
Do the Google Hangout Hop

Bit.ly + static hangout links = quick feedback loops. Quick feedback loops are core to the Pivotal process....

Next
How To Win In Mobile App Development, Quickly
How To Win In Mobile App Development, Quickly

With over a billion smartphones, which 90% of people keep within arm’s reach at all times, businesses must ...