Semantic, Yet Flexible: BOSH Release Versioning

July 21, 2014 Karl Isenberg

featured-BOSH-versioningThere are many different version formats to choose from, but sorting, comparing and incrementing requires a known format and a set of semantic rules. To support the complexities of modern release practices, BOSH now supports semantically versioned releases. Ultimately this makes BOSH more flexible by allowing release creators to branch and patch their releases in a way that matches their development patterns, rather than forcing them into a linear, low-information versioning pattern.

Hopefully, you’ve heard of Semantic Versioning (SemVer) by now. If not, SemVer is an increasingly popular versioning standard born from the pain of “dependency hell” that comes with managing an increasing number of projects and dependancies. Unfortunately, SemVer doesn’t satisfy everyone’s needs, but it is significantly better than a single number. So we used SemVer as a basis for BOSH’s new versioning support, with a twist.

Little Known Facts

Most people know SemVer supports “<MAJOR>.<MINOR>.<PATCH>”, but not everyone knows that it also supports “pre-release version” and “build metadata”. Unfortunately, the format of SemVer isn’t quite as well named as one would like—everything’s a “version”!

For improved clarity, I’ll be using “primary segment” for the “<MAJOR>.<MINOR>.<PATCH>” segment, “pre-release segment” for the pre-release version, and “post-release segment” for the build metadata. For additional specificity, each of those segments may include multiple period separated components.

Semantic Version Format:
<primary.segment>[-<pre.release.segment>][+<post.release.segment>]

Pros & Cons

SemVer has a lot going for it. It’s the culmination of years of learning from managing dependencies, each with their own unique snowflake version number formats and edge cases. SemVer holds your hand and tells you how to version. It saves you from learning the hard way.

Why everyone loves Semantic Versioning:

  • Driven by code changes
  • Allows reasoning about reverse compatibility & new features based on version
  • Well defined sorting
  • Well defined equivalence
  • Allows for abstract version handling without project-specific business knowledge
  • Supports pre-release designations, like alpha, beta & release candidate
  • Support post-release metadata, like build, hotfix, & revision
  • Allows for branching, non-linear versioning

Unfortunately, the rigidity of SemVer can get in the way when confronted with the complexities of non-development requirements or modern branching practices.

Why no one loves Semantic Versioning:

  • Driven by code changes, not marketing or business requirements
  • Exposes breaking of reverse compatibility to users, even if it only affects a rarely used edge case
  • Adds significant complexity to version incrementing, which is hard to automate
  • Allows for branching, non-linear versioning
  • Doesn’t specify how to handle patches of patches
  • Doesn’t specify how to handle 3rd party modifications (ex: by redistributors)

As you can see, it’s a bit of a love-hate relationship. While being able to reason about what a version change means is a useful feature, the cost is flexibility, and not everyone wants to pay that price.

This Is Where BOSH Comes In

While developers may want to individually use SemVer to version their projects, as platform developers, we didn’t want to require BOSH users’ projects to use SemVer.

One of our primary requirements, for example, was a desire to continue supporting single number versions, as BOSH has since the beginning. Additionally, since Cloud Foundry is our primary in-house user, we wanted to enable more flexible versioning for our enterprise customers who have been adding their own proprietary patches and re-versioning for internal use. Internally, we also have many teams contributing to shared releases who would like to version their forks and development branches independently. Any organization of a certain size will certainly run up against the same issues.

In effect, BOSH doesn’t really care what your versioning means, only that it can be compared for equivalence, sorted in a deterministic order, and incremented when creating a new release. So when researching existing parsers and patterns, we looked for something that could support SemVer, but also allow longer, more flexible versions.

We came up dry…

Introducing Semi-Semantic

So we wrote a new version parser called Semi-semantic.

Semi-semantic can parse and compare SemVer with ease, but it also supports the expanded and backwards compatible syntax our users want. One of the primary concerns is support for unlimited components in the primary version segment (ex: 174.0.0.1.2 or 174 instead of just 174.0.0), and alphanumeric components in the primary version segment (ex: 174.cf.1).

These new features, of course, come with their own trade-offs. With unlimited components, we now have to worry about equivalence between segments with a different number of components (ex: 174.1 == 174.1.0). With alphanumeric components, we now have to worry about how to increment alphanumerics.

For equivalence, we went with right alignment. You can append or remove components on to the right side, but you can’t prepend components or sorting with old versions will break.

For incrementing, we simply require that your least significant digit be numeric, if you want BOSH to increment it for you. On the other hand, if you want to specify your own version, you can now do that with the –version argument (ex: `bosh create release –final –version 174.0.1`). If you reeeeaaaaaaly wanted to (please don’t), you could even have your primary version segment end in an alphanumeric (ex: `bosh create release –final –version 174.2.abc`). It would still sort relative to other versions, but for every final release you would then have to explicitly specify what version to create.

As often happens, we made some tradeoffs to meet our requirements. On the plus side, final releases of the old format are still supported and reverse compatible. Users of an old bosh_cli will still be able to interact with an upgraded director. Yay!

The down side? The old dev format (ex: 174.2-dev) doesn’t conform to SemVer or Semi-semantic. So we have to do some pre-parsing in BOSH to transform old-style dev versions to new-style versions (ex: 174.2-dev => 174+dev.2). As much as possible, we tried to make this translation transparent and “just work”, but as with most changes of this type, there’s no way to smooth all the rough edges. If your current dev release version uses the old format, your next auto-incremented dev release version will be in the new format.

How Do I Use Semi-Semantic with BOSH?

The primary usage of BOSH is to deploy a distributed system of release components to the cloud (or virtual machines). Normally, this involves four steps:

  1. Create a new release version
  2. Upload the release to the BOSH Director
  3. Create a deployment manifest to describe how to compose components from the release to form a distributed system
  4. Deploy the specified distributed system

In order to define and package a release, the release must have a name and a version. Creating a release creates a tarball that includes all the release components, and also decides what version the release should have.

Final Releases

Previously, BOSH dictated what the next final version was going to be. It incremented from the previous release version, as recorded in the local release index (releases/index.yml). With the new versioning changes, BOSH allows you to specify a version.

# Specify a version
# (ex: 174 => 174.1 or 174.9 => 175.0)
bosh create release --final --version 175.0

Letting BOSH auto-increment still works, but now since there may be multiple components in the primary segment, BOSH only increments the least significant one.

# Increment automatically
# (ex: 1 => 2 or 174.1 => 174.2 or 174.1+dev.42 => 174.2)
bosh create release --final

Development Releases

As before, BOSH still manages development versions. However, since the old dev suffix format is incompatible with semantic versioning, BOSH now converts it into the new format before incrementing.

# Increment automatically
# (ex: 174.1 => 174.1+dev.1 or 174.1-dev (old format) => 174+dev.2)

bosh create release

Prod Patches

Old versions occasionally need patches. Previously, new forks of old releases could only be made by creating a dev release, but with semi-semantic versioning it becomes significantly easier.

# Add or increment a new primary segment component
# (ex: 174 => 174.1 or 174.0.0 => 174.0.1)
bosh create release --final --version 174.0.1

Pre-Releases

The software industry loves pre-releases. Alpha, Beta, and Release Candidate (RC) are all examples of commonly used pre-release identifiers. BOSH now supports using these in “final” releases. While calling a pre-release “final” may be confusing, the point is that they both get released to a wider audience, and thus commonly require more flexible versioning logic.

# Promote from development to pre-release
# (ex: 174.9+dev.42 => 175.0-alpha.1)
bosh create release --final --version 175.0-alpha.1

By default, –final truncates the pre- and post-release segments and increments the primary segment. For pre-releases, the primary version segment should have already been incremented. To make a final release, following a pre-release, you must specify the new version.

# Promote from pre-release to final release
# (ex: 175-RC.2 => 175)
bosh create release --final --version 175

Forked Releases

Normally, we would recommend changing the release name if you want to have a long running fork of a release, but sometimes it’s easier to just apply a small patch set on top of a new release. In this case, it’s now possible to use the post-release version to indicate the version of those patches.

# Append a new post-release version segment for a final release
# (ex: 174.1 => 174.1+p.10.0)
bosh create release --final --version 174.1+p.10.0

While you can use the post-release segment for final releases, you can still only have one post-release segment. To preserve these custom post-releases, you create a new dev release and BOSH only increments the least significant post-releases component. For example, a dev release based on “174.1+p.10.0” would get the version “174.1+p.10.1”. While this works, we don’t expect it to be a very common case.

For Best Results

Grab the latest bosh_cli and redeploy your director with the latest bosh stemcell for your environment.

Learning more and getting involved:

About the Author

Biography

More Content by Karl Isenberg
Previous
Demo This! How to ROCK your product demo
Demo This! How to ROCK your product demo

In June, I was honored to be a guest instructor for the NYC Big Apps participants. I chose to speak about s...

Next
Pivotal People—Andrew Clay Shafer, DevOps Polymath, Joins Pivotal
Pivotal People—Andrew Clay Shafer, DevOps Polymath, Joins Pivotal

We are pleased to announce Andrew Clay Shafer as a welcome addition to the Pivotal team. A ‘start up person...

How do you measure digital transformation?

Take the Benchmark