Does Your Platform Engineering Team Have a Train Driver? Here’s Why You Should Consider this New Role.

October 30, 2018 James Wynne

At Pivotal our mission is to transform how the world builds software - supporting software in production is an important part of that. In fact, Pivotal’s Cloud Ops team aims to become the ideal operator of Pivotal Cloud Foundry. This post is the latest in a series that shares the team’s learnings. Here, we explain the role of “Train Driver,” and how it has helped the CloudOps group at Pivotal improve the life of our on-call engineers.

Here’s What the Train Driver Role Helps You Achieve

When you change an IT system, problems can occur. In CloudOps, we categorize these problems as a car crash or a barge crash. A car crash is a catastrophic incident. It happens almost immediately after the change has been applied. In contrast, the barge crash is an incident that happens a significant period after the change. These are chronic failures; like two slow-moving barges inching closer and closer together.  

When these crashes happen, you want your teams aligned to get the issues fixed. But there’s a common organizational problem that gets in the way. In many companies, there’s a troubling disconnect between those who implement changes, and those whose role it is to be on-call. The roles may exist in different orgs and departments. They are measured differently, with different incentives that often conflict.

The train driver can mitigate the car crash and barge crash scenarios, while aligning engineers across your organization. Here’s how...

At Pivotal, the Train Driver is the person coordinating and executing platform updates, as well as holding the pager for a given week. Previously, this had been two separate roles. This new way, the Train Driver links the deployment phase to the support and on-call phase. Maximum context is retained. In the event of a failure, the person who has most context about the change is the person tasked to remedy it.

So what does the Train Driver do? What makes this role effective? As we’ll see, it has a fair amount in common with actual train drivers in the real world.

The Schedule - Plan Your Changes in a Manageable Way

Before a train can leave the station, the driver must be familiar with their cargo and the route. The driver must be aware of the risks associated with the cargo. Furthermore, the driver must know how long each trip should take. In our CloudOps team, this means that the Train Driver (who is typically engaged for a week) will look at the changes and upgrades to be done. They estimate how long each change will take, and allow extra time (just in case!) for those changes with an element of risk. Most importantly, only one change will be made per trip. This approach aligns with the agile principle of small iterations rather than “big bang” changes. An added benefit: it simplifies the root cause analysis of any problems that may occur.    

The Daily Report - Build a Body of Evidence to Make Informed Decisions

A friend of mine drives actual trains. He described the daily report used to catalog the day’s events:

“...All occurrences, incidents, accidents, faults, delays etc have to be reported into the traffic events database. This gives a total daily report...”

In CloudOps, we use something similar: a deployment report. The report details any problems encountered during a deployment, and the corresponding troubleshooting steps. Other important details are recorded, like the time spent applying a change. We also catalog the emotional experience of the deployment. Was it stressful? Confusing? Easy?

You can imagine how useful the report is. For our team, it serves two important functions:

  1. It adds to our body of evidence. So future Train Drivers can build more sustainable and realistic schedules as a result of this data.

  2. We can send actionable and real world feedback to the development / product teams regarding the installation and upgrade experiences. (CloudOps at Pivotal use a lot of Pivotal software. Part of our role is to provide feedback to our product teams. In your organization, this might not be as important or even possible.)

Section of our deployment report.

The Whistle - Start the Upgrade with Confidence

Before the deployment starts, final checks are made. If the train driver is happy, we hear the iconic whistle blow. But the whistle only blows if certain conditions are met.

We are guided by Site Reliability Engineering* (SRE) principles. So before the whistle is blown on a given day’s deploy train, the Service Level Indicators (SLIs) are checked against our Service Level Objectives (SLOs) (we describe these concepts in this handy blog post). The Train Driver then makes the final decision whether the deployment train should depart. If deployments earlier in the week have eaten into the error budget, the train may be suspended until we have more budget in reserve. When this happens, the Train Driver will try to focus on increasing reliability until the team has the budget to allow the train to leave the station again.

The Conductor - Achieve Success with a Continuity of Knowledge and Experience

Here at Pivotal, we pair program each day. There is always a Conductor to help the Train Driver. The duties of the Conductor are much the same as that of a driver (one key difference: the Conductor may change on a day to day basis).

But the role of the Conductor is of utmost importance to the team in two key scenarios:

  1. At the end of each week, the person taking up on-call and train driving duties in the following week, pairs with the current Train Driver. This allows a hand over of context of all of the week’s deployments. Now, the new Train Driver can take over on Monday feeling comfortable with the state of affairs. And if there’s an unexpected barge crash, they will be ready!

  2. The Conductor is also a useful role for new members of the team. By pairing with the Train Driver, they can get uninterrupted time with a more seasoned CloudOps engineer. This time can used to explore new tech and investigate legacy issues. Most usefully, the shared experience prepares the new team member for going on-call!

The Hat - Make an Important Role Feel Rewarding and Special

Finally the hat! When you have a Train Driver on your team, you need to instill your engineers with the confidence to lead. The performing upgrades and being on-call are never easy. In fact, they are normally associated with hardship and strife. Being a Train Driver and wearing the hat is reflection of respect and confidence.  

Shane (Anchor Cloudops-EU) and some of our guest Train Drivers.

Stay Tuned for Part 2, Where We Examine the Role of a Train Driver in a World of Platform Automation

This post describes a grand experiment - where a designated train driver made all platform changes. The trial ran from May 2017 to April 2018. During this time, we built up a shared familiarity and experience when it came to manual platform upgrades and handling incidents. In April 2018, a big change happened -the CloudOps team automated all of our platform upgrades, by converting them to Concourse pipelines. Was this the end of the Train Driver? Anything but -  who would give up such a fancy hat!? In our next post, we’ll review how the role of the Train Driver changed as a result of our team’s move towards automation.

In the meantime, check out these links for more useful reading on this topic.

About the Author

James Wynne

James is a Staff Engineer with Pivotal Cloudops-EU. His day job is to keep the foundation hosting Pivotal Tracker running. He has 20 years of experience in support, infrastructure and operations; mostly in the mobile wireless space, but has also worked in aviation, media and financial services. He is known to love all things German and is an avid BvB fan.

More Content by James Wynne
Previous
How to Measure Software’s Ability to Deliver Value
How to Measure Software’s Ability to Deliver Value

A dive into why an organization should take Pivotal’s Benchmark survey, which measures how well an organiza...

Next
Audit Trails, New GUI, Kubernetes CronJob Integration, Streaming Application DSL, and More! Spring Cloud Data Flow 1.7 is GA.
Audit Trails, New GUI, Kubernetes CronJob Integration, Streaming Application DSL, and More! Spring Cloud Data Flow 1.7 is GA.

Spring Cloud Data Flow (SCDF) is a toolkit for building data integration and real-time data processing pipe...