Matt Walburn is on a mission: to save you from building your own platform.
Prior to joining Pivotal, Walburn (featured left) was an engineering director for a popular Fortune 50 retailer.
In 2013, he started a multi-year effort to build an IT platform to run the company’s e-commerce and mobile apps in the public cloud.
The decision to build a platform from scratch was the right call at the time, Walburn notes.
“The project started out as a way for us to accelerate software delivery, to get new features to our customers much faster. We wanted to be more like a startup, and we needed a platform to do that,” he said.
“Unfortunately, we didn’t see a way to get there without doing it ourselves. Looking back, it was such a huge endeavor that there was never going to be enough time or people to really hit the mark.”
After two years of blood, sweat, and tears, Walburn and his team ultimately launched a functional platform. But the “do it yourself” journey is not one he would recommend. And certainly not in 2016.
In fact, Walburn spends much of this time espousing the exact opposite approach—buy a platform and focus your engineers elsewhere.
Can we focus on becoming full lifecycle engineers instead of full stack? Apps add value. Perfectly crafted containers and infracode do not
— Matt Walburn (@mattwalburn) August 16, 2016
Walburn recommends “building” projects that deliver value to the bottom line: the custom software applications that enable the company respond to new business opportunities faster.
“Software engineers are so valuable. You want an environment where they can think of a feature in the morning, then have it running in production later that day,” he says. “You need a full-service platform for them to use. And that platform should be bought, not built by your engineers.”
Why not build your own platform? Walburn grimaces and goes on to cite several lessons he learned first-hand.
1. It requires more components than you think. A lot more.
“Organizations want to put IT at the heart of their business, but rapid software development is new to them. You’ll find leadership that is so focused on containers and container orchestration, thinking that is a platform. But that’s an incomplete picture. To get the power of a platform, you need much more.”
How much more?
“The reference architecture for the platform we built had 5 areas: infrastructure, operations, continuous delivery, middleware & data, and security. Under each one of those, we had 7 modules. I’ll show my customers this chart, and ask ‘Are you thinking about all this?’ Very few of them have. That’s when the idea of buying a platform starts to become really attractive.”
The components of Walburn’s homegrown enterprise platform.
Container tech is necessary, but not sufficient, Walburn says.
“It’s easy to get carried away with all this hot tech. But when it comes time to run production applications at scale for a big company, you want your platform to be boring and stable. You can focus on business transformation when the underlying platform is hardened and just there like a dial tone.”
The most commonly overlooked areas? The so-called ‘day 2’ problem—the features that help run the software in production.
“You have to consider the operational perspective: how are you going to consistently deploy and automate infrastructure? What about monitoring, capacity planning, and financial reporting? These were important considerations for our team at my last job, but not everyone thinks about them.”
2. The pieces you need to build a platform were not meant to work together.
All told, over 40 products from dozens of vendors were used in the “do-it-yourself” platform. Walburn’s team selected solid open source tech with good APIs and useful documentation. But that doesn’t mean connecting them was fast or easy.
The vendors and products that can be used in a do-it-yourself platform.
“The team spent so much time integrating all the parts and pieces, supporting them with a ton of custom code. Everything worked at the launch of the platform, but the investment required to maintain it all was growing quickly.”
3. If you succeed in building the platform–congratulations! You’re at the starting line, not the finish line.
Walburn reminds customers that it’s not enough to just build a platform. You have to run it and maintain it. He recalls going live in production.
“At last, after two years of development, we were ready to get started modernizing our apps. But then we had to think about product planning and developing the next release. We had to consider the upgrades and lifecycle of everything that was in production. That was enormous complexity to manage.”
Buy a platform, and make these issues someone else’s problem, Walburn advises.
“When I talk to customers about Pivotal Cloud Foundry, I tell them we have hundreds of engineers building it, maintaining it, ensuring all the services work together in a way that’s invisible to the developer and the operator.”
There’s the roadmap of the platform, too. Once developers and operators start using it, they expect it to get better over time based on their feedback to the platform team.
4. You cannot keep up with new requirements.
It’s one thing to work with users of the platform on incremental improvements that boost individual productivity. It’s quite another to digest sweeping changes that executives sometimes mandate for technical teams. Walburn’s experience is common.
“We had a new CIO come in. He wanted to maximize the return on some of our existing data centers over the next decade. That meant we had to use our on-premises resources as our baseline capacity, then consume public cloud for spikes in traffic.”
Walburn’s team had planned for some elements of platform portability across infrastructure. But a shift to this type of architecture wasn’t something that could be done quickly, or cheaply.
It was at this moment Walburn knew the homegrown platform wasn’t up to the task of serving the business.
5. Do you want to start your transformation in two years, or tomorrow? That is what “build vs. buy” really means.
Walburn notes that his peers today have an advantage that he didn’t in 2012—mature platforms to choose from.
“This is the platform era. If something like Pivotal Cloud Foundry was around in its current form in 2012, it would have been an easy decision go with that. Many of our requirements at the start of this whole effort with the same as Cloud Foundry—be cloud agnostic, embrace microservices, optimize for speed and self-service.”
He cautions that there will always be the temptation for developers to “do it themselves.” The popularity of open source software and cheap computing power can fuel this mentality.
“Engineers can get overzealous, and think ‘I can build the perfect thing for us.’ Business leaders need to keep the development teams honest. Get your team focused on time-to-market, time-to-value, and impact to the business.”
This framing, says Walburn, can be illuminating.
“When you stay focused on the backdrop of what needs to happen in 12 months, 24 months, the decision becomes pretty obvious. You should just buy a platform, and get on with transforming your business.”
Walburn notes with pride that his effort did have an enduring impact: the IT culture at his former employer now resembles that of a startup: DevOps, lean thinking, and an agile mindset. Those skills will serve the company in the years to come, even if the platform he built will not.
Now, he helps other enterprises avoid a similar fate.
“Building and running a platform wasn’t our core business, it was never going to be. This is a universal truth for companies that want to transform.”
About the Author
Follow on Twitter Follow on Linkedin More Content by Jared Ruckle
