Data is Big, the predictive enterprise is the way of the future, and data scientists are in high demand: you can’t glance at technology news sites in 2012 without being aware of these developments. But there’s another challenge facing organizations as they deal with the influx of data, one which receives less attention: a lack of the custom applications, skills, and development methodologies necessary to tap into its value.
William Davis, Greenplum’s Senior Product Marketing Manager, characterizes this as Big Data’s “last mile problem.” With the acquisition of Pivotal Labs, Greenplum is now well-equipped to address their customer’s ever-changing needs.
Founded in 1989, Pivotal Labs is an agile software development and consulting company that has made its name on building amazing software for customers using innovative development and collaboration methodologies. The company’s emphasis is on delivering high-quality custom software developed in close collaboration with clients in real-time.
“When customers have needs that aren’t being met by off-the-shelf software, we come in and build them exactly what they want” says Parker Thompson, Director of Business Development at Pivotal Labs. “We do it really fast and at really high quality, and work with their team to transfer those skills in-house.”
“Our approach is ‘look, let’s embed our team with your team, let’s start off with what you know you want now, and if the problem changes while we’re building this thing and the application needs to change to reflect that, our goal is to build the right thing for your business.’”
There are two tightly intertwined components to Pivotal’s approach: the first technical, the second collaborative. “We build great software and do it every time,” Thompson says, a result of “the technical and team practices that we bring to bear on these projects.”
“On the technical side, Pivotal is really informed by agile development: test-driven development, pair programming, continuous integration, and refactoring,” he says. “You hear many companies saying that they’re doing agile, but they aren’t really doing what Pivotal does: they’re not doing pair programming, they’re not delivering actual user-facing software every single day.”
Pair programming is key to Pivotal’s methodology. The agile development technique divides a team of coders into groups of two, who work on code together at a shared workstation. This structured team methodology is “both a technical and non-technical process,” Thompson notes.
Parker Thompson and colleague pair programming.
Pair programming necessitates ongoing communication and empowers team members to think about software architecture and execute dynamically. “Through this process you’re solving problems, and preventing bugs and architectural missteps in real-time, while you’re building, rather than learning about it six months later.” The methodology’s focus on communication and efficiency also reduces the risk of knowledge silos. “We don’t want to have a front-end guy, a back-end guy, and an API guy” working without knowledge of the larger team’s activity, Parker notes.
Pivotal aims to not only empower the coders building the custom software, but the entire client organization. “We sit down with customers, and break everything down into a list that is written in plain English, customer-verifiable, and at a high level of granularity,” Thompson says. “In practice, it’s possible to be non-technical and still be 100% driving the work, and know what the team is delivering every day. It’s about taking those communication and collaboration knobs and ‘turning them up to 11.’”
Pivotal’s methodology is reflected in Pivotal Tracker, a project management tool the company developed to support internal projects, and has since released to the public as a SaaS application. “Pivotal Tracker is sort of the software embodiment of our development methodology around how we organize development teams,” Thompson explains. “It allows a team to get behind a single set of priorities and execute as one.”
Thompson jokingly describes Pivotal’s approach as “the Spider-Man strategy.” When working with the company, a client is “going to have great power and also great responsibility,” he says. “We’re giving you the power as long as you’re going to be really engaged and driving this work. We’re working with clients to train them on how to be a good product owner, and really break stuff down to a granular level.”
It’s a philosophy that has secured high-profile clients including Twitter, Best Buy, Groupon, Salesforce, EMI, the Associated Press, Linden Lab, and even prior to EMC’s April acquisition of Pivotal Labs, Greenplum. Pivotal Labs collaborated with Greenplum on the development of the agile data science platform Greenplum Chorus.
The acquisition is both competitive and serendipitous. As Greenplum co-founder Scott Yara told GigaOM earlier in the year, numerous companies have boasted of their goal to become the “Pivotal of Big Data”. Since coming into the EMC Greenplum fold, Pivotal Labs is on track to become just that. But Pivotal’s core methodologies and principles remain the same: agile development through collaboration and a start up-inspired culture based out of loft spaces in San Francisco and New York City that stays tightly connected through employee perks like free catered breakfast and happy hours.
According to Greenplum’s Davis, Pivotal’s skills and agile methodology addresses a key problem shared by many of Greenplum’s customers. “We’ve found that a lot of customers need help productionalizing the insights they’re generating through the use of Greenplum’s platform,” he says. Using Pivotal’s approaches and skills, “We can provide them with a development engine that can take the predictive models performed in Greenplum’s Hadoop distribution or Greenplum Database and build a front-end or delivery mechanism for putting those insights to use for the business”
“The application space is the last mile in Big Data,” Davis states. “We’re going to work with companies that are looking at our platform, with Pivotal Labs, to develop these next generation applications that leverage Big Data.”
As Pivotal moves into the Big Data space, its core values and methodology remain. “We take this really seriously,” Thompson says. “We give clients an experienced team that knows the technology, the practices, and how to deliver on this promise.”
About the Author