“If I was forced to only use the product on my iPad, I would want to put a bullet in my head.”
The user interview was conclusive. The feedback couldn’t be any clearer, and we had heard this from more than one user that day – iPad-only was not the way to go. Problem was, we were already three days into developing the product for iOS. We needed to change course – but at what cost?
Conclusively arriving at the wrong decision
The second Wednesday of the engagement we brought together 7 stakeholders, laid out the facts as we saw them, and near-unanimously agreed to ultimately go down the wrong track. The Wednesday after that we had the aforementioned user interview. How were we so misguided about our decision only a week before?
In retrospect we were making the classic mistake of building for a persona that doesn’t exist. Our key deciding factor was “mobility,” the idea that a user would want to use the product anywhere, regardless of the cell phone service. We envisioned giving users iPads (we had a captive user group, so iOS vs. Android and iPad market penetration were not concerns) and these users would use the application in the back of taxicabs, on the subway between client meetings, in abandoned industrial lots with no electricity, you name it.
At the point of our decision, we had only conducted one user interview, and for whatever reason had neglected to ask anything that would address our platform concern. The “on the go” user belief still stood tall. We knew we lacked the necessary information, and we knew this was a big concern, so why decide to decide?
Design was split between web and iPad. Without putting a stake in the ground, we couldn’t progress in our designs. And while having arbitrary deadlines is sometimes helpful to keep the team on track, oftentimes it can lead to rash decisions like the one we made.
But, with 7 people in a room all agreeing together, it seemed like we had made a defensible choice.
The truth dawns on us
At Pivotal Labs, we try not to work past 6 pm. We believe in sustainability, that willpower is a finite, depleting resource (http://en.wikipedia.org/wiki/Ego_depletion). For developers, coding late into the night makes for a higher error rate and less robust code that needs rewriting in the morning. For product managers and designers, making decisions after 6pm can lead to emotionally-charged, desperate conversations about the state of the project.
Turns out adults and toddlers aren’t so different. We both become better people after naptime.
The Tuesday after our decision, we had one of those desperate early-evening conversations about the iPad decision. The client team comprised 3 people, only one of which was in the initial platform conversation. The one client kept repeating to the others “if you were only in the room, you would be on our side,” but as the talk continued, that line became less of a declarative statement and more of a doubtful platitude. Nobody was saying iOS-only was wrong, but everybody saw pitfalls in a more limited platform than the web.
The next morning, bright-eyed and with two user interviews scheduled for that day, we decided to mitigate our fears by attacking the platform question head on.
Our fears were confirmed instead.
Donald Rumsfeld once had a much-maligned quote that surprisingly succinctly sums up the types of knowledge you get out of user interviews, needed when building a product:
“…There are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns — the ones we don’t know we don’t know.”
The user interviews that day had illuminated an unknown unknown, that our users were not necessarily individuals, they were firms. And firms, employing multiple people, could not run their entire businesses using an iPad – the medium was far too limiting for small businesses.
Our “on the go” user was an illusion; the “work at your desk alongside coworkers” user was reality.
It… wasn’t that bad of a change
We gathered the entire client team in a room (lesson: always have all decision-makers present for less-reversible decisions), laid the facts out as we now saw them, and decided to pivot. But what of all our iOS-focused research and wireframes?
Turns out the impact was actually pretty minimal. Like good Agile practitioners, we plan our development backlog in user stories, which are small, concrete tasks that users want to accomplish. User stories are naturally user-focused, describing functionality that should be available to the user, and specifically refrain from addressing implementation details.
So with great trepidation we laid out our prioritized set of user stories (on index cards) to assess the damage, and… we had to add a login page. That was it. Changing the platform, one of the biggest technical decisions we had to make, had practically no effect on our product backlog.
By using a business value focused backlog tool such as user stories, we were able to separate product decisions from technical decisions almost entirely. We could have been using a TI-83 calculator as a platform and it wouldn’t matter – because the business logic did not impose on the technical decisions, we were able to pivot immediately and start work on a web version without trashing any of the user research and product work done to date.
What being Agile really means (minimizing future opportunity cost is the mantra)
“The biggest source of waste in a startup is building something that nobody wants.” – Eric Ries
Sure, we lost a few days of development work, but from an Agile mindset that’s just the cost of reducing product risk.
Where companies struggle with Agile methodologies most is in recognizing and truly internalizing the concepts of opportunity costs and sunk costs. Humans hate recognizing small losses (http://en.wikipedia.org/wiki/Prospect_theory); this bias contributes to gamblers playing “just a few more hands” so they don’t end up in the red, to people investing in steady low-risk long-term government bonds over more lucrative long-term stocks, and to continued usage of less-than-optimal software practices.
Turns out that people actually prefer one gigantic loss over many little small losses, even if the small losses don’t come close to adding up to the one large one. We’re all guilty of not purchasing that useful app for a dollar while shelling out way too much for rent (here in NYC at least); every day people purchase the thousand dollar extras with their new car then take the long route home to avoid a $3 toll.
Traditional organizations bury their heads in the sand and refuse to acknowledge that something might go wrong with product development until problems inevitably surface at the end of the process, culminating in a gigantic loss function they must recover from. Agile/Lean organizations recognize that things will inevitably go wrong and assumptions will ultimately change, and so actually seek out small losses so they can be addressed as soon as they appear. Traditional software building only recognizes losses bundled together, Agile/Lean methodologies are specifically built to counteract our natural biases and tackle the small losses.
At the end of the second decision meeting, nobody was upset. In fact, we were elated – we had just successfully avoided a problem that would have sunk the business. The consensus was that losing some development time was a bummer, but not even close to the bummer that would exist had we released an iPad app that everybody hated.
It’s easy to lie to yourself, to tell yourself that you know what people want and that all decisions made are the right ones. It takes true courage to admit that your assumptions are just that – assumptions – and that at any moment your view of the customer and product can (and should) be challenged, reassessed, and updated by the latest evidence. It’s not weakness to admit you know nothing; Silicon Valley is littered with failed companies whose founders “knew” the right path to take and never deviated from their beliefs.
I’m proud of the team for being able to turn on a dime and move in a different direction. Lean methodologies gave us the tools to recognize our own mistakes, while Agile processes ensured that pivoting was as painless as it could be. Just goes to show that with the right mindset, any mistake – no matter how fundamental – can be rectified, as long as you’re willing to tell yourself the truth.
About the Author
