Obie Fernandez

The Next Big Leap

In my view, FITness, Cucumber and other acceptance test frameworks represented a big leap in software development effectiveness because they allow coders to get intimately involved with (and in some cases replace) Product Management/QA roles. Extreme Programming has developers working directly with product owners and capturing their requirements as "acceptance criteria" on story cards. Eventually, the desire to automate acceptance testing and best practices for how to express acceptance criteria led to GIVEN-WHEN-THEN support baked into actual testing frameworks such as Cucumber.

That's okay, but it's not nearly enough. Let's talk about how the next big leap in effectiveness is going to happen. Assume the following story is true:

• Given a specification for a feature that has not been demonstrably linked to business value,
• When a developer works to deliver an implementation of that feature,
• Then the worth of the implementation work is indeterminate until some future time when its business value is validated

Most feature requests that I've ever worked with in the early stages of a project, are ideations of the product visionary/designers/entrepreneur, based on intuition rather than customer feedback. And that's perfectly normal and acceptable to me. The more innovative a creation, the more work you usually have to do before you put it out in front of customers and get meaningful feedback. But the impact of those features on key business metrics are assumed, if it gets any explicit thought at all.

Aside: As for the implementation of those feature requests, I'm going to assume that being faithful to the specification is implicit, as it should be when we're talking about professional programmers. (Not working with professional programmers? GIGO)

Given those assumptions, I think what myself and others have realized as a problem with so-called "high-ceremony" adherence to methodologies such as BDD (Behavior-Driven Development) is that by obsessing over correctness of the implementation as faithful to the spec, it becomes easy to lose sight of the true value of our work: to deliver business value, as measured by key business metrics. We may obsess over crafting pristine, bulletproof, maintainable code, and yet its worth is indeterminate at best. And why, as developers, would we be satisfied with that!

"Measure as much as you can - no feedback == no direction"  Hugo Rodger-Brown

How can we establish that our work is not a waste of time? One technique that I hope to work on in near future is the specification validation of business value right in my acceptance criteria as part of its test cases. Said differently, I am looking for an easy way to specify non-functional requirements as part of acceptance criteria in such a way that it is automatically tested. It would be an extension or evolution of the current BDD toolset, not a replacement. You can even use de rigeur GWT syntax:

Given the implementation of email notifications,
When 100 customer events have been processed,
Then the mean time between events metrics should improve by 30%

or better

Then the net revenue per event should improve by 30%

or best

Then the net revenue per event should increase by $100

I care more about business value than faithful adherence to a particular functional requirement. My specifications should reflect that mindset. What makes this whole approach non-trivial is the challenge of establishing credible causal relationships in my acceptance criteria, given the multi-variate nature of dozens of interleaved features in a particular application and the way that key business metrics overlay many features at once.

Despite the challenges, I think the concept I'm describing is the next big leap. The first challenge is to get people talking about it and definition of goals. The second is to design a methodology and technical framework for accomplishing those goals.

What do we call the whole thing? Business-Driven Development? (no) What do we call the specs? Can you write specs against metrics in a tool such as KissMetrics? It's hard (or at least has been for me) to figure out what metrics I need until after I've written the features that yield measurement data used to calculate those metrics. How do I fix or at least improve that workflow? How do I describe the discipline needed to think of the metrics prior to specifying a feature?

Right now all I'm trying to do via this blog post and my public comments is to start a dialogue. I'm not trying to make any blanket statements about the correctness of BDD/TDD or testing in general, just pointing out that there's lots of room outside of that particular box for innovation.

As for my comments at RailsConf and Goruco, it seems to me that at a startup that has not achieved market fit, it's wrong to spend much time ensuring that a given piece of software matches its functional specification. (Specs that often won't exist, because the same person specifying the functionality is the person writing the code, in which case I'm saying don't bother writing the specs. Just "cowboy" it and do a complete rewrite later if it proves necessary.) Once you have your MVP, write specs for how particular code changes will impact key business metrics. Regressions will probably fail to improve key business metrics.

The question is not "should I test (or spec)", but rather it is "what should I test?" As an industry, to the extent that TDD/BDD has caught on at all, I don't think we're testing the right things, with our emphasis on correctness of implementation. When we figure out how to rigorously test for business value it's going to be the next big leap. That's a big green field waiting for all of us to play in and I want to be one of the first.

The Dark Side Beckons?

Lord Ries by Cody San

For years I've advocated a high-ceremony, test-first development methodology where use of BDD (or TDD) is sacrosanct and uncovered production code is a sin. Then I got to know Eric Ries and studied his Lean Startup stuff and got started with my own lean startup venture and over time...

Something's happening. I'm not the Agilist I should be. I want more. But I know I shouldn't.

I no longer believe in high-ceremony development practices for early-stage startups. Actually, I'm pretty sure that they're poisonous for most web startups, hence my lightning talk at Railsconf 2011. Until you are able to prove that you have a viable market, that customers will give you money for your product, you shouldn't be sinking a lot of time and money into implementation. Assuming you are a competent Rails developer, you should be able to hack out features and take on quite a bit of technical debt to build throwaway proof-of-concept versions of your product. I'm pretty much advocating construction of large "spike solutions" except that unlike the traditional XP definition, the problem you're trying to solve is the problem of finding your market fit. It has nothing to do with the technology.

Also, we are commonly told that spikes (aka prototype systems) are not meant for production and should be thrown away. I think that advice is very dependent on the skills of the people writing the prototype system. If they are experienced developers, they're going to inherently know a lot of the application patterns that should be used and the amount of technical debt introduced will be low enough to feel comfortable with the app in production.

Some caveats:

• I'm specifically talking about typical web-based applications built with Rails where nobody is going to die if you occasionally get a 500 error on an edge case. Use continuous deployment so that you're comfortable that you can turn around code improvements and bug fixes almost immediately.
• Use TDD to "discover" the design of parts of the system that are unfamiliar to you. This primarily applies to unit testing
• By all means have some integration-level specs that test the "happy path" through your system so that you don't inadvertently break your entire app. AKA "smoke tests"

Early on in the startup process, it's much more important to be testing against business metrics than anything having to do with code. One of the most thought-provoking pieces of advice I've heard lately came from Hiten Shah of KISSmetrics who told the latest LSM NYC crowd to consider the effects of a feature of key business metrics to be the acceptance criteria for that feature.

In other words, before you implement a feature, you should have an assumption that tells you what the effect is going to be: higher click-through, engagement, satisfaction ratings, etc. This results in an acceptance test that is time-shifted; you won't know the results until you deploy. I believe this is fertile ground for technical innovation. While there are plenty of frameworks that simplify AB testing and real-time monitoring, I don't believe there exists anything along the lines of Cucumber that involves developers in cultivating a mindset of provable business value. David Chelimsky and I joked around that perhaps such a tool should be called Squash.

One of the natural questions that arises from this line of thinking is: When am I no longer an early-stage startup? The implication being that at some point, you know enough about your market fit and product requirements that you can transition into what I've been calling high-ceremony development practices.

I don't have the answer to that question. I suspect that the size of the development team has a lot to do with how you'll want to answer that question for your own effort, because multiple workstreams on a single codebase has a tendency to skyrocket technical debt. All of the above is premised on your ability to keep technical debt in control -- I have no doubt that irresponsible dependence on technical debt will kill your company just as effectively as relying too much on financial debt -- I've seen it happen.

The bottom line is that once again I find myself rejecting dogmatic adherence to a particular mindset. And contrary to my initial metaphor, that doesn't seem evil at all. Maybe I got it backwards and the high-ceremony Agilistas are the dark side, and the lean startup hackers are the Rebel Alliance... what do you think?

Hey Web Startups, Hire Consulting Firms On The Cheap!

So you're an entrepreneur and you're doing a lean web startup using Ruby on Rails. You're teaching yourself Rails and proud of yourself for going at it alone and not spending a bundle of cash. You know that you can't afford to hire a world-class shop like Hashrocket, Thoughtbot or Pivotal Labs to work on your project, because the minimum project budgets they are interested in are in the six figures.

Incidentally, this post is strictly my personal opinion is in no way endorsed by or representative of anyone at Hashrocket, ThoughtBot or Pivotal.

Well, allow me to let you in on a little secret. You can absolutely work with those firms and maximize the benefit provided to you for as low as a couple thousand dollars. Just not in the way that you might think. And it won't be the way that they would prefer to work with you, but they'll still do it.

See the jars above? Consider the container to be the size of the available billable hours in a month. The large rocks and pebbles are current projects. It is impossible to fully utilize the space available unless you start adding smaller rocks and pebbles and eventually, grains of sand. Even the busiest consulting shops always have small amounts of utilization available. It's the slack in the system, time for people to relax, the empty volume of the jars. Despite the importance of slack, the profits of a consultancy live and die by utilization. When I was in charge of Hashrocket, which was generally busy, do you think I stressed about utilization? Yes I did. Every fucking week of the year. It's one of the reasons I got out of the consulting business. Will most consultancies take advantage of opportunities to bump up their utilization with profitable small work? Yes! The trick is how to make them do it, even if they resist at first.

Before I describe that trick, let me explain the backstory that provoked this post: Last night I got to know a couple of young entrepreneurs from NYC, Chris and Philip. They are working on an interesting little web startup named Meeteor. It's technology driven. During a conversation about testing, Philip told me that he spends a couple hours manually testing the app everytime they're ready to do a deploy. They don't do any automated testing, because they don't have the time to figure out how to do it well. My ex-ThoughtWorks friend Kurt Schrader was present and we did the requisite head shaking and tsk-tsk-ing about their failure to test their code. But the truth is, if I were in these guy's shoes I would probably do the same thing. It takes years to get really good at TDD and integration testing to the point where it doesn't slow down your progress tremendously. Sure, without tests you'll build up technical debt to the ceiling and the whole project may eventually collapse under its craptastic weight, but in the meantime the perceived slowness of figuring out how to test your stuff will demoralize and destroy your productivity. It's a no-win situation for newbies.

My advice to the Meeteor guys (and to all of you in similar situations) is the secret alluded to above: Hire help from Hashrocket (or any other competent firm) but only for a day or two at most. And I'm going to tell you exactly how to do that:

1. Call the shop up and get a decision maker on the phone. Convey that you know exactly why you're calling them and what you need.
2. Request their top Rails (and/or testing) expert for 1 day. State right off the bat that you're willing to pay their full hourly rate for those 8 hours. If you can travel to where that person is then state that you're willing to do so. Otherwise offer to pay for reasonable travel expenses to get the expert on site with you for a day.*
3. Be insistent and don't lowball them on the price. Even if they want a substantial markup on their normal rates it will still be worth it. Be prepared to have to wait a few weeks for that person to be able to take a day off from their current project. Be prepared to wire half the money up front as a deposit or even pay in full upfront.
4. Make sure the contracting goes as smoothly as possible. Do not involve your lawyer in the paperwork or negotiate on price. Do not keep anyone on the phone for a long time answering questions. Anything that makes the sale time-consuming to close (read: expensive to handle) will make the whole gig unprofitable and they will rightly tell you to get lost.
5. Once you are working with your expert, sit as a pair at the computer with them but make them drive the entire 8 hours. Do not touch the keyboard at all. Your aim will be to get the expert to tackle as many sticky problems as possible in 8 hours. Dedicate the first hour to making them set your development environment up properly, because it the way you have it right now sucks. Don't let them try to teach you VIM if you use TextMate, like don't even spend 5 seconds on that, but let them use whatever they normally use. Do not do any storycarding or get any consulting about your consulting process whatsoever. Remember that you're not there to learn directly from your expert. What you are there for is to direct the work, answer questions about the domain and keep your expert focused and productive for the duration of the 8 hours. That includes getting coffee, snacks, cigarettes, whatever. You are there to get them to do as much work on your app as possible and you will learn later on from analyzing their work.

* Right now (Mar '11) a total budget of $3k is probably very reasonable for what I'm describing.

In the case of my friends at Meeteor, they'd want their expert to implement some cross-browser selenium integration test cases. All they need is one of their most difficult ones, because once they have the scheme setup and one or two examples to draw from, they're smart enough to figure out the rest themselves.

Remember that the point of this technique is to buy as much great coding as possible in a day or two. Then you can use the results for inspiration for your own hacking. This technique works with any superstar programmer whether they are coming from a shop like Hashrocket or not. Shop refuses to work with you? Hire one of their guys for a single-day side gig! In Europe and using Mongo? Throw a couple thousand at my friend Durran to pair with you for a day. You get the idea. Just make sure that the person you hire works full-time as a programmer every day. This scheme wouldn't work with someone like me; I have plenty of depth, but slow velocity since it's not what I'm doing everyday.

The productivity boost you can get as a newbie developer from working a true expert for just a day (or two) is exponentially more valuable than what it will cost you in cash.

Template To Document Your Customer Discovery Hypotheses

Chapter 3 of Steve Blank's Four Steps to the Epiphany instructs you to State Your Hypotheses as Phase 1 of the Customer Discovery Process. In the process of doing that, I put together a template with the relevant outline of questions that need to be answered. Sharing the document with the hopes that it helps other lean entrepreneurs.

Download hypotheses_template.doc

Steve Blank on when not to add features

I'm studying Steve Blank's Four Steps to the Epiphany with an intensity that I haven't mustered for any book in years. I'm talking liberal use of yellow highlighter markers and scribbled notes in the margins. It's gold, I tell ya! I love Steve's advice about uncovering a market that matches your original product vision:

This rigor of no new features until you've exhausted the search of a market space - counters a natural tendency of people who talk to customers – you tend to collect a list of features that if added, will get one additional customer to buy. Soon you have a ten page feature list just to sell ten customers. The goal is to have a single paragraph feature list that can sell to thousands of customers.

The quote is from a passage talking about the second synchronization meeting between the customer development team and the product development team. Roughly speaking, in software anyway, that divide is between Sales/Marketing/Product Management and the Designers/Developers. In a 2-person startup, one founder will probably focus mostly on customer development and the other on product.

You arrive at a true Minimum Viable Product (MVP) after the product development team has already started working on the original vision and customer development team has "left the building" to validate its assumptions by talking to real prospective clients. The feedback from clients is used to trim features from the current product feature list, but NOT to add features. You don't add features until you've become convinced that you're building the completely wrong product - at that point you're probably going to scrap the idea and start over.

This valuable book came out in 2007 and I can't believe that it took me this long to study it! If you are in any way involved in startup activities and haven't bought this book yet you should absolutely get it as soon as possible: Buy the book from Amazon (or if you're not convinced yet then download the first three chapters as a PDF for free from Stanford first.)

Further reading online: Steve blogs regularly and his related thoughts on MVP are here. There's also an epic blog post about MVP that takes into account Eric Ries' thoughts on the matter here.