Obie Fernandez

The Next Big Leap

In my view, FITness, Cucumber and other acceptance test frameworks represented a big leap in software development effectiveness because they allow coders to get intimately involved with (and in some cases replace) Product Management/QA roles. Extreme Programming has developers working directly with product owners and capturing their requirements as "acceptance criteria" on story cards. Eventually, the desire to automate acceptance testing and best practices for how to express acceptance criteria led to GIVEN-WHEN-THEN support baked into actual testing frameworks such as Cucumber.

That's okay, but it's not nearly enough. Let's talk about how the next big leap in effectiveness is going to happen. Assume the following story is true:

• Given a specification for a feature that has not been demonstrably linked to business value,
• When a developer works to deliver an implementation of that feature,
• Then the worth of the implementation work is indeterminate until some future time when its business value is validated

Most feature requests that I've ever worked with in the early stages of a project, are ideations of the product visionary/designers/entrepreneur, based on intuition rather than customer feedback. And that's perfectly normal and acceptable to me. The more innovative a creation, the more work you usually have to do before you put it out in front of customers and get meaningful feedback. But the impact of those features on key business metrics are assumed, if it gets any explicit thought at all.

Aside: As for the implementation of those feature requests, I'm going to assume that being faithful to the specification is implicit, as it should be when we're talking about professional programmers. (Not working with professional programmers? GIGO)

Given those assumptions, I think what myself and others have realized as a problem with so-called "high-ceremony" adherence to methodologies such as BDD (Behavior-Driven Development) is that by obsessing over correctness of the implementation as faithful to the spec, it becomes easy to lose sight of the true value of our work: to deliver business value, as measured by key business metrics. We may obsess over crafting pristine, bulletproof, maintainable code, and yet its worth is indeterminate at best. And why, as developers, would we be satisfied with that!

"Measure as much as you can - no feedback == no direction"  Hugo Rodger-Brown

How can we establish that our work is not a waste of time? One technique that I hope to work on in near future is the specification validation of business value right in my acceptance criteria as part of its test cases. Said differently, I am looking for an easy way to specify non-functional requirements as part of acceptance criteria in such a way that it is automatically tested. It would be an extension or evolution of the current BDD toolset, not a replacement. You can even use de rigeur GWT syntax:

Given the implementation of email notifications,
When 100 customer events have been processed,
Then the mean time between events metrics should improve by 30%

or better

Then the net revenue per event should improve by 30%

or best

Then the net revenue per event should increase by $100

I care more about business value than faithful adherence to a particular functional requirement. My specifications should reflect that mindset. What makes this whole approach non-trivial is the challenge of establishing credible causal relationships in my acceptance criteria, given the multi-variate nature of dozens of interleaved features in a particular application and the way that key business metrics overlay many features at once.

Despite the challenges, I think the concept I'm describing is the next big leap. The first challenge is to get people talking about it and definition of goals. The second is to design a methodology and technical framework for accomplishing those goals.

What do we call the whole thing? Business-Driven Development? (no) What do we call the specs? Can you write specs against metrics in a tool such as KissMetrics? It's hard (or at least has been for me) to figure out what metrics I need until after I've written the features that yield measurement data used to calculate those metrics. How do I fix or at least improve that workflow? How do I describe the discipline needed to think of the metrics prior to specifying a feature?

Right now all I'm trying to do via this blog post and my public comments is to start a dialogue. I'm not trying to make any blanket statements about the correctness of BDD/TDD or testing in general, just pointing out that there's lots of room outside of that particular box for innovation.

As for my comments at RailsConf and Goruco, it seems to me that at a startup that has not achieved market fit, it's wrong to spend much time ensuring that a given piece of software matches its functional specification. (Specs that often won't exist, because the same person specifying the functionality is the person writing the code, in which case I'm saying don't bother writing the specs. Just "cowboy" it and do a complete rewrite later if it proves necessary.) Once you have your MVP, write specs for how particular code changes will impact key business metrics. Regressions will probably fail to improve key business metrics.

The question is not "should I test (or spec)", but rather it is "what should I test?" As an industry, to the extent that TDD/BDD has caught on at all, I don't think we're testing the right things, with our emphasis on correctness of implementation. When we figure out how to rigorously test for business value it's going to be the next big leap. That's a big green field waiting for all of us to play in and I want to be one of the first.

The Dark Side Beckons?

Lord Ries by Cody San

For years I've advocated a high-ceremony, test-first development methodology where use of BDD (or TDD) is sacrosanct and uncovered production code is a sin. Then I got to know Eric Ries and studied his Lean Startup stuff and got started with my own lean startup venture and over time...

Something's happening. I'm not the Agilist I should be. I want more. But I know I shouldn't.

I no longer believe in high-ceremony development practices for early-stage startups. Actually, I'm pretty sure that they're poisonous for most web startups, hence my lightning talk at Railsconf 2011. Until you are able to prove that you have a viable market, that customers will give you money for your product, you shouldn't be sinking a lot of time and money into implementation. Assuming you are a competent Rails developer, you should be able to hack out features and take on quite a bit of technical debt to build throwaway proof-of-concept versions of your product. I'm pretty much advocating construction of large "spike solutions" except that unlike the traditional XP definition, the problem you're trying to solve is the problem of finding your market fit. It has nothing to do with the technology.

Also, we are commonly told that spikes (aka prototype systems) are not meant for production and should be thrown away. I think that advice is very dependent on the skills of the people writing the prototype system. If they are experienced developers, they're going to inherently know a lot of the application patterns that should be used and the amount of technical debt introduced will be low enough to feel comfortable with the app in production.

Some caveats:

• I'm specifically talking about typical web-based applications built with Rails where nobody is going to die if you occasionally get a 500 error on an edge case. Use continuous deployment so that you're comfortable that you can turn around code improvements and bug fixes almost immediately.
• Use TDD to "discover" the design of parts of the system that are unfamiliar to you. This primarily applies to unit testing
• By all means have some integration-level specs that test the "happy path" through your system so that you don't inadvertently break your entire app. AKA "smoke tests"

Early on in the startup process, it's much more important to be testing against business metrics than anything having to do with code. One of the most thought-provoking pieces of advice I've heard lately came from Hiten Shah of KISSmetrics who told the latest LSM NYC crowd to consider the effects of a feature of key business metrics to be the acceptance criteria for that feature.

In other words, before you implement a feature, you should have an assumption that tells you what the effect is going to be: higher click-through, engagement, satisfaction ratings, etc. This results in an acceptance test that is time-shifted; you won't know the results until you deploy. I believe this is fertile ground for technical innovation. While there are plenty of frameworks that simplify AB testing and real-time monitoring, I don't believe there exists anything along the lines of Cucumber that involves developers in cultivating a mindset of provable business value. David Chelimsky and I joked around that perhaps such a tool should be called Squash.

One of the natural questions that arises from this line of thinking is: When am I no longer an early-stage startup? The implication being that at some point, you know enough about your market fit and product requirements that you can transition into what I've been calling high-ceremony development practices.

I don't have the answer to that question. I suspect that the size of the development team has a lot to do with how you'll want to answer that question for your own effort, because multiple workstreams on a single codebase has a tendency to skyrocket technical debt. All of the above is premised on your ability to keep technical debt in control -- I have no doubt that irresponsible dependence on technical debt will kill your company just as effectively as relying too much on financial debt -- I've seen it happen.

The bottom line is that once again I find myself rejecting dogmatic adherence to a particular mindset. And contrary to my initial metaphor, that doesn't seem evil at all. Maybe I got it backwards and the high-ceremony Agilistas are the dark side, and the lean startup hackers are the Rebel Alliance... what do you think?

My RubyConf India 2010 Talk Highlights

Hashrocket sponsored the inaugural RubyConf India event primarily to support the nascent, yet growing community of Rubyists here, but also due to my personal love of Indian people and culture. I've been eager to get back to Bangalore for the last couple of years and this was the perfect excuse. I deliberated about what talk to deliver to this audience. At all costs I wanted it to be entertaining and provocative. In my opinion, too many Indian service companies are "doing it wrong" -- they stick to the stereotypical modus operandi of offshore providers, with waterfall processes and fixed bid. I wanted to impress on the audience that there is a better, more fun way to run your business. After opening comments by the organizers and Roy Singham, Chairman of ThoughtWorks, I delivered the second general audience talk.

I was charged up with the right kind of anxious energy and this being material that I know inside and out, it came straight from the heart. The talk was not entirely new, previous versions of it had similar material. However, I did go through and restructure the outline significantly as well as adding new material based on the last 12 months of operations at Hashrocket. Luckily I also added some elements that I knew would appeal to my audience -- slides based on the blockbuster Indian movie "3 Idiots". Little that I know how important the concept of "All Izz Well" would prove to be to myself!

On a scale of 1 to 10, I would rate myself a 9 on delivery. I was funny and engaging, making eye contact with the audience members and spending much time in storytelling mode. Unfortunately, I would have to deduct points for making the mistake of dropping the "F-bomb" too many times and as a result getting interrupted and vocally dressed-down by Tina Vinod halfway through! Somehow, although nervousness infected my voice and I started forgetting to breathe, I did pat my heart and tell myself audibly "all izz well, all izz well" and charged ahead with my talk -- the audience got a good laugh at that. Other challenges included the organizer mistakenly telling me I was out of time when I actually had at least 15 minutes left! Luckily my Keynote display presenter mode with its timer saved me. I did pick up the pace a bit, but finished in 48 minutes, just 3 minutes over my alloted time.

Reading List

In the section of the talk about Process and Practices, I mentioned that it is necessary to go "by the book" when starting out. Too many people criticize practices that they haven't properly implemented. You must learn to crawl (so-to-speak) before you can evolve your own practices. 

The list of books I recommended constitute a large part of the orthodoxy of progressive software development nowadays. I asked the audience, how many of you have read at least half of these books. Sadly, only a dozen or so hands (out of 400) shot up. I truly believe that you cannot be effective nowadays without studying these books and applying them to your day to day.

Agile Software Development, Principles, Patterns, and Practices

Agile Retrospectives: Making Good Teams Great

Apprenticeship Patterns: Guidance for the Aspiring Software Craftsman

Design Patterns: Elements of Reusable Object-Oriented Software

Domain-Driven Design: Tackling Complexity in the Heart of Software

Extreme Programming Explained: Embrace Change (2nd Edition)

Refactoring: Improving the Design of Existing Code

Secrets of Consulting: A Guide to Giving and Getting Advice Successfully

Software Craftsmanship: The New Imperative

Test Driven Development: By Example

The Principles of Product Development Flow: Second Generation Lean Product Development

User Stories Applied: For Agile Software Development

Working Effectively with Legacy Code

 After you've mastered the fundamentals, then you can begin to evolve your process and practices to actually achieve the best results for your context. I must have said 20 times during my talk that "context is king."

 

One of my favorite and most controversial pieces of advice, at least based on the feedback and comments that it generated:

"I don't defend breathing... I don't defend Agile. It just is."

I believe unequivocally that the time to be evangelizing and advocating proven Agile techniques is over. It's been at least 10 years since Extreme Programming hit the scene. We know that TDD yields excellent results. We know that pair programming yields excellent results. We won. To continue being in defensive mode against stupidly obstinate management is a waste of time. It may take a long time from now, but eventually people that don't work in smart Agile ways will fail and go away. As for you, concentrate on being excellent, perfecting your craft and building a loyal and devoted group of clients that will provide you with strong references and testimonials.

If you want to put into practice my advice, refuse to work for clients for whom success is not 100% important. This will necessarily exclude most large enterprise clients, where politics and status quo are as important as results. Those types of clients will drag you down and make you compromise your values. Startups and entrepreneurs usually are 100% focused on successful results. These stakeholders will not be worried about your technology choices or whether you want to pair program and do test-first. They will let you do your job, and the way that you sell them on using you is with successful track record, demonstrated by copious amounts of testimonials from previous clients. I stressed the importance of photos and videos to capture the spirit of your success in a vivid and emotional way. We have the technology -- use it!

 

My advice to fire clients if they need to be fired was perhaps shocking. It shouldn't be. I've only had to fire a client a few times in the history of Hashrocket. That's considering a list of now almost 60 happy clients of all shapes and sizes. The key to not having to fire clients, is to not get the wrong clients to begin with. This is known as being able to "choose your clients." It's not as hard at it seems. Sure, when you're starting out it is somewhat difficult to put this advice into effect and you will make mistakes. That's when it's important to have the confidence to cut ties and look for greener pastures.

"Never allow your clients to corrupt your values or integrity."

The worst thing that can possibly happen, more than having money difficulties, is to corrupt yourself by compromising on your values and integrity due to pressure from a bad client. I'm not saying to be inflexible! I'm saying that there are core values that will allow you to reach for excellence and achieve it. You won't know what those values are when you're starting out, but you can be informed by studying the successful results of others and internalizing that knowledge. Then when you're on a roll and doing great work for your clients, you will realize what those core values are. From that moment on, no matter how big the pressure is, you should not compromise. The more confident you are about your way of working, the easier it will be for you to sell your services and acquire quality clients. The last thing you ever want to do is to appear desperate -- at that point great clients will go away running and bad clients, those that will take the worst advantage of you will become very friendly! Never allow your clients to compromise you. Stick to your values and focus on success.

When I'm evaluating software developers to hire, I don't give much credence to resumes, cover letters, referrals or work history. The only things I care about are work ethic and cultural fit. And there's only one way to determine whether a candidate satisfies my criteria for success -- actually see them in action over the course of a week or more. Since we always pair program, it is trivial to do short contract gigs with qualified candidates and bring them in putting them on multiple real projects. Seeing them working on real code in real situations is very instructional. Again, context is king. Over the course of time, we've created a very desirable work environment at Hashrocket, which means that there is a strong demand for people to come work with us. That means I can diverge easily from traditional hiring models.

Conclusion

Indian business culture, at least the stereotypes that are propagated to the west, doesn't strike me as particularly fun. But if you're doing great work for your clients and sticking to your core values, then you're going to have a lot of success and be able to enjoy it by building a very fun work environment. My final series of slides was a photo set of various fun activities that Hashrocket folks engage in on a regular basis, from participating in user groups, to music and sporting events, games, having parties, drinking and dancing and doing crazy stunts like all jumping into a swimming pool at the same time.

I truly believe that if you're not having fun at work, you're doing it wrong. A happy, healthy environment is one that leads to the best results. However, you can't impose fun on a group by fiat, just like you can't create culture. It has to emerge from hiring happy, progressive folks that want to do the best work possible while sticking to strong core values. This is the final message of my talk and I hope that the audience found it inspiring and worth passing on...

 

(MSA Series #4) Warranties

BUY THE ANNOTATED MSA+SOW DOCUMENT TEMPLATE BUNDLE!

The documents discussed in this blog posts are now available from me for only $99. Click here to learn more.

 

This is the fourth entry in a long-running series of blog posts about the Hashrocket Master Services Agreement (MSA). Parts one, two and three are also available. When the series is finished, you might be able to piece together a valid MSA document from the examples presented here. However, keep in mind that I am not a lawyer, and that the context of your business may be different than mine. Also, remember that laws vary wildly from country to country and what is presented here might be useless outside of the USA.

Section 5, Warranties

This section of the MSA establishes what warranty, if any, you are guaranteeing on the work performed under the scope of your contract with your client.

The first clause is a bit of legalese stating that each party is authorized to enter the agreement:

5.1 Warranty of Authority; No Conflict
Each party warrants that it is authorized to enter into this Agreement and to perform its obligations hereunder, and that its performance hereunder shall not conflict with, limit or be contrary to any other agreement. 

The second clause of section 5 establishes a warranty on the nature of the work provided, that it will be performed in a professional manner and by qualified personnel.

5.2.1 Professional Manner Hashrocket warrants that all services will be performed in a professional manner using qualified professional personnel. 

Of course, the words professional and qualified are subjective, which means they could be the basis of dispute later on, if problems were to arise. I've been involved in arbitration hearings as an expert witness, testifying on whether work performed lived up to generally-accepted best practices and industry standards. Some of the big factors that affect whether I consider a Rails project to be professional have to do with the quality of testing on the project, and adherence to the Rails conventions and best-practices.
This paragraph edited on 9/23, because the original wording made it sound like I was casting judgment based on my book The Rails Way.

The following paragraphs are of terrific importance in terms of protecting the vendor from unreasonable liability, which is why they are printed in prominent all-caps:

THE WARRANTIES SET FORTH IN THIS AGREEMENT ARE EXCLUSIVE AND ARE IN LIEU OF ALL OTHER WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. EXCEPT WHEN OTHERWISE STATED IN WRITING THE MATERIALS PRODUCED UNDER THE TERMS OF THIS AGREEMENT ARE PROVIDED TO CLIENT "AS IS," THAT IS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE AND/OR SERVICES PROVIDED UNDER THIS AGREEMENT RESTS SOLELY WITH THE CLIENT.  SHOULD THE SOFTWARE OR PROGRAM PROVE DEFECTIVE, CLIENT SOLELY ASSUMES THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION, INCLUDING WITHOUT LIMITATION ANY "DEBUGGING."

Significant in this paragraph is the stipulation that work is provided as is, that is, the client cannot force you to continue working on the project if you don't want to do so, or for free. The fact that you are not providing a warranty decidedly does not mean that you are planning to give your client junk. In fact, unless you have a great reputation and client testimonials to back it up, you're probably going to have trouble getting clients to agree to this clause.

There's more to this clause, specifically the stipulation that the client is solely responsible for the "risk as to the quality and performance of the software." The only effective (and moral) way that I know to put that risk on your client is to establish small feedback loops via a well thought-out acceptance process. At Hashrocket, we use the features of Pivotal Tracker to facilitate constant, ongoing acceptance of story functionality and tracking of defects. Delivery of new features and bugfixes happens on a daily basis and so does client acceptance.

The following graph, generated by Pivotal Tracker as part of its Project Points Breakdown report, visually shows progress of a project along the stages of our workflow. The overall height of the bars gives an indication of scope. As the black bars (unstarted work) shrinks, you can see the project headed towards completion.

The vertical axis denotes the number of story points in the project. Note how narrow the bands of green (Delivered) are during the progress of this project. The graph tracks the state of stories on a daily basis. Green bars represent functionality that has been delivered to the staging server for the client to test, but not accepted yet. Occasionally the circumstances cause the client to fall behind for a day or two, but gentle pressure and assistance brings them back in line.

I won't try to explain the legalese of the second all-caps paragraph, simply because I don't necessarily understand it fully myself. This text is in there under the strong advise of our lawyer.

EXCEPT AS OTHERWISE STATED ABOVE, NEITHER PARTY MAKES ANY WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, WARRANTIES RELATED TO INFORMATION OR BUSINESS ADVICE PROVIDED, WARRANTIES RELATED TO OUTCOMES BASED ON INFORMATION OR ADVICE PROVIDED, WARRANTIES OF MERCHANTABILITY OR MERCANTILE QUALITY, WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE OR USE, WARRANTIES OR CONDITIONS ARISING BY STATUTE OR OTHERWISE IN LAW, OR WARRANTIES OF ANY PRODUCTS OR SERVICES PROVIDED BY THIRD PARTY VENDORS.

What I will say is that the wording of this paragraph, and indeed all of the all-caps paragraphs are of supreme importance. There is established case-law about these kinds of warranties, and mucking with the wording is a good way to have the entire clause invalidated.

Finally, the supremely-important liability disclaimer, which helps ensure an angry client doesn't put you out of business.

THE PARTIES AGREE THAT NEITHER PARTY’S LIABILITY FOR DAMAGES FROM ANY CAUSE OF ACTION WHATSOEVER, REGARDLESS OF THE FORM OF ACTION, WILL EXCEED THE FEES PAID OR TO BE PAID BY CLIENT PURSUANT TO AN APPLICABLE STATEMENT OF WORK UNDER THIS AGREEMENT. IN NO EVENT SHALL EITHER PARTY BE LIABLE FOR LOST PROFITS OR ANY INDIRECT, INCIDENTAL, CONSEQUENTIAL OR SPECIAL DAMAGES OF ANY NATURE WHATSOEVER, INCLUDING, WITHOUT LIMITATION, DAMAGES ARISING FROM LOSS OF USE OF ANY SOFTWARE OR HARDWARE, COSTS OF PROCUREMENT OF SUBSTITUTE PRODUCTS OR SERVICES, LOST DATA, LOST PROFITS OR REVENUE, OR FOR ANY CLAIM OR DEMAND BY ANY THIRD PERSON, ARISING OUT OF OR RELATED TO THE AGREEMENT OR THE PERFORMANCE OR BREACH THEREOF, EVEN IF ADVISED OF THIS POSSIBILITY.

If things go completely wrong and lawsuits are filed, you agree beforehand that the maximum liability incurred will not exceed the amount of money that has changed hands. In other words, if you completely fuck up the project, no matter how hurt the client is the most he can do is to get his money back. Absent this stipulation, you could face lawsuit claims in the millions of dollars to account for all sorts of opportunity costs and damages asserted by the client. Luckily this language is fairly common in service contracts.

5.2.2 No Infringement The parties represent and warrant that their disclosure and delivery of any information, documents, software and other materials, and use thereof, as contemplated by this Agreement, will not knowingly infringe or violate any proprietary right of any third party, including, without limitation, any copyright, known patent or trade secret right.

This clause is entirely for the protection of your client. It binds you to a promise not to deliver intellectual property to them that belongs to someone else. In practice this prevents you from taking code that you've written for Client A and selling it again to Client B, without the express knowledge and permission of Client A. Since it's common for software consultancies to produce non-domain-specific code with potential for profitable reuse across clients, a different clause in the MSA establishes a method of differentiating intellectual property produced under billable time, and how the rights to such vary.

5.3 Noninterference with Business
During the term of this Agreement and for all periods thereafter, Hashrocket agrees not to directly or indirectly compete or interfere with Client’s Business in any manner.  Additionally, and without limiting the foregoing, during the term of this Agreement and for a period of two years thereafter, Hashrocket agrees not to, directly or indirectly solicit or induce or attempt to persuade any employee, independent contractor, vendor, supplier, outsourced third-party, director or other participant of Client to terminate an employment, contractual or other relationship with Client, or to enter into a relationship with Hashrocket or into any business organization in which Hashrocket may be directly or indirectly involved.  The term “enter into a relationship” shall include, but not be limited to, acting as a paid or unpaid director, officer, agent, employee of, or consultant to, or acting or participating as owner, partner, manager, member, or shareholder.  During and for a period of two years immediately following termination of this Agreement, Hashrocket further agrees not to (a) directly or indirectly contact any person or entity disclosed by Client for the purpose of taking advantage of a business opportunity, (b) otherwise circumvent a relationship with the Client  or establish a relationship with a party with whom Client already has a relationship or foreseeable relationship with whom Hashrocket has never had a relationship, or (c) seek to establish any rights, including but not limited to intellectual property rights, anywhere in the world in conflict with Client’s pre-existing, herein established, or hereafter established intellectual or other property or proprietary rights.

Interestingly, as I was reading this clause for inclusion in the article, it struck me that its limitations are not mutual. (I'll be fixing that soon.)

All this final paragraph of the section does is to limit you as the vendor from screwing your client in various ways, such as poaching their business model or employees, behavior that would harm your reputation and future business anyway. It's smart to make your clients feel comfortable that you won't engage in nasty behavior as described above. You should keep aware of this clause, particularly as it applies to hiring people that work (or used to work) with your clients.

The next article in the series will cover Fees, Invoices and Payments.

Explain Variable Scope with Ponies

Lately I've been using the following analogy as a valuable tool in my sales process. Used properly, it's totally killer and you're gonna love me for it. I promise. It involves ponies, kind of...

First of all, let me set the context by explaining the problem I'm helping you solve. (Mind you, I know that not everyone does Agile the way that I do at Hashrocket, so it definitely won't apply to everyone.) But if you do try to sell Agile, even if it's only to your managers, bear with me and tell me if this situation sounds familiar:

1. You talk to a new client
2. Client has a list of features desired, perhaps even a requirements document or functional spec
3. Client wants to know how much it will cost to get their project done (or just a ballpark estimate)
4. After explaining that it's up to them how much it will cost (since they will be required to prioritize for business value and manage scope) Client still wants to know how much you think it will cost to "get done"
5. After explaining for the tenth time that it's really up to them how much they want to spend and optionally giving them a wide (and meaningless) estimate like $50 to $150 thousand, they still want to know how much you think it will cost.

People that haven't done Agile software generally don't understand the concept of variable scope. They think they know exactly what they want, and they want a price. If you don't give them a price, they'll keep looking until they find a provider that gives them a price, for better or worse... usually worse.

Here's the analogy, which serves to illustrate variable scope and price very effectively. Compare the requirements to a child's Christmas list or letter to Santa. (or three kings or whatever) For reasons that will be obvious in a moment, this analogy going to be most effective with actual parents that have to go Christmas shopping for their kids every year.

Rattle off a realistic gift list like the following one:

• Barbie doll
• Barbie book that comes with makeup
• Playhouse with kitchen
• Barbie video game
• Barbie coloring book
• Pony

It's important that you communicate that all or most of these items involve ambiguity about what is actually desired.

Ask the client (or yourself): How much do you have to spend to get this list "done"?

How much does a Barbie doll cost nowadays? Twenty? Thirty? Less? Who knows? You can ballpark it, sure, but what about some of the other things on the list?

It's actually impossible to say how much it will cost to fulfill this list until you actually do the Christmas shopping! Why? Because of the ambiguity!

If you're a parent on a budget, then you might have an idea of how much you're going to spend fulfilling the desires on this list, but only because you have a budget in mind. And even in that case, you still don't really know exactly how much you're going to spend, because you're going to make buying decisions on the fly as you visit the stores where you're going to buy the gifts. Stuff will be on sale or out of stock, you might feel better or worse depending on other people you have to buy gifts for. No matter what, you must try to maximize the happiness of your child without breaking the bank and making everyone else happy.

Even rich parents that don't go shopping with a particular budget in mind will understand this analogy! They still have to maximize the child's happiness without knowing exactly what you're going to end up spending. For instance, it's the rare child that will actually get a pony for christmas, but maybe while shopping the parent sees a stuffed pony doll, or a My Little Pony playset and spends $20 to buy it and cross "Pony" off the list. Maybe they don't. I have no idea how to fulfill that Pony request, and neither does your client, until they actually start shopping.

That's where I'm going with this. Actually implementing the prospective client's project is like doing the Christmas shopping described above. You might know approximately what things on your client's requirements list will cost beforehand, but there's never a way to know exactly what you're going to end up spending until you actually start crossing items off of that list (clearing the backlog) and decide that you're done (release the software).

Limitations of a Story-driven process

Over the course of the last six months, time permitting, Hashrocket has been working on a product initiative codenamed "Launch Pad". Hashrocket does a lot of Web 2.0 apps, and most of them have a similar assortment of basic functionality and social networking features: user accounts, event notification, photo uploading, commenting, etc. Naturally, there is a desire to take our sum knowledge of building these types of features and codify it in the form of a base application and suite of plugins.

I'm describing the reuse proposition, one of our profession's holy grails, which ideally would bear fruit in terms of cost savings and shortened time-to-market for our clients. Not to mention that a productized Hashrocket Launch Pad is a product that I could sell at a profit, proven by folks such as Pivotal, with their social media platform, licensing for which starts at around 30k.

Initially, Launch Pad was an informal effort and we didn't have a good way to measure progress. Months ago, we went through a storycarding effort to identify the features desired and establish accountability for progress. The stories were entered into our story-tracking tool, Pivotal Tracker, and estimated.

Here's a illustrative snapshot as an example of the type of  stories we ended up with:

If you're familiar with Tracker, you know that the green background on those stories means they were finished and delivered. Acting as stakeholder for the project, I went into our testbed deployment and verified that the features requested were present and met my expectations. So far, so good. After getting through most of the stories, we now have a version of our Launch Pad that exists in the form of a "baseapp". Not quite salable, but useful in the following way: we can clone it as a starting point for greenfield projects, to get that big initial boost in velocity.

Therein lies a problem. It's one thing to have a rich set of functionality in a baseapp when you haven't started a project yet. WIN! It's a whole 'nother thing to have a baseapp with desired functionality when you need to add that functionality to an existing project. FAIL!

The answer of course, is to factor the functionality into more-reusable forms, namely Rails plugins (or gems) that can be added piecemeal to either a barebones baseapp or an existing project. How do you track that work?

We recently had a storycarding session to try and figure out how to do that. The issue is that now we are not focusing on end-user functionality. The user role (in terms of acceptance) is the application developer that needs to use the plugins and gems that we ultimately end up with. Unfortunately, it took quite a bit of time to reach that somewhat obvious conclusion!

Round and round we went, trying to determine how to name the story, describe it and structure acceptance criteria. After much argument and fretting about implementation, we eventually we ended up with something like this:

Story: Adding Photos

In order to provide photo collections for my client,
As a developer,
I want to be able to add photos to a domain model

Acceptance:

• Installation instructions
• Declarative macro makes model act like a photo collection
• Generator-created: Photos controller that includes photo resource module
• Generator-created: Views (new, form partial, javascript, FancyUploads)
• Generator-created: migration
• API documentation

This is a very high-level story! One of my guys called it a "12-pointer" to denote how much it needed to be broken down in order to fit in with our typical process (limited to 1, 2 and 4 point stories).

The problem is that as the stakeholder, I don't want this story broken down! This is the level at which I want to do acceptance, for at least a couple of reasons: 1) I already went through a long acceptance process for many of these stories the first time they were implemented, in the language of the end-user. 2) In order to properly define acceptance criteria for this story, I would have to know exactly how it's going to be implemented, and that information is not available until they actually sit down to figure out how to do the extraction into gems and plugins. In fact, one of the reasons that the storycarding session was feeling so painful was exactly that we were spending so much time arguing about implementation -- out of place in a typical storycarding process where you leave implementation details for later.

That said, to do acceptance for an application developer story, I would need to go in and do things such as follow the API instructions to add the photo functionality to a testbed application. It was starting to feel like I was using our typical story-driven process in anger. It shouldn't be this hard to do something as conceptually simple as an extraction to plugins.

Somewhat frustrated, I ended the storycarding meeting and changed direction. Come in tomorrow and take one of the features from our existing testbed application. It has really good spec coverage, so spend the day extracting it to a plugin and make sure you don't break the testbed app's specs. The following morning we'll have a meeting to see how you did.

Where have you experienced limitations of a story-driven process and how did you deal with it?