Obie Fernandez

Excellence Is In the Details: GMail EOM

It always gives me such great pleasure when I notice excellent attention to detail in software that I use on a daily basis. In this case, it's a detail that I've noticed before, but never have heard anyone talk about.

Every once in awhile I send someone an email with just a subject line. If you enter a subject in GMail and hit send, you'll get a warning dialog asking if you really want to send without a body.

However, if you follow the convention of suffixing your body-less subject line with the string [eom] and hit send, then GMail doesn't bother prompting you with the warning dialog, because someone on that team knew that EOM stands for "end of message".

Excellence is in the details. They don't always have to be visible or even talked about. In fact, when you discover those great little details in practice it can really be a really delightful and loyalty-building experience.

The Rails 3 Way Highlights: ActiveSupport's Class.class_attribute

As I'm wrapping up work on The Rails 3 Way, I'm coming across little nuggets of interesting content suitable for blogging. Time permitting, I'll publish them here over the course of the next few months.

---

In this case, I cover a new ActiveSupport core extension to Ruby's Class object. It allows creation of class-level attributes in a simple and powerful manner that I think is a lot clearer than the mess of cattr_ and class_inheritable_ methods that precede it.

class_attribute(*attrs)

The class_attribute method declaratively defines one or more class-level attributes whose value is inheritable and overwritable by subclasses and instances, like so:

class Base
  class_attribute :setting
end

class Subclass < Base
end

>> Base.setting = "foo"
=> "foo"

>> Subclass.setting
=> "foo"

>> Subclass.setting = "bar"
=> "bar"

>> Subclass.setting
=> "bar"

>> Base.setting
=> "foo"

This behavior matches normal Ruby method inheritance: think of writing an attribute on a subclass as overriding the parent's reader method. Instances may overwrite the class value in the same way. (Note that the following code samples create anonymous classes to illustrate usage in a more concise fashion.)

klass = Class.new { class_attribute :setting }
object = klass.new

>> klass.setting = "foo
=> "foo"

>> object.setting = "bar"
=> "bar"

>> klass.setting
=> "foo"

To opt out of the instance writer method, pass :instance_writer => false.

klass = Class.new { class_attribute :setting, :instance_writer => false }

>> klass.new.setting
=> NoMethodError

The class_attribute method also works with singleton classes, as can be seen in the following example.

klass = Class.new { class_attribute :setting }

>> klass.singleton_class.setting = "foo"
=> "foo"

For convenience, a query method is defined as well, which allows you to see if an attribute has been set on a particular class instance. 

klass = Class.new { class_attribute :setting }

>> klass.setting?
=> false

>> klass.setting = "foo"
=> "foo"

>> klass.setting?
=> true

Professional Ruby Series Rough Cuts Available

My series has three exciting titles coming in 2010 that I wanted to share with you, since they're now available in preview form.

The Rough Cuts service from Safari Books Online gives you exclusive access to an evolving manuscript that you can read online or download as a PDF and print. A Rough Cuts book is not fully edited or completely formatted, but you'll get access to new versions as they are created.

Service-Oriented Design with Ruby and Rails By Paul Dix Published August 28, 2009 Read a free sample chapter: Chapter 2: An Introduction to Service Oriented Design; Overview Buy Now This book covers the tools and techniques for building service architectures in Ruby that are designed to scale and operate in the cloud or with legacy enterprise systems.

Rails AntiPatterns By Chad Pytel, Tammer Saleh Published Jan 5, 2009 Read a free sample chapter: Chapter 3: Excess Ruby in the Views: Learn the View Helpers That Come With Rails Buy Now In Rails AntiPatterns, the authors provide many real world antipatterns examples along with practical advice for how to avoid them in the first place. Each AntiPattern is demonstrated with real world code, and solutions for refactoring are presented that are based on sound Object Oriented principles and established Ruby on Rails best practices.

Reporting and Dimensional Modeling with Ruby By Matthew Bauer Published Jan 5, 2009 Buy Now Both a reference and tutorial, this book shows how to implement the reporting and dimensional modeling concepts using a variety of tools in Ruby.

10 Reasons Pair Programming Is Not For the Masses

Hashrocket, or more specifically our beloved "Big Tiger" Jim Remsik, was recently profiled in the New York Times in an article about pair programming. They titled it For Writing Software, a Buddy System and it's gotten a fair amount of attention on Twitter, Hacker News and Reddit. (If there are other big discussions going on about it, please let me know in the comments. Thanks.)

I think pair programming is one of our most important competitive advantages at Hashrocket. My teams produce some of the highest-quality code I've ever seen in 15 years in the biz. We are able to effectively leverage XP-style client acceptance in our process because our code is so defect-free.

I'm damn proud of what we do at Hashrocket and talk about it often, but usually while trying to stress the supreme importance of context and talent in making it work. My homepage still says "Helping to evolve the world of software into a more fun and productive place to live, work and play", but pair programming, especially all the time, is one of those things where increasingly I have to advise most Agile idealists that it probably won't work for them, and here are ten reasons why:

10. Most software managers don't want to invest in the necessary hardware

Either they don't want to invest in the appropriate equipment (for personal or political reasons), or they are not allowed to do so.

Our pairing workstations are fully-outfitted Mac Minis ($1049) with a $99 Dual-link DVI adapter connecting them to an Apple's 30" Cinema Display ($1800). Each pairing workstation also requires a pair of mice ($98), keyboards ($98) and elevators notebook stands ($80). Finally, we also buy a pair of MagSafe power adapters ($159) for developer's notebooks. Mind you, we don't invest in all this fancy Apple gear because it's cool. We do it because we care about having the best tools available, a sentiment that the masses usually have to live without.

Incidentally, each of Hashrocket's developers has their own MacBook Pro, that they are required to own and maintain, based partly on the philosophy that a craftsman has his own tools.  That is a hard policy to pull off in mainstream software shops, because it can be seen as a turn-off for potential recruits, not to mention that most corporate IT shops suffer conniptions about not having control over machines that are connected to their networks. C'est la vie.

9. Most software shops are not configured for pair programming

Cubicles do not work for pair programming. Period.

Most office environments have cubicles and most software managers are stuck with the environment they have and lack the political or budgetary capital to do anything about it.

In order to effectively pair program all the time, you have to have the right furniture and workspace. Comfort is key. Big wide desks, comfortable chairs and elbow room. Our desks are 2 m x 1 m, cost about $600 each and are height-adjustable. There is enough room for the pairing workstation in the middle of the desk, plus dual notebook computers to each side.

There are no cubicles or private spaces available, because developers are not expected to work in private, nor are they supposed to be doing non-work tasks in the office. However, our office is partitioned into a number of "war rooms" and casual seating areas that can be appropriated depending on needs of the team. In particular, client calls are sometimes conducted in quieter areas.

8. Most software shops use traditional hiring practices

Relying on recruiters to source candidates and then hiring someone after a few phone calls and a day of interviews, you know, the way that most places do it, is one of the worst ways to bring on talent. You really don't know what you're getting. The savvy mainstream software manager might swing a contract-to-perm policy, but still has to recruit the contractor in traditional ways.

The best way to hire talent is to drop them in actual work situations for at least a week and see what happens. And that's the catch. Since most shops don't pair all the time, they don't have a built-in infrastructure to accommodate just "dropping someone into a real work situation". The poor candidate would be in over their head. Our candidates spend their interview week rotating on actual production code, pairing with the same people they'll be working with if they're hired. Everyone gets a say in whether to hire a new recruit, and hesitance from a developer that actually paired with them means they do not get hired.

7. Most software shops tolerate anti-social behavior (halitosis included)

At your average software shop, someone can be considered a "good programmer" even if they are a complete asshole. Nobody wants to pair all the time with assholes, even other assholes. Same thing goes for prima-donnas, chronic farters, people who don't bathe enough, people with halitosis, jerks, idiots, lazy bums. There are myriad conditions that could make pair programming a gruesome experience and it is absolutely management's job to make sure that those problems are kept under control or eliminated. As in censored. Or fired.

Face it, most software managers are under-staffed (see below), overworked and/or under-qualified. They would rather let the group deal with Asshole Architect, Mr. Stinky or Jimmy the office jerk than to go around provoking difficult conversations and making more work themselves. Besides, Stinky's code is pretty good, isn't it? Right... I've had to dismiss otherwise-qualified people for being assholes. People that I otherwise-needed. It wasn't easy, but it is crucial to do it anyway.

Full-time pairing requires strict implementation of the No-Asshole Rule. In the words of Robert Sutton, "enforcing the no-asshole rule isn't just management's job". The entire team has to feel empowered to call out and reject anti-social behavior. That kind of empowerment is extremely rare in corporate America. It's easier to take it during the day, rush out of the office at 5 pm on the dot and vent your frustration at happy hour.

6. Most software people don't understand pair productivity

Those of us that have seen it in action know that good full-time pair programmers consistently produce higher-quality code faster than individuals working alone.

To the average manager, with his average mindset, two people working on the same thing equals half the work done. To the average programmer, having to pair full-time means they won't get to waste time during the day on Reddit or Hacker News.

Full-time pair programming keeps developers focused and coding, all day, every day, week-in, week-out. The average developer spends maybe a few hours a day actually coding and finds all sorts of other activities to fill in the rest of his day. The difference in experience gained towards mastery of the craft adds up quickly.

I see pairing work so well every day that I consider my career prior to my current job to have consisted mostly of wasting time. When I think back to all the code I’ve written for a job, I’m annoyed at how much less efficient I was then since I wasn’t pairing, and how much better my code and my products would have been if I had paired on them full time.

When you have a second person working with you, you find that you try harder to code well. You’re far, far less likely to be willing to apply duct tape to a problem, because someone else is working with you and he or she is more likely to object to the duct tape. If two people approve a hack and implement it, it’s almost certainly because it’s the right approach given the time constraints. If one person sitting alone decides to pursue a hack, it’s more likely to be a situation where there is a better approach.

Two people attacking a design problem together is easier than one person doing it. Your partner will think of things you don’t, and vice versa. In a non-pairing environment, you can always call someone to a whiteboard to help you work through a design issue, but when you do that you’re only calling for help when you know you need it. The times when you need help and don’t know it go unaddressed – and yet despite the fact that you need help, I guarantee that you’re coding SOMETHING and checking it in. It’s almost certain to be shitty code. The amount of time you spend trying to figure out “what’s going on here” or “why isn’t this working the way I expect” is severely reduced. I’m talking hours of debugging that literally turn into seconds.

Rod Hilton in I Love Pair Programming

5. Most software shops employ under-qualified developers

Hashrocket is a boutique shop, so we can be very picky about who we hire. Candidates have to pair with us onsite for at least a week before the team decides whether to hire. We've hired qualified interns, but we don't hire apprentice or junior-level folks. Occasionally we do extended pairing matchups with client developers in our office, but they have to fit in culturally for it to work.

The bar for success in this industry is set very low. Most software shops have one or two great, productive hackers, offset by a boatload of net-negative producers:

"When doctors screw up (massively) they get sued. When (permanently employed) programmers screw up they get let-go, and they go find a new job, usually with more responsibility and a pay raise. There are no ramifications for software malpractice." 

Jay Fields in The Cost of Net Negative Producing Programmers

Hiring only great people and compensating them very well is expensive. Most software managers don't even buy into the advantages of only hiring great people, thinking that a mix of talent levels is more advantageous.

4. Most software shops are overworked and under-staffed

This reason is related to #5 in the sense that most software shops don't have enough good people to take responsibility for the truckloads of work that they're expected to do. You can't put idiots in charge of important projects, therefore pair programming requires double the amount of "good" people as not pair programming.

Promiscuous pair programming, where there is a weekly, or even daily, rotation of pair partners, requires more people than discrete workstreams, so that pairs can switch without incurring big ramp-up penalties.

3. Most software developers don't like everyone they work with

Good luck putting together more than 15 people in any type of group without some of them hating each other's guts. Now limit your pool of resources to average programming geeks with anti-social tendencies and making your group get along just got a lot harder.

Lots of teams out there are simply too big for a happy group to even be possible. There is research on ideal group sizes for teams, with the consensus falling in the 8 to 14 member range. Unless you're committed to small teams of sociable people, you won't be able to get everyone working together happily. 

Most software managers would rather pay 20 mediocre and/or anti-social developers $75k/year instead of hiring 10 great developers at $150k/year. (Why? I'd bet on they don't want to pay their developers more than what they make.)

2. Most software developers just don't want to work that hard

Laziness is a virtue for programmers, isn't it?

Laziness - The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer.

Larry Wall in Programming Perl

Unfortunately, it seems like over the years as an industry we've simply reduced that advice to "go to great effort to reduce overall energy expenditure" and forgotten about the rest.

Pair programming makes developers work harder than they've ever worked before in their lives. You constantly have someone making sure that you're not distracted or wasting time running down dead ends. Unless you're an asshole who just doesn't care (see #7) you're not going to abandon your pair while you endlessly do non-work-related stuff. Pairing keeps you honest. Being honestly hard-working is still one of the best ways to earn accolades and raises. Accolades and raises usually bring some measure of added happiness.

All that hard work really pays off in terms of success for everyone involved as well as personal growth, but why bother trying? Most software developers are happy to skate along doing the least amount of real work possible without getting in trouble. In other words, they're lazy.

1. Most software shops don't really care about excellence

There are so many things to worry about at your average shop that have nothing to do with excellence. Corporate America cares about lines of code, the bottom line, toeing the line, and whether Joe in Accounting is doing lines in the bathroom during his breaks.

Truly caring about excellence means considering your work to be more than just a job or a paycheck. You have to treat is as a craft. You have to have passion for the success of your clients, company and co-workers. You have to be justly rewarded for that passion and encouraged. You have to care. And frankly, that is just way too much to ask of most people.

Heroku and GoDaddy CNAME

If you're trying to get Heroku working with your GoDaddy-hosted DNS, you're going to have problems adding a CNAME record for "@". The system will complain and ask for a valid hostname, even though GoDaddy's own documentation says that "@" is okay.

The solution is to use the name of your domain instead. A trailing period after the TLD seems to be optional. An A record for @ is not needed with this configuration.

The Hashrocket Way: Pair Programming

To use an idiom that Bryan Liles effectively coined at this year's Ruby Hoedown, at Hashrocket, we "pair all the fucking time". It is a vital part of The Hashrocket Way, our internal philosophy of how to most productively service our clients:

The photo above captures a normal morning at Hashrocket headquarters, where every desk is set up for effective pair-programming: 30" monitor, dual keyboards, dual mice. Carmelyne Thompson and Wes Gibbs are the pair in the foreground.

Veez Remsik, my inspiration for this post and one of Hashrocket's Rescue Mission experts, gives us a must-read personal account of pair-programming as a way of life:

Hashrocket has a few core philosphies that are essentially required for team members. Agile development processes, an adherence to ruby and rails best practices and idioms, and a firm belief in pair programming are three of the biggies. Of these three, pair programming is the technique that has changed me most, and required the most change in my thinking. And not surprisingly, pair programming has proven the most critical to my success at Hashrocket.

Veez has a traditional software development background (unlike the one that several of us 'rocketeers cultivated at places like Thoughtworks). I like the way he describes himself pre-Hashrocket: "You know, the one who does his best work in the morning with his headphones on and drinking an obscenely large coffee. The guy who types madly for a few select hours a day and, during those few hours, produces as much as regular-speed developers do over the course of a day."

Durran Jordan and Matthew Bass pair on our latest 3-2-1 Launch project. Quite often, one of the keyboard/mouse sets is replaced by an actual MacBook Pro in conjunction with the most excellent teleport utility from abyssoft.

Most Rails people I know profess adherence to Agile practices and principles, yet judging from the anecdotal evidence I've collected at conferences and comments made by our guest stars, pair-programming is still very much the exception. Make that real pair-programming, in the extreme programming sense, where all production code is produced by two developers sharing control of one workstation.

Getting a chance to truly pair for the first time is turning out to be one of our "guest stars" favorite experiences at Hashrocket. In the pic above, I captured Paul Barry and Hampton Catlin getting their first taste of pair-programming the way it's supposed to be done.

There are certainly challenges to making pair-programming part of normal business practices. Starting with the fact that it implies keeping an even number of people assigned to a project on a daily basis. Even at ThoughtWorks, where Agile is an important part of the culture, management did not necessarily make this aspect of pairing easy for us — as the principle "resource manager" of Hashrocket, I have to juggle the financial needs of our clients with our need to work in pairs. It's occasionally very tricky, but when you do it all the time, I assure you it gets easier.

---

I'm trying to get back into blogging on a regular basis. The "Hashrocket Way" is a term that has gotten much traction for us internally as a catch-all description of what makes us special, and I plan to write about different aspects of that as much as possible.

About Rails and Ghettos

This is a post that has been a couple weeks in the making (in my head), but the more that time passes by the less interested I am in writing it. Nevertheless, there are a couple of points I want to get out there both for the record and for possible discussion.

First of all, it is obvious that Zed was using that ugly word in its current derogatory sense, and I'll keep that sense in my writing as well. According to the dictionary one of the definitions of ghetto is "an isolated group" or "a situation that resembles a ghetto especially in conferring inferior status or limiting opportunity".

In the greater realm of IT?

Does that definition ring true of Rails in the greater realm of Information Technology, or even just the programming world? I don't think so. The Rails community is not an isolated group, having members across a wide, diverse, range of enterprises and groups of individuals, from the highest-paid professional consultants, to the humblest program-for-fun enthusiasts.

What about inferior status, or opportunity limitations? That doesn't ring true either. If anything, the mainstream fascination with dynamic languages was ignited by Ruby on Rails a couple of years ago, which gives popular Rails practitioners a disproportionate amount of influence (ahem) in the programming world, and a prominent voice in online conversations about the future of our craft.

From my own (admittedly-biased) viewpoint, Rails is continuing to explode in popularity. Among other things, my book The Rails Way is selling extremely well, and we just had to do a rush reprint because demand exceeded even our optimistic estimates. It's currently a top-seller on Safari (a good indicator of traditional corporate developer interest levels), and has been hovering in the hundreds ranking on all of Amazon.com.

But where are the jobs?

To those that would use the number of Rails-related listings on major job boards as evidence of opportunity limitations, I point out that due to extraordinary demand for Ruby talent in the marketplace, it is almost pointless to advertise Ruby job positions via conventional channels. The way to find Rails talent is via targeted Ruby job boards, word of mouth, and blogging. Counts on mainstream job boards are meaningless at best, misleading at worst.

Most Rails developers worth their salt are contracting, simply because it is very lucrative right now. Rates for premium Rails talent is now in the neighborhood of $200/hour and continuing to climb. Average Rails talent costs at least $100/hour.  The work is finding them, not vice-versa. Why are clients willing to pay so much? Because they know we are a cost-effective bunch, doing the same amount of work (as individuals) that would take multiple traditional programmers in other mainstream languages, and with higher-quality as the  "icing on the cake".

How about just in the Ruby community?

Considering Zed strictly within the confines of the Ruby community, and not in the much bigger sphere of programming, I think he's still wrong. Like a newbie OO programmer violating the rule prefer composition over inheritance, he made an is-a/has-a mistake. What is true perhaps, metaphorically, is that Rails has ghettos, and Zed made the mistake of attributing qualities of a ghetto to the whole.

What I"m trying to say is that there are groups of people that have come over to work in Rails, the city of Rails if you will, who have settled into neighborhoods that can be very dangerous (monetarily) to inexperienced clients.

But why can't Zed get a job then?

What put me off about Zed's rant was not so much the content, but rather that it achieved such wide circulation. It wasn't meant to be read so widely, and I don't believe that he had destructive intentions when he wrote it. I lived and worked with Zed for five months during 2007, a time in which we became (and remain) good friends. I swear to you that Zed is a good, friendly guy, in person. (His created, over-the-top, narcissist online persona is another story!) The rant was filled with tons of inside jokes and commentary that really only make sense to other clued-in Ruby people, stuff that comes across as just mean, stupid or crazy to everyone else.

For instance, a significant percentage of readers obviously understood from the rant that Zed is unemployed, starving, and/or stark-raving mad. All three could not be further from the truth. Zed is happily employed by a major, household-name financial institution in New York City, where he comes into work everyday with a nice shirt and tie, and leads a Rails (yes, Rails) development team. Of course, none of this is news to those of us that really know him, but the rest of the world got a totally different story.