Obie Fernandez

My Reasoned Response about Scala at Twitter

I'm glad that Twitter is working to resolve its scaling issues. It's a service that I love and use on a daily basis and from which I have benefitted immensely. As far as I'm concerned, Twitter is a case-study in how Ruby on Rails does scale, even in their hands. Also, I know Alex Payne in person. Not very well, but enough that if I was in San Francisco and sat down to share a beer and discuss the Ruby/Scala debate, I imagine that it would be a fairly civil conversation.

My interest in the question of Ruby vs. Scala at Twitter had mostly consisted of curiosity and amusement, at least until last night. First I had an exchange of tweets with Alex which raised my warning levels, and second Alex wasted no time in creating for himself a virtual nintendo invincibility star for himself with this blog post: Mending The Bitter Absence of Reasoned Technical Discussion.

Who can disagree with the need for more reasoned technical discussion? Nobody, but frankly the timing of his essay made me throw up a little in my mouth. The nausea only grew this morning when I checked on how much it's being re-tweeted. Even though Alex's star is probably going to take a little while to wear off, here are some of the facts of this "debate" for your consideration and historical record:

First of all, my exchange with Alex last night, which I triggered by re-tweeting Jeremy McAnally:

RT @jm: If you have to use kind_of? all over your code to mimick a "type system," you're doing it wrong. (I had exactly the same thought!!!)

To which Alex responded:

@obie Indeed, using kind_of? in that way is "doing it wrong" in Ruby. But as our codebase grew, it became a necessity to combat bugs.

Hmm... A "necessity to combat bugs". What bugs, exactly? Bugs in Ruby or bugs in Twitter's codebase? Non-deterministic bugs? At this point I wasn't angry or anything, just curious and a little bit suspicious. So I asked:

@al3x Doesn't make sense. Do you mean non-deterministic bugs? An in-depth explanation of the bugs you're talking about would be enlightening

To which Alex quickly replied:

@obie Yes, I mean non-deterministic bugs in the giant, legacy, spaghetti parts of our system. Unexpected objects flying around.

Ah, suspicions confirmed. Alex is indeed scapegoating Ruby for Twitter's shitty legacy codebase and the mistakes that they made during what is unarguably a very steep growth curve that would have challenged even the most brilliant team of developers. But don't take my word for it, here are some direct quotes from Alex himself in the recent Artima interview:

Twitter continues today to be primarily a Rails application, with a bunch of Ruby daemons doing asynchronous processing on the backend.
Over time we found that although Rails works great for doing front-end web development, for doing heavy weight back-end processing, Rails had some performance limitations at runtime. And I think that—and this is more my personal opinion—the Ruby language lacks some things that contribute to reliable, high performance code, which is something we’re very interested in as we’re growing as a business. We want the code we write to be correct and maintainable. We want to keep our costs down—all the things most businesses want out of their stack. So that’s why we started looking at Scala.

Most damning:

I’d definitely want to hammer home what Steve said about typing. As our system has grown, a lot of the logic in our Ruby system sort of replicates a type system, either in our unit tests or as validations on models. I think it may just be a property of large systems in dynamic languages, that eventually you end up rewriting your own type system, and you sort of do it badly. You’re checking for null values all over the place. There’s lots of calls to Ruby’s kind_of? method, which asks, “Is this a kind of User object? Because that’s what we’re expecting. If we don’t get that, this is going to explode.” It is a shame to have to write all that when there is a solution that has existed in the world of programming languages for decades now.

Adding fuel to the fire, Alex seems to have created quite a negative impression about Ruby and Rails with his recent Web 2.0 talk in which he advocated usage of Scala. Tell me the impression you get from coverage of his talk in articles such as this one in MIT's Technology Review, which makes the following incendiary assertions:

The company is leaving behind a programming language that has caused it much pain in the past, and instead embracing a new and somewhat obscure language called Scala.

[...]

Right now Twitter's service is a hybrid of programming languages, Payne says. The user interface runs on Ruby on Rails, which is "fine for people clicking around Web pages," he says. But by the end of the year, Twitter hopes to have a set of services in the back end that are written entirely in Scala. And it's the company's plan to make sure that all the third-party services that connect to Twitter via the application programming interface (API) go through Scala code, bypassing Ruby on Rails completely. "When you're talking about a bunch of programs hitting the API rapidly," Payne says, "We found we can better optimize things...using Scala."

Did you actually say "fine for people clicking around Web pages", Alex? Maybe that Technology Review article is just sloppy journalism, a subject that Alex himself lamented not too long ago. But I don't think that's the case. I think that Alex is slyly pushing his Scala agenda forward, yet acting surprised when Ruby folks call him out for it.

What agenda? His agenda as an advocate of Scala and author O'Reilly's Programming Scala book. He said so himself in Recession Engineering:

I’m first to admit that I have something of an agenda with this prediction. I spend a fair bit of my time working with and writing about the Scala programming language, which provides the expressiveness and flexibility of dynamic and functional languages like Ruby, Python, and Lisp with the performance of C++ or Java. - Alex Payne

Have I been guilty of this stuff in my own past as a Rails advocate with a strong agenda? Yeah, probably so. Which is why I know it when I see it. :)

RMM Behind the Scenes

Rails Maturity Model (Behind the Scenes) from Hashrocket on Vimeo.

Take sneak peak at what Hashrocket is cooking up based on the ideas for RMM (Rails Maturity Model). It's not about certification, rather it's about having an objective and neutral way of tracking the behaviors that Rails shops consider their best practices and having those vouched for and endorsed by the community at large. We don't think this implementation controversial at all, but here's your chance to decide for yourself.

For Giles, et al.

RMM IS NOT ABOUT CERTIFICATION

However, in honor of Giles' 1000+ word obliteration of a concept that I'm not pushing, it's my honor to present him with this special Strawman University certification:

I'm starting to think I should forget about pushing higher standards of web development and pick up the cause of reading comprehension.

Apology for a Rails Maturity Model

5/16/09 UPDATE: The Rails Maturity Models website is now soft-launched and available for evaluation at http://railsmaturitymodels.com It allows Rails development firms to document their own set of practices, their "maturity model" so-to-speak. Many people have misunderstood the concept of the site and attacked it, but I don't think it is a particularly controversial endeavor. As a matter of fact, I believe strongly that the site will be a useful tool for community introspection and understanding our success.

The outcry over my idea of establishing certification of Rails development shops based on a "Rails Maturity Model" (RMM) gave me pause, to say the least. (It also gave me a bit of heartburn.) I was wrong to suggest auditing and certification, although I want to point out that I didn't write that message for a wide audience and it was purposely composed in a tongue-and-cheek and informal style. Lesson learned.

To be clear, what I'm saying is that I agree that certification is a bad idea, full-stop. Especially for individuals. I mean, it might only be a matter of time before someone tries to establish a certification program for Rails, but I promise it's not going to be me. At the moment, nobody is proposing Rails certification, so if you're one of those people that got riled up about it, just stop and chill the fuck out. Seriously.

[ As I was writing this blog post, I took a break and walked into the kitchen where Desi is finally wrapping up some of the unpacking from our recent move. Guess what was sitting on the counter? My framed Sun Certifications from 1999! Couldn't resist posting a pic of the Architect certificate, even though I expect to be mocked mercilessly for doing so. ]

Maturity Models

On the other hand, a "maturity model" does not have to be coupled to certification and I don't think there's anything fundamentally objectionable or offensive about trying to define a maturity model for Rails. I envision it serving as a roadmap for Rails developers to adopt best practices defined by the community:

A maturity model can be described as a structured collection of elements that describe certain aspects of maturity in an organization. A maturity model may provide, for example :

• a place to start
• the benefit of a community’s prior experiences
• a common language and a shared vision
• a framework for prioritizing actions
• a way to define what improvement means for your organization

A maturity model can be used as a benchmark for comparison and as an aid to understanding - for example, for comparative assessment of different organizations where there is something in common that can be used as a basis for comparison. Wikipedia

I think an official maturity model is crucial as Rails enters the mainstream software development industry. In the words of Robert Fischer: "Once upon a time an idea like #rmm would be silly: the very fact you knew Rails implied something about the kind of dev you were."

That time has passed, but Zed is still wrong. Rails itself is not and was never a ghetto. It's more like the entire field of custom web software development in general is a ghetto. And what's happening now is that every year that passes more of the inhabitants of that ghetto switch from PHP, .NET, Java and ColdFusion to Rails as their drug of choice:

Rick Bradley: "Furthermore, In reality, where we happen to live, there seems to actually be a power law at work:  the vast majority of software and teams plying our trade are so bad that they fail nearly every metric we could reasonably apply.  Only the vanishing minority of products and practitioners pass even a single gameable metric.  Yet we ultimately have little means of discerning for ourselves, for our clients (or for differentiating for our future clients) the difference between one or the other."

In my original RMM message to the mailing list I referred to some people as "unwashed masses". Yes, that sounds elitist. No, I don't think there's anything wrong with that. Note that if you're reading this, then I'm probably not talking about you. I'm talking about people that don't keep their code in source control, nevermind git vs. svn. I'm talking about people that don't do any automated testing, ever. I'm talking about people who write an entire big application in one controller. I'm talking about people that code big database queries into both sides of a spaceship operator. The list goes on and on; I've even done major conference talks about their horrors.

A Rails maturity model would point the way out of the ghetto for individuals and organizations who wish they were doing high-quality work, but have no idea what it means to do so.

Rick Bradley: "To be able to deploy a standard of comparison (whether distributed or centrally administered and computed) that forces products or teams to either score in the vast gutter of mediocrity, or work diligently to at least game performance on a number of axes (when such gaming is, realistically, often more difficult than just working to better oneself) to separate from the pack -- would imply a vast improvement in the current and long-standing state of affairs."

Lots of people on the original RMM thread suggested that market forces should be allowed to run their course. The problem is that there seems to be a bottomless well of amateurs ready to step up and take the place of the ones that go out of business. Where there's money to be made you will have opportunistic players ready to give it a go, and don't get me wrong -- I'm not trying to say that there's anything fundamentally wrong with being an amateur. It's just that at the very least the community should give those newcomers a more-or-less official set of guidelines for how to produce quality work:

Rick Bradley: "In my limited experience the difference is usually striking and obvious:  "this is trash" vs. "wow, this is pretty good".  In the Rails community the latter is really only elicited by the products of about 10-20 shops (those shops Obie was trying to catalog in one of his solicitations this evening).  It would be great if we could get that number up to 50, 100, or even 1000.  I'd like to believe that putting a workable set of metrics out there would eventially bias progress in that direction."

I have no idea whether Rick is correct that only 10-20 shops in the Rails community consistently produce "pretty good" work. None whatsoever. That questions is actually what prompted the RMM thread on the mailing list, because I asked members to reply with their list of the top 10 shops (consultants) in their opinion. It felt like everyone had a pretty hard time coming up with answers. In fact, I asked because the question came up among myself and a colleague, and I had a hard time coming up with a solid list myself, and arguably I'm one of the most well-connected people in the community.

I don't think there's anything inherently bad in what I'm talking about here. I don't think it'll take the fun out of Rails. I don't think it kills innovation or my credibility. I don't think it's enterprise-y. I don't think it's ridiculous or that it means Rails has really "jumped the shark" (that's happening in Las Vegas in May of this year, j/k). I do think it's about leadership and taking responsibility for maturing the community that lets us make a happy living, thereby ensuring its longetivity. I do think it's about expanding the pool of qualified and competent Rails developers out there.

Actually, I hate to end on a bad note, but I really think that there's some urgency to all this and that our collective failure to set high standards for the community is strangling the continued growth of Ruby and Rails overall.

Tom Mornini: Our customers at Engine Yard are screaming for more developer capacity, and in fact, TWO previous customers abandoned Rails (one for PHP and the other for Java) simply because they could not find enough skilled and savvy Rails developers to move their projects forward.

It's not a problem only for Tom and Engine Yard. At Hashrocket we've been helping organizations transition from legacy technologies to Rails and they all have difficulties finding qualified developers. I know that folks like Courtenay at ENTP are having the same problem. Please don't shoot the messenger. Something has to be done about this, and I think it's more important than Rails 3 or anything else we are currently excited about.

Notice I didn't actually propose the contents of the RMM this time around. I have been chastised by the community reaction to my first list, particularly for including things like pair-programming. I do have an idea for a way to have the RMM be community-generated and maintained, and I'm going to put it on my list of things to do in my copious free time. If someone wants to help me with it, let me know via email.

Railsconf 2008 Slides for Worst Rails Code...

I'm happy to make available the slides for my well-received Railsconf 2008 presentation: The Worst Rails Code You've Ever Seen (and how not to write it yourself)

obiefernandez-worstrailscode-railsconf2008_slides.pdf

Greg Pollack (of RailsEnvy.com fame) included an interview with me about my talk in his Railsconf 2008 Videos collection. Direct link to the Vimeo clip is here.

I want to publicly thank Rocketeer Rein Henrichs for his somewhat late-minute agreement to co-present, even though it meant being up and alert at 9 am on Sunday morning after extreme sleep deprivation for the duration of the conference. We were able to riff off each other and crack up the audience -- without resorting to too many inside jokes. (Did you see what I did there, Rein?)

I also want to thank all my readers that responded to my call for bad code examples a couple of weeks ago. Explore the comments on that post for some good examples that I ended up not using.

One of the coolest things about the talk was that several community-benefiting ideas popped up because of it. I mentioned the possibility of signing authors to write a Rails Antipatterns book for my series from the stage, and Chad and Tammer from Thoughtbot volunteered pretty much immediately. Greg from RailsEnvy also suggested that he help me get the talk recorded as a video podcast and that'll happen sometime soon I'm sure, as well as the idea of turning the talk into some sort of recurring video podcast by me and Rein. We'll see... copious free time and all that being what it is.

Oh yeah, on a final self-congratulatory note, I also claim bragging rights for packing the room with 1000+ attendees on a Sunday morning! Maybe the scheduling gods will be kinder to me next year in Vegas?

Truly Awful Rails Code (Send it to me!)

There's tons of material out there telling us how to properly write code (or attempting to anyway), but in an effort to be creatively different, I decided that my Railsconf talk this year will be all about bad Rails code. Entitled "The Worst Rails Code You've Ever Seen...", I'm basing it on both code and practices that I've both done myself, stumbled across as a neutral observer or rescued during the last few years.

After working through a draft, I'm feeling a little short on examples, so I'm reaching out to you for some help. Please email me your heinous Rails coding examples, stuff that you're truly embarrassed about. I'm looking for stuff that has obvious "better ways" or not, the latter being good topics for discussions of how Rails occasionally leads developers down the wrong path.

If you want to be a little bit more anonymous than email permits, you can also post a comment to this entry linking to snippets on Pastie. Not to worry, none of the example code in the talk will be direct copy-paste of actual client code or stuff that you submit. I plan on changing variable names and anonymizing stuff to the best of my ability and for my legal protection.

If I get enough material, I plan to make the content of the talk and submissions into a regular series of posts on this blog. Thanks in advance for your help!