Entries Tagged 'Ruby' ↓

I’ve been working at Intent Media for a while now and I’m just getting some time to come up for air to talk about what we’ve been doing while building our software platform.

This will be the first in a series of articles about what I’ve learned over the last year.

For a little background, our engineering team is currently 9 people strong (in a 20 person company). When I joined the company to help build the team, there wasn’t a line of code written and I was the only technical member of the staff.

Over the last 9 months we’ve built what I consider one of the finest pieces of software that I’ve ever worked on. It’s scalable, it uses the right languages for the right jobs, and it does exactly what it needs to do.

It’s also remarkably bug free.

I credit a large part of that (on the tools and good decisions side, that is; I credit most of it to our team) to Cucumber.

One of the first decisions we made (after writing a unit test and our first bit of code) was to write our first Cucumber test. One of the things that I learned while working at Collaborative Drug Discovery was the importance of automated functional tests.

I, for one, will never do another project without them.

At Intent Media we use Cucumber to drive Celerity across our entire software stack. Currently, this consists of 3 Java applications and a Rails application that interact with MySQL and a HDFS data store. Every single time we check in new code we build the entire stack in TeamCity and run our entire suite of functional tests against it.

As of today, I’m currently greeted with this on our TeamCity server:

Each of those “Tests passed” is a step in a Cucumber scenario. For example:

Given I login as “admin@intentmedia.com”

Over the last 9 months we’ve slowly built up a library of steps that encapsulate everything that we need to do with our apps so far. We’ve also found that we’ve gotten a huge amount of reuse out of our existing steps as we’ve grown the app. So far, there haven’t been any scenarios that we couldn’t tackle in an automated manner.

This suite of tests gives us the best safety net I’ve ever had on a project to try large refactorings. In fact, we often do refactorings across the data model that affect several pieces of software in the stack, often in unexpected ways. Cucumber (along with our unit tests) lets us immediately surface any problems in both our applications themselves and the communication between the apps.

If you haven’t checked out Cucumber yet, you owe it to yourself to do so. It’s one of those life-changing pieces of software that’s amazingly simple to get started with, but quickly becomes one of the most important tools in your toolbox.

Quick update on the last 6 months:

• Moved back across the country to NYC
• Took a job as Head of Engineering at Intent Media (We’re Hiring! We’re funded!)
• Been working in Java and Rails

I’ve got plenty of things to write about, so I’ll try to get back on the horse soon.

The hardest thing to see is what is in front of your eyes. -Goethe

It’s been about 6 or 7 months since I’ve done any Rails work.

I’m in a post-Rails world.

After working in Rails for 3 years, I was burnt out. I took some time off to try to learn the finer points of marketing and running a business, and then attempted to start a new business just before the market cratered. Not a great idea.

Now I’m back writing code all day and it’s great. My days now consist of coming up with test cases to solve problems and then solving them.

Something is different though.

I used to write a ton of test cases in the context of my Rails apps.

Now I write high level test cases and then write code and pick and choose technologies to solve them.

I feel like this is a natural progression for those of us that practice TDD.

Take one more step back, use something at an even higher level to write tests (I’m using some Cucumber and some rSpec), and then implement a bunch of little apps to make what you want to happen, happen.

It keeps things simple, and keeps the problem from growing too large.

In my current project I’ve got a cluster of Nanite agents up and running with a couple of Merb apps talking to them on the front end. It’s something that I never could have done if I was trapped inside of the Rails box.

Hell, it’s probably something that I never even would have thought of.

I think that the Ruby world is eventually going to end up in a model like this, writing small simple apps that all talk to each other, and can be replaced or upgraded at any time.

By writing a lot of tiny apps, I’ve been able to solve problems now in days that used to take weeks or months.

Plus I’m not getting locked in to anything. If I want to replace my Merb app with a Sinatra app next month, it’s not a huge project. If I want to swap in some code running on the Maglev Alpha that I’m playing with or deploy some components written in Clojure, I can do that too, without being intimately tied to anything.

All of my hard/long running logic is well tested, encapsulated, and most likely running in little agents on the wire.

I guess what I’m saying is that if you’ve been writing Rails code for a long time, perhaps now is a good time to step outside of the rules that Rails imposes upon you and look at things from a different angle.

You might find it refreshing and helpful.

I certainly did.

If you haven’t already heard, Rails is now Merb, and Merb is now Rails.

I’m a bit torn about how to feel about this, having dedicated a lot of time lately telling people to use Merb and why I think that it’s superior to Rails.

On the one hand, it’ll be good for Merb usage, as people using Rails 3 in big projects will now essentially be using what Merb 2 would have been.

On the other hand, I’m concerned with the technical hurdles that the core teams are going to have to deal with in order to make this actually work. There’s a lot of old crufty code in Rails that needs to go, although I’m not sure what removing it would do as far as backwards compatibility is concerned.

I’ve been talking in IRC all morning about this, and I’m sure I’ll have more thoughts as things move forward, but for now I’ll just post some of my favorite quotes from today’s IRC session here:

  <devjj> Modular meaning, what, not included by default, gem install rails-bad-ideas
  <gruntruk> this is like biggy n' pac hugging each other
  <bitbckt> so, I presume the quality of conversation in #merb will disintegrate into #rubyonrails :-/

Certainly should be an exciting couple of months as this merge happens…

(Credit goes to an IM from my friend Will for the title of this post.)

Just a quick note to let people know that the Merb community is finally starting to address the lack of good documentation on Merb. I’m personally helping out with the open source Merb Book.

A preview version can be found here: Open Source Merb Book

The source can be found here: GitHub

We’ve got 20 plus people working on it, and considering that it’s only been up on GitHub for 12 days it’s pretty amazing what we’ve already accomplished.

If you want to learn more about Merb you should definitely check it out.

A long time ago, I did a more than a little bit of open source work. Unfortunately, I always found the process of being a part of an open source project to be a pain.

The process usually went something like this:

• Fix a bug in the code.
• Post patches to the dev list.
• Bug people to get your patches accepted and committed.
• By the time they are accepted, someone else has committed something that breaks them.
• Wash, rinse, repeat

Once you do this enough times, you get commit rights and you don’t have to deal with the process anymore.

I’ve never enjoyed it. Far too much overhead and bullshit.¹

So for the last (more than a) couple of years I’ve taken to forking code when I needed to and then never bothering to give the patches that I wrote back to the maintainers.

Clearly not the best situation, but it worked for me.

GitHub changes all of that.

On GitHub, I can fork anyone’s code, patch it, push my changes back, send a pull request, and then not worry about it.

I can share my code with no overhead.

Example: I fixed a bug in the merb cucumber bindings last week. I pushed my changes to GitHub and sent a pull request to the maintainer. My changes haven’t been pulled into the main tree yet, but I don’t really care.

One look at the GitHub merb cucumber network graph should show any prospective users of the library that development has happened on my tree beyond that of the original tree. Hopefully they’ll take the time to look at it and use the most up-to-date code.

Either way, I don’t have to worry about it anymore.

Also, I love that GitHub lets me share my half finished projects.

I recently posted a partially done Merb based blog called Gutenberg on GitHub.

Will I ever finish it? Who knows.

At the very least it’s a simple Merb example project for people to learn from, and I already know people who have looked at it to learn more about how Merb works. That already makes it worthwhile to me.

Again, no overhead.

Just push code onto GitHub and it’s live to the world.

No pain. No overhead. Just code.

1. As an aside, I really like commit policies like one that Rubinius has, namely, “anyone who submits a patch for some part of the Rubinius project and has the patch accepted will be given commit rights to the mainline repository.” We need more projects with commit policies like that.

As some of you know, we’re building our new startup on top of Merb. One of my favorite things about Merb is how much easier it is to understand than Rails is when you need to hack on something that’s outside the realm of what the framework expects you to use it for. I think that Ezra Zygmuntowicz put it best in his recent tech talk on Merb:

A lot of people tend to end up treating Rails as a blackbox because it is eighty thousand or so lines of code and it’s a little bit hard to step into and figure out what’s going on because there’s so much advanced Ruby meta-programming going on inside of there. I kind of feel like the place for meta-programming and that kind of stuff is in your own application code. I’d rather have the foundation that you’re building on top of be as simple and as easy to understand as possible so that when you do hit the wall and you need to look inside the framework you don’t get grossed out and you can get in there and actually do stuff with it.

Amen.

I was having a conversation with a friend the other day who said something along the lines of “lately at work, we’ve pretty much just been rewriting Active Record to make it not suck as much”.

I was thinking about it, and I’ve never really heard anyone say anything good about Active Record. It’s slow, it uses a ton of memory, it’s not thread-safe, etc.

Luckily, the community is starting to tackle those problems with a couple of new projects aimed at doing a better job of OR mapping.

The one that I’ve used the most, and that we’re building the software for my new company on, is Datamapper.

The development philosophy behind Datamapper tries to keep everything as fast and as lightweight as possible. It does this by being as lazy as possible in what it loads to keep memory usage down, and by breaking out as much functionality as possible into plugins, so that you only need to include what you’re really using in your application.

In my stress tests here using my new application, I’ve seen significant speed increases across the board compared to ActiveRecord. (I’m writing a Merb app, and it’s relatively easy to switch back and forth between the two of them at this early stage in development.)

Datamapper also has most of the major ActiveRecord plugins ported to it (as well as a CouchDB adaptor that we’re using for some of our models).

If you’re starting a new Ruby web application, I strongly suggest checking it out instead of just settling for the default choice of Active Record.

The easiest way to run Merb from edge:

$ sudo gem install sake
$ sake -i http://github.com/wycats/merb-more/tree/master%2Ftools%2Fmerb-dev.rake?raw=true
$ sake merb:clone
$ sake merb:install:all

This will install the edge versions of all of the merb gems.

To keep things up to date, navigate to the directory that you ran the commands in and do:

$ sake merb:update
$ sake merb:install:all

For the last year or so, this blog has been running slow, slow, slow.

From time to time, I tried to figure out what was going on, but I figured that Typo upgrades were just causing things to slow down and use more and more memory.

Until tonight.

Tonight I did a database dump and it came to 44 MB.

There is no way I have 44 MB of content in this blog.

So I had a look at the dump and realized that I had close to 100000 sessions in the sessions table, and that there was a table for Sitealizer plugin that’s no longer part of typo.

So after a drop of the Sitealizer table and a

rake db:sessions:clear

Typo is now using 20% of the memory it was using before, running much, much faster, and the database dump comes to a slim 476 KB in size.

So this is my reminder to myself (and everyone else) to be sure to clear out your sessions table from time to time.