Entries Tagged 'Programming' ↓

I’ve been working at Intent Media for a while now and I’m just getting some time to come up for air to talk about what we’ve been doing while building our software platform.

This will be the first in a series of articles about what I’ve learned over the last year.

For a little background, our engineering team is currently 9 people strong (in a 20 person company). When I joined the company to help build the team, there wasn’t a line of code written and I was the only technical member of the staff.

Over the last 9 months we’ve built what I consider one of the finest pieces of software that I’ve ever worked on. It’s scalable, it uses the right languages for the right jobs, and it does exactly what it needs to do.

It’s also remarkably bug free.

I credit a large part of that (on the tools and good decisions side, that is; I credit most of it to our team) to Cucumber.

One of the first decisions we made (after writing a unit test and our first bit of code) was to write our first Cucumber test. One of the things that I learned while working at Collaborative Drug Discovery was the importance of automated functional tests.

I, for one, will never do another project without them.

At Intent Media we use Cucumber to drive Celerity across our entire software stack. Currently, this consists of 3 Java applications and a Rails application that interact with MySQL and a HDFS data store. Every single time we check in new code we build the entire stack in TeamCity and run our entire suite of functional tests against it.

As of today, I’m currently greeted with this on our TeamCity server:

Each of those “Tests passed” is a step in a Cucumber scenario. For example:

Given I login as “admin@intentmedia.com”

Over the last 9 months we’ve slowly built up a library of steps that encapsulate everything that we need to do with our apps so far. We’ve also found that we’ve gotten a huge amount of reuse out of our existing steps as we’ve grown the app. So far, there haven’t been any scenarios that we couldn’t tackle in an automated manner.

This suite of tests gives us the best safety net I’ve ever had on a project to try large refactorings. In fact, we often do refactorings across the data model that affect several pieces of software in the stack, often in unexpected ways. Cucumber (along with our unit tests) lets us immediately surface any problems in both our applications themselves and the communication between the apps.

If you haven’t checked out Cucumber yet, you owe it to yourself to do so. It’s one of those life-changing pieces of software that’s amazingly simple to get started with, but quickly becomes one of the most important tools in your toolbox.

The hardest thing to see is what is in front of your eyes. -Goethe

It’s been about 6 or 7 months since I’ve done any Rails work.

I’m in a post-Rails world.

After working in Rails for 3 years, I was burnt out. I took some time off to try to learn the finer points of marketing and running a business, and then attempted to start a new business just before the market cratered. Not a great idea.

Now I’m back writing code all day and it’s great. My days now consist of coming up with test cases to solve problems and then solving them.

Something is different though.

I used to write a ton of test cases in the context of my Rails apps.

Now I write high level test cases and then write code and pick and choose technologies to solve them.

I feel like this is a natural progression for those of us that practice TDD.

Take one more step back, use something at an even higher level to write tests (I’m using some Cucumber and some rSpec), and then implement a bunch of little apps to make what you want to happen, happen.

It keeps things simple, and keeps the problem from growing too large.

In my current project I’ve got a cluster of Nanite agents up and running with a couple of Merb apps talking to them on the front end. It’s something that I never could have done if I was trapped inside of the Rails box.

Hell, it’s probably something that I never even would have thought of.

I think that the Ruby world is eventually going to end up in a model like this, writing small simple apps that all talk to each other, and can be replaced or upgraded at any time.

By writing a lot of tiny apps, I’ve been able to solve problems now in days that used to take weeks or months.

Plus I’m not getting locked in to anything. If I want to replace my Merb app with a Sinatra app next month, it’s not a huge project. If I want to swap in some code running on the Maglev Alpha that I’m playing with or deploy some components written in Clojure, I can do that too, without being intimately tied to anything.

All of my hard/long running logic is well tested, encapsulated, and most likely running in little agents on the wire.

I guess what I’m saying is that if you’ve been writing Rails code for a long time, perhaps now is a good time to step outside of the rules that Rails imposes upon you and look at things from a different angle.

You might find it refreshing and helpful.

I certainly did.

If you haven’t already heard, Rails is now Merb, and Merb is now Rails.

I’m a bit torn about how to feel about this, having dedicated a lot of time lately telling people to use Merb and why I think that it’s superior to Rails.

On the one hand, it’ll be good for Merb usage, as people using Rails 3 in big projects will now essentially be using what Merb 2 would have been.

On the other hand, I’m concerned with the technical hurdles that the core teams are going to have to deal with in order to make this actually work. There’s a lot of old crufty code in Rails that needs to go, although I’m not sure what removing it would do as far as backwards compatibility is concerned.

I’ve been talking in IRC all morning about this, and I’m sure I’ll have more thoughts as things move forward, but for now I’ll just post some of my favorite quotes from today’s IRC session here:

  <devjj> Modular meaning, what, not included by default, gem install rails-bad-ideas
  <gruntruk> this is like biggy n' pac hugging each other
  <bitbckt> so, I presume the quality of conversation in #merb will disintegrate into #rubyonrails :-/

Certainly should be an exciting couple of months as this merge happens…

(Credit goes to an IM from my friend Will for the title of this post.)

A long time ago, I did a more than a little bit of open source work. Unfortunately, I always found the process of being a part of an open source project to be a pain.

The process usually went something like this:

• Fix a bug in the code.
• Post patches to the dev list.
• Bug people to get your patches accepted and committed.
• By the time they are accepted, someone else has committed something that breaks them.
• Wash, rinse, repeat

Once you do this enough times, you get commit rights and you don’t have to deal with the process anymore.

I’ve never enjoyed it. Far too much overhead and bullshit.¹

So for the last (more than a) couple of years I’ve taken to forking code when I needed to and then never bothering to give the patches that I wrote back to the maintainers.

Clearly not the best situation, but it worked for me.

GitHub changes all of that.

On GitHub, I can fork anyone’s code, patch it, push my changes back, send a pull request, and then not worry about it.

I can share my code with no overhead.

Example: I fixed a bug in the merb cucumber bindings last week. I pushed my changes to GitHub and sent a pull request to the maintainer. My changes haven’t been pulled into the main tree yet, but I don’t really care.

One look at the GitHub merb cucumber network graph should show any prospective users of the library that development has happened on my tree beyond that of the original tree. Hopefully they’ll take the time to look at it and use the most up-to-date code.

Either way, I don’t have to worry about it anymore.

Also, I love that GitHub lets me share my half finished projects.

I recently posted a partially done Merb based blog called Gutenberg on GitHub.

Will I ever finish it? Who knows.

At the very least it’s a simple Merb example project for people to learn from, and I already know people who have looked at it to learn more about how Merb works. That already makes it worthwhile to me.

Again, no overhead.

Just push code onto GitHub and it’s live to the world.

No pain. No overhead. Just code.

1. As an aside, I really like commit policies like one that Rubinius has, namely, “anyone who submits a patch for some part of the Rubinius project and has the patch accepted will be given commit rights to the mainline repository.” We need more projects with commit policies like that.

The easiest way to run Merb from edge:

$ sudo gem install sake
$ sake -i http://github.com/wycats/merb-more/tree/master%2Ftools%2Fmerb-dev.rake?raw=true
$ sake merb:clone
$ sake merb:install:all

This will install the edge versions of all of the merb gems.

To keep things up to date, navigate to the directory that you ran the commands in and do:

$ sake merb:update
$ sake merb:install:all

Then no software would get written, but at least we wouldn’t have any crap software…

My friend Jay posted two blog entries yesterday on the subject of raising the standard for programmers in our industry. I left a comment about it on his blog and talked with him about it on IM, but I figured I’d expand a bit on my thoughts on it here.

First things first, I completely disagree with him.

I don’t think that 50% of programmers should leave the profession. I’ve worked on a lot of projects with a lot of different teams and I can’t ever remember thinking “man, this project would be a lot easier if I got rid of half of this team.”

I can also only think of few Net Negative Producing Programmers that I’ve worked with and most of them usually leave pretty quickly.

Most of the time, I’ve found that when a person isn’t performing up to par, it’s because of a management failure. I think that it’s up to the development leads to analyze their teams and figure out how everyone can best contribute and make sure that people are performing at the top of their game.

Too often I’ve seen people fall into the trap of always being assigned to, or always choosing to work on, the easiest features in the project. They then do a half-assed job because they’re always working on the boring part of the project and don’t feel like a valued part of the team.

Maybe, for a change, we should assign those people the hardest part of the project for a few weeks. Sure, they might not so the best job, but they will work harder because, for once, what they’re working on is challenging and exciting. Plus they know that the entire team is depending on them and is there to help them. That’s how you bring people back into the fold and make them a productive, happy member of the team.

The point that’s neglected in Jay’s arguments is that experienced programmers shouldn’t just be responsible for writing code. It should also be the job of experienced programmers to take those who are less experienced or interested and pull them up to our level.

Forcing them out of the profession is just as much of a failure on our part as it is on theirs.

For the last year or so, this blog has been running slow, slow, slow.

From time to time, I tried to figure out what was going on, but I figured that Typo upgrades were just causing things to slow down and use more and more memory.

Until tonight.

Tonight I did a database dump and it came to 44 MB.

There is no way I have 44 MB of content in this blog.

So I had a look at the dump and realized that I had close to 100000 sessions in the sessions table, and that there was a table for Sitealizer plugin that’s no longer part of typo.

So after a drop of the Sitealizer table and a

rake db:sessions:clear

Typo is now using 20% of the memory it was using before, running much, much faster, and the database dump comes to a slim 476 KB in size.

So this is my reminder to myself (and everyone else) to be sure to clear out your sessions table from time to time.

I use Omnifocus everyday to track what I’m doing, and I love it, but there are tons of UI problems with it that drive me absolutely crazy. There’s a good article about some of the failings of the interface design here (with a video showing some of them), but there’s one interface decision that’s been absolutely driving me nuts lately.

Look at this picture:

See anything wrong with it?

If you said, “There’s no way to set the due date for a project to Tuesday without clicking the little arrow at the top left to go to next month” then you’re right.

What a horrible failure of UI design. I end up up dealing with this at the end of every month, and everytime it’s a pain in the ass. Compare it to the iCal day chooser:

See that? Much better. They just grey out the days in the upcoming month. (They even go above and beyond the call of duty and show two upcoming weeks.) This is what the Omnifocus date chooser should be.

Hopefully there will be an Omnifocus update soon that addresses most of the problems with the interface. I expect more at the price point ($89) that they’ve positioned it at.

I was reading on a plane recently, and was about halfway through the Django book, when I realized that I hadn’t seen anything about how to test my application yet. I tend to be a test first developer and use lots of mocks to play with things, especially at the beginning of a project, so I flipped to the index, turned to the test page, and saw the following:

Testing was still under development when this book was being written, so to learn more you’ll need to read the documentation online at http://www.djangoproject.com/documentation/0.96/testing/

Not a whole lot of help when you’re on a plane.

Still, I was willing to give them the benefit of the doubt. Maybe they could create a bolt-on testing framework that would really be something.

Unfortunately, it’s just not very good.

Perhaps I’m spoiled by Rails and Rspec, but to me, the Django testing framework feels like something that they put in place just because someone told them that they needed a testing framework. It’s not an integral part of the system.

Let’s look at their test types.

First of all we have doctests. These emulate the output of the Python interactive interpreter exactly inside of your test code. All this does is to encourage the bad practice of writing some code, firing up your your interpreter, running a few cases, and then cutting and pasting the results into a file. There’s no need to think through all of the cases or what you’re really doing here. It’s just an after the fact, feel-good, safety check.

Second, we have Unit tests. Not the most sophisticated testing framework in the world, but these look like a better choice, very much like any sort of xUnit tests.

The biggest problem I have with these, and the Django testing framework in general, is that the tests don’t look to be compulsory. Why, in this day and age, don’t we make the generation of a test automatic, and make the user decide to remove it if they really, really don’t want to test, not the other way around?

Testing as an afterthought inevitably leads to poorly tested code.

I certainly hope that, as Django continues to grow and mature, testing take a more central role in the future.

I’m somewhat taken aback by all of the negative reaction to Gemstone being a closed source product in the wake of the Maglev announcement.

Look at this:

Do you really think that the people that own this ship care about paying for software?

I doubt it, but they probably care about things like development speed. And software reliability. And the ability to get someone on the phone immediately if something goes wrong with the software.

That’s the world the Gemstone comes from. I would bet that if the software running the shipping platform for this company falls over, it would lose more money over the course of a few hours then it would cost them to license Gemstone for an entire year.

Here’s another example where JP Morgan has a system built on top of Gemstone that they can not afford to shut down or migrate to another language because it would cost them too much money.

How is being exposed to the toolset that powers things like that going to be a bad thing?

Sure, it’s closed source, but if enough people use it and benefit from it then the community will just do what we always do:

Some of us will pay for it, some of will find clever ways to subvert the license, and the rest of us will get to work on building an open source clone of it.

That’s what always happens, and when it does happen some people will continue buying the closed source supported version (Oracle) and others will move to the free version (MySql).

What makes anyone think that something different is going to happen here?