I was playing around with Cascalog today, working on rewriting some of our mapreduce jobs, when I came across this post on knowledge debt by Nathan Marz.
Knowledge debt is something that I think about a lot (although I never had a good name for it until today). When you’re working in a startup like the one that I’m currently involved in, there are often times when you need to get a release out of the door by a certain date. Missing a release date with a deal with Expedia on the line, for instance, isn’t really an option.
At the same time, it’s hard to draw a clean line as to when some of the things that we’re doing aren’t really working anymore, or are actively slowing us down. It always seems like we’re making progress, but who really knows if we could be doing some of our work in a much more efficient way.
Of course, money helps with everything, and we have more and more of that lately, which I’m planning to put to use to help solve this problem.
My current plan is to hire 12-15 more engineers over the next year in order to give us more breathing room to do new things. We need time to try out things like Cascalog (and Scala, Clojure, Cassandra, etc) and to spike out new approaches in our codebase, while at the same time continuing to make steady progress on the work that needs to get done.
Hopefully some of these experiments will pan out and give us a big boost in velocity, but even if they don’t we should still be doing them.
I’m also hoping that each new engineer on the team shakes things up a bit. Every person that walks in the door should be teaching us something new, or else they shouldn’t be walking through the door in the first place. I already know that we have a pretty amazing codebase but, looking at some of the problems that we’re about to tackle, I can’t wait to see what 2012 is going to bring.