Kurt Schrader's Blog

CTO @ Intent Media

Scaling Communication at a Growing Startup

When you're first starting a company, communication is the least of your worries. If you have less than 10 or so people, everyone knows what everyone else is doing (and if they don't then they can just walk over and ask them).

Things start to get a little more complicated after that.

At most startups I've seen, once you get to 15 or so people you enter a period of time where everyone still wants to know what's going on everywhere, but the amount of information about what's happening slowly starts to exceed what any one person can know.

This is a dark time, characterized by two things:

  1. The email explosion: Expect to receive tons of emails a day, as the old "just include the entire company on this" starts to overwhelm things.
  2. The tools explosion: People realize that they're overloaded, so they start to look for tools to make life easier. Unfortunately, everyone now needs logins to and needs to know how to use Basecamp/Asana/Jira/Trello/etc.

I think that a big part of the problem is that the creators of these tools don't think about how the tool is going to grow with the company as it grows. They write their tools to do a good job managing one project, and don't stop to think about how that's going to scale up.

Github is a great example of a tool that solves the communication problem around software projects. I don't use Github just because Git is better than SVN, I use it because the toolset they've built around it makes it super easy for me to see what my team is working on and to communicate with them about it.

It's (almost) just as easy for me to put one project on Github and see how it's progressing as it is for me to watch twenty projects (or more).

Tom Preston-Warner, Github's CEO and co-founder recently said:

We focus a ton on communication in this company. And all of the tools that we build are really about enhancing communication; that’s what GitHub itself is all about. It’s about enabling communication between a broader set of people than would otherwise be efficient.

It's great that they're thinking that, but their product is still too technical for most people to use.

Hopefully over time they (or someone else) will be able to build up a suite of tools that lowers the bar on powerful, scaled-communication to a level that anyone from a developer to a salesperson to a janitor can use to see both what the company is doing and how they're doing it (and to dig in if necessary).

For now, if you have a growing startup and communication is getting harder, I suggest doing two things:

  1. Remind people to really think about the number of people that they're sending emails to. People have a natural desire to get all of the emails in a company, but if someone really doesn't need to be included then don't include them.
  2. Avoid a tools explosion like the plague. Encourage people to experiment with new tools, but be ready and willing to throw away bad ones. Just because someone started a Trello board doesn't mean that Trello needs to be a part of your company forever.

All of that said, I still don't think that there's a great answer here (yet).

If you know about a great tool for scaling communication in a growing startup please let me know about it in the comments.

Comments

Building a Data Science Team at a Startup - An Engineering Perspective

A few random observations from spending the last few years building a data science team at a startup, from the perspective of an engineering manager:

You have no idea what your data science team is going to do.

You're just going to know that you have some data and some problems, and that you don't really know how to solve them. Of course, that leads into:

The way that you're thinking about the problem is probably wrong.

Your data scientists are going to need to come in and really blow you away. In the canonical "building data science teams" article by DJ Patil he talks about giving people 90 days from when they come in the door to "knock the socks off" of the company. I think that's a good benchmark. If a data scientist hasn't made you think about your data or your problems in a new way by then they probably aren't ever going to.

A data science team needs a different sort of support structure than a normal engineering team.

A data science team seems to end up having people from a much more diverse set of backgrounds than a normal engineering team does, and because of that you're going to need to figure out how to fit the data science team into your organization. One thing that we've eventually come around to is to throw our normal engineering practices out of the window and to start working towards having the data science team work with whatever tools they need in order to move as quickly as possible in their own way.

You're going to need to figure out how to get stuff into production.

Data scientists are going to end up building things that need to be translated into production code, usually to save resources ("in order to generate this model, I need 20 machines for a week, and it needs to be done every day"). One thing that we're working on is building out the interface between engineering team and our data team in order to make it more and more seamless. Hopefully soon we'll be constantly rotating an engineer (or several engineers) on the data team in order to help do this.

Having data scientists around will make you better at thinking about your problem set

Data scientists tend to come from a number of different backgrounds (math, statistics, music, etc) that you don't normally have on an engineering team. Supporting them and interacting with them on a daily basis forces you to think about things differently and should help you build things in a more sensible way to target a larger audience within the company.

Everyone in the company is going to want to understand what (and how) the data science team is doing

Just like we've worked over the years to keep technical business people focused on the "what" of what we need to build, you need to work on keeping every one without a math background away from the data science team. Trying to explain some of the hard math that's going on to the entire company isn't a productive use of time.

The toolset that exists for jumpstarting a data science team is still immature

The tools for doing big data analysis are still pretty immature and difficult to set up. There are many companies working to solve this problem right now (although most of them focusing on big companies, not startups) but it's a bit overwhelming figuring out where to even get started with some of this stuff.

Comments

Tackling Knowledge Debt

I was playing around with Cascalog today, working on rewriting some of our mapreduce jobs, when I came across this post on knowledge debt by Nathan Marz.

Knowledge debt is something that I think about a lot (although I never had a good name for it until today). When you're working in a startup like the one that I'm currently involved in, there are often times when you need to get a release out of the door by a certain date. Missing a release date with a deal with Expedia on the line, for instance, isn't really an option.

At the same time, it's hard to draw a clean line as to when some of the things that we're doing aren't really working anymore, or are actively slowing us down. It always seems like we're making progress, but who really knows if we could be doing some of our work in a much more efficient way.

Of course, money helps with everything, and we have more and more of that lately, which I'm planning to put to use to help solve this problem.

My current plan is to hire 12-15 more engineers over the next year in order to give us more breathing room to do new things. We need time to try out things like Cascalog (and Scala, Clojure, Cassandra, etc) and to spike out new approaches in our codebase, while at the same time continuing to make steady progress on the work that needs to get done.

Hopefully some of these experiments will pan out and give us a big boost in velocity, but even if they don't we should still be doing them.

I'm also hoping that each new engineer on the team shakes things up a bit. Every person that walks in the door should be teaching us something new, or else they shouldn't be walking through the door in the first place. I already know that we have a pretty amazing codebase but, looking at some of the problems that we're about to tackle, I can't wait to see what 2012 is going to bring.

Comments

Moving to Jekyll

I'm always meaning to do more blogging, but I always seem to have a million reasons that I never do. One of the things that I've always disliked was dealing with all of the overhead. I don't want to log in and type things in a Javascript window, etc. I just want to blog.

For that reason I've moved my blog to Jekyll. Jekyll is a highly customizable static site generator that's used to generate most of the pages on Github. It lets me write all of my blog posts in a text editor, so that (hypothetically) I can no blog anywhere and everywhere. The philosophy behind it can be found in the original blog post on it, Blogging Like a Hacker.

So there's one more excuse for not blogging knocked off of my list. Hopefully I'll do a better job of it.

(Oh, also, porting my blog over is a work in progress. If you see any bugs please let me know in the comments.)

Comments

Large-Scale Cucumber Revisited

At Intent Media we've been using Cucumber for 2.5 years to drive a large portion of our testing (on top of RSpec and JUnit). We're finally getting around to upgrading to the newest version (from version 0.4.4, which is, to put it lightly, a bit rusty), so I'm going to throw out some anecdotes about our experiences using it, which will hopefully help others avoid some of the problems we've had.

First a few facts about our test suite:

  • We currently have over 15000 Cucumber steps that run for each build (we use TeamCity for CI, which counts the number of steps, not the number of scenarios, which is kind of annoying, to tell you the truth)
  • We distribute our build over 3 machines using tags instead of a custom tool. Every scenario that goes in gets tagged with @agent1, @agent2, or @agent3 to indicate which build machine to run on. Hacky? Yes. Implemented in a couple of days with no problems? Also yes.
  • It currently takes over an hour to run a complete integration build. This is too long, and we'll probably distribute the build out to more agents soon.

Some lessons learned:

  • One of our original goals was to get our Product Managers involved in writing and vetting test cases. That didn't happen with us, and I now think that it would be very hard to get it to happen anywhere. Even though we have very well written stories with an annotated list of deliverables, the mental hurdle of going from the the story, to finding the scenarios related to it, to verifying them all just seems too high. Our PMs just want to grab the newest builds, go through whatever mental model they have to test the feature, and then move onto the next thing. Perhaps if we had made it part of the process from the very beginning to somehow tie the scenarios back to the stories, then this might have worked, but I think we missed our opportunity here.
  • Following on from above, you should figure out a way to tie your scenarios back to the their related stories. One of the first things that our new QA person did was to push the team to always add a comment to completed stories with the names of the related scenarios. This saves her the time that it would take to figure out what's relavent to testing each story. If you're not doing this in some way, you should be.
  • We made a decision early on to test everything from the outside in (no fixtured data, etc). At first this worked really well for us, because it gave us a lot of confidence that all of our screens were extremely well tested. Unfortunately, it now adds a ton of time to our test runs by running through the same set of screens hundreds of times. We've started discussions about how to fixture in this data after some of the steps have run to completion, but we still need to get around to making it happen. (As an example, a step like 'And I create a new admin user with email address "example@example.com"' could hypothetically run for every scenario in the system.) Doing it all through the UI is just a time sink at this point. What we'd like to do is run the step through the UI once, and then set a flag and fixture in the data after that.
  • If you are making large scale use of Cucumber and you aren't using IntelliJ/Rubymine then you should be. Its Cucumber support is top notch, and will make your life significantly easier.
  • All of that said, Cucumber has been vital in allowing us to release on a daily basis. Unlike some other startups, if we screw up a release we start losing a large amount of money pretty much immediately. We have an automated deploy process, but the cucumber suite has given us the confidence to know that anytime we push our newest green build everything is going to work.

So there you go, a few things off of the top of my head. If you have any other good anecdotes about Cucumber please add them in the comments, I'd be interested in seeing what other problems people have run into.

For more random development anecdotes, you should follow me on Twitter here.

Comments

Driving Large Scale, Multi-Language Development With Cucumber

I've been working at Intent Media for a while now and I'm just getting some time to come up for air to talk about what we've been doing while building our software platform.

This will be the first in a series of articles about what I've learned over the last year.

For a little background, our engineering team is currently 9 people strong (in a 20 person company). When I joined the company to help build the team, there wasn't a line of code written and I was the only technical member of the staff.

Over the last 9 months we've built what I consider one of the finest pieces of software that I've ever worked on. It's scalable, it uses the right languages for the right jobs, and it does exactly what it needs to do.

It's also remarkably bug free.

I credit a large part of that (on the tools and good decisions side, that is; I credit most of it to our team) to Cucumber.

One of the first decisions we made (after writing a unit test and our first bit of code) was to write our first Cucumber test. One of the things that I learned while working at Collaborative Drug Discovery was the importance of automated functional tests.

I, for one, will never do another project without them.

At Intent Media we use Cucumber to drive Celerity across our entire software stack. Currently, this consists of 3 Java applications and a Rails application that interact with MySQL and a HDFS data store. Every single time we check in new code we build the entire stack in TeamCity and run our entire suite of functional tests against it.

As of today, I'm currently greeted with this on our TeamCity server:

Integration Build

Each of those "Tests passed" is a step in a Cucumber scenario. For example:

Given I login as "admin@intentmedia.com"

Over the last 9 months we've slowly built up a library of steps that encapsulate everything that we need to do with our apps so far. We've also found that we've gotten a huge amount of reuse out of our existing steps as we've grown the app. So far, there haven't been any scenarios that we couldn't tackle in an automated manner.

This suite of tests gives us the best safety net I've ever had on a project to try large refactorings. In fact, we often do refactorings across the data model that affect several pieces of software in the stack, often in unexpected ways. Cucumber (along with our unit tests) lets us immediately surface any problems in both our applications themselves and the communication between the apps.

If you haven't checked out Cucumber yet, you owe it to yourself to do so. It's one of those life-changing pieces of software that's amazingly simple to get started with, but quickly becomes one of the most important tools in your toolbox.

Comments

Falling Off the Face of the Earth

Quick update on the last 6 months:

  • Moved back across the country to NYC
  • Took a job as Head of Engineering at Intent Media (We're Hiring! We're funded!)
  • Been working in Java and Rails

I've got plenty of things to write about, so I'll try to get back on the horse soon.

Comments

Living In A Post Rails World

The hardest thing to see is what is in front of your eyes. -Goethe

It's been about 6 or 7 months since I've done any Rails work.

I'm in a post-Rails world.

After working in Rails for 3 years, I was burnt out. I took some time off to try to learn the finer points of marketing and running a business, and then attempted to start a new business just before the market cratered. Not a great idea.

Now I'm back writing code all day and it's great. My days now consist of coming up with test cases to solve problems and then solving them.

Something is different though.

I used to write a ton of test cases in the context of my Rails apps.

Now I write high level test cases and then write code and pick and choose technologies to solve them.

I feel like this is a natural progression for those of us that practice TDD.

Take one more step back, use something at an even higher level to write tests (I'm using some Cucumber and some rSpec), and then implement a bunch of little apps to make what you want to happen, happen.

It keeps things simple, and keeps the problem from growing too large.

In my current project I've got a cluster of Nanite agents up and running with a couple of Merb apps talking to them on the front end. It's something that I never could have done if I was trapped inside of the Rails box.

Hell, it's probably something that I never even would have thought of.

I think that the Ruby world is eventually going to end up in a model like this, writing small simple apps that all talk to each other, and can be replaced or upgraded at any time.

By writing a lot of tiny apps, I've been able to solve problems now in days that used to take weeks or months.

Plus I'm not getting locked in to anything. If I want to replace my Merb app with a Sinatra app next month, it's not a huge project. If I want to swap in some code running on the Maglev Alpha that I'm playing with or deploy some components written in Clojure, I can do that too, without being intimately tied to anything.

All of my hard/long running logic is well tested, encapsulated, and most likely running in little agents on the wire.

I guess what I'm saying is that if you've been writing Rails code for a long time, perhaps now is a good time to step outside of the rules that Rails imposes upon you and look at things from a different angle.

You might find it refreshing and helpful.

I certainly did.

Comments

So I Wake Up and Merb is Rails?

If you haven't already heard, Rails is now Merb, and Merb is now Rails.

I'm a bit torn about how to feel about this, having dedicated a lot of time lately telling people to use Merb and why I think that it's superior to Rails.

On the one hand, it'll be good for Merb usage, as people using Rails 3 in big projects will now essentially be using what Merb 2 would have been.

On the other hand, I'm concerned with the technical hurdles that the core teams are going to have to deal with in order to make this actually work. There's a lot of old crufty code in Rails that needs to go, although I'm not sure what removing it would do as far as backwards compatibility is concerned.

I've been talking in IRC all morning about this, and I'm sure I'll have more thoughts as things move forward, but for now I'll just post some of my favorite quotes from today's IRC session here:

  <devjj> Modular meaning, what, not included by default, gem install rails-bad-ideas
  <gruntruk> this is like biggy n' pac hugging each other
  <bitbckt> so, I presume the quality of conversation in #merb will disintegrate into #rubyonrails :-/

Certainly should be an exciting couple of months as this merge happens...

(Credit goes to an IM from my friend Will for the title of this post.)

Comments

The Open Source Merb Book

Just a quick note to let people know that the Merb community is finally starting to address the lack of good documentation on Merb. I'm personally helping out with the open source Merb Book.

A preview version can be found here: Open Source Merb Book

The source can be found here: GitHub

We've got 20 plus people working on it, and considering that it's only been up on GitHub for 12 days it's pretty amazing what we've already accomplished.

If you want to learn more about Merb you should definitely check it out.

Comments

These are the ten most recent posts, for older posts see the Archive.