Can Agile work for a team of experts?

My last article about considering Agile produced a lot of interesting feedback. Most of all, it prompted me to do a lot more research into Agile software development and specifically Scrum as a project management framework.

As has been mentioned before, many of these Agile ideas are just common sense. This is often pointed out by its critics, but is that really a bad thing? Since common sense is not as common as we would like it to be, Agile frameworks and methodologies codify their practices in rather specific rules on how to do things. Basically, these rules are designed to help (force?) a team to adopt a lot of common sense practices, which in traditional software engineering may have been forgotten. I have to say, I feel like I actually 'get' many of them and I might talk about some specifics here in future blog entries.

In the meantime, though, I'm still trying to wrap my head around one particular issue: Agile seems to work best - in fact it requires - that there are no 'silos' in the engineering organization: Particular knowledge should not be locked up in the head of one developer or a small group of them. Instead, knowledge should be shared across the team and some Agile practices, such as pair programming, for example, are specifically designed to address this. But in same cases those silos are unavoidable. Read on for an example of that. And in the moment you have knowledge silos in your team, many of the benefits of Agile seem to just evaporate.

Let me explain what I mean. For truly Agile development it must be possible to assign user stories and their related tasks to almost any team member. At least, there should be more than one team member who's able to take on a specific task.

If you have a team like this it allows a number of interesting things to happen:

1. Once a developer is done with a particular task they can merely grab the next highest priority item from the backlog. This makes it possible for the team to really deliver higher-priority items in order, thus producing most value to the product owner in each sprint.
2. You can use story point estimating, maybe utilizing surprising techniques such as planning poker, and it will actually make sense: Over the course of several sprints you get a feeling for the velocity of the team and even each individual developer. The nice thing about estimating in story points is that it is self-tuning and doesn't depend on the experience level of the individual developer. After a while you will know that John - a senior developer - can do about 20 story points per sprint, while Bob - fresh from university - may be able to do only 10. So, you can re-assign user stories without having to re-estimate them - well, you need to re-esitmate the associated tasks, but that is easier with these metrics - and still quickly get a good idea of how your team will be able to do in the sprint.
3. Every developer can just jump in and help others in the team. That is of course a given and is helpful in every kind of team, Agile or not. If a staff member goes on vacation or is sick the work can be picked up by whoever is available (see also point 1). Likewise, if a developer is held up with a previous task for longer than expected then the next task that he was supposed to tackle doesn't have to wait. If someone else is free or doesn't have a higher priority item to work on, that is.

Imagine you are in a team that develops web-applications, using Django or RoR or some such framework. There is a large number of user stories, all of them describing the various features that need to be implemented. All of your developers will have to work in pretty much the same environment and under the same constraints provided by the framework. In a situation like this, I can imagine Agile to work very well, since all developers essentially need to have the same abilities for the job anyway: Know the language, know the framework, know the infrastructure you use for developing and testing. That covers a lot of ground and establishes large areas of commonalities between developers. Everyone is much more likely to be able to take on any specific task during the sprint.

But now imagine a different team in a different company and suddenly we encounter trouble: Assume you are in a small startup, which doesn't just devlop a web-app, or work with a single framework or environment. Assume you are working on a project that hacks the Linux kernel for some innovative network monitoring solution, produces a lot of data that needs to be sliced and diced and requires a web-app to drive an interactive front end. You are a small startup with some angel investment, so you get yourself the smallest team you need to get the job done: An experienced networking/kernel hacker, a database expert, someone who knows Django or TurboGears (or a similar web-app framework), and a user-interface person taking care of the HTML/JS/CSS for the front-end.

So, that's your team. Now imagine a planning poker session for a specific feature: "Ok, on the count of three: What's your estimate for hacking the ipchains kernel module to add feature XYZ?" Guess who is going to put down a card with a number and who is going to put down a card with a question mark (meaning: "Don't know...")? Well, the kernel person may have a decent estimate, while everyone else - those who never hacked around in the kernel before - realistically will have no idea how complex that task is! You have just one person on the team who can do the job and can provide estimates. You may just as well estimate in hours then, and you will be back at much more traditional planning. You need to define exactly ahead of time who will do which tasks, since each person is so specialized that in effect they are the only ones being able to estimate and do it. Say hello to your Gantt charts.

I think this is an example where some of the best aspects of Agile seem to fall flat.

Maybe you will say: Well, you shouldn't have teams like this anyway, you should pair-program to spread the knowledge, you should assemble your team so that you don't have this formation of knowledge silos.

To those arguments I will have to say: VCs don't have a lot of patience and they don't hand you a lot of money to start with. You can only get a small team, and you can't turn an HTML/CSS/JS person into a Linux kernel expert just like that. Especially not when you deal with a very tight deadline and budget. You can try, but then who's doing the front end work, while you are sending that person through the Linux kernel 'apprenticeship'? No, you need subject matter experts that are as efficient as they can be and can crank out a piece of functionality with the least amount of fuss and delay. And all you can afford at that stage is one expert for each subject area.

If you have a team of experts, each expertly knowledgeabe in their own area with very little overlap: Does Agile still make sense?

As I said at the beginning, I can see a lot of value in many of the Agile practices and ideas. Yes, it's basically common sense, but I understand how by codifying it you can bring a lot of this common sense back into engineering organizations. But while I have learned to appreciate a lot about Agile since my [previous post], I have to stand by the assertion I made then: Agile works in some teams and environments, but there are also some cases where it simply makes much less sense. Some of its aspects might still be useful, but under some circumstances it loses a lot of value to the point where you have to ask yourself: Why bother with Agile at all then?

Agile seems to make sense only once your team can grow in size - or alternatively in the spread of knowledge across the team - to a point where most tasks could be effectively tackled by more than just one person on your team. If your work team consists of generalists, and your user stories and tasks are suitable to be tackled by those generalists. Note, with 'generalist' here I mean someone who is able to deal with all the aspects of your project. I'm not advocating that developers shouldn't have speciality areas.

This is just my impression. If you have experience working in Agile teams that just consist of a small group of specific and distinct subject area experts, please comment and let me know how you handled this problem. I would love to know.

In the meantime, I can imagine that non-homogeneous teams, with too many knowledge silos, may be the reason for many failed Agile projects: The team thinks it's agile, but when push comes to shove, they realize that actually they have been hand-cuffed by the insufficiently spread knowledge in the team. After a few sprints the situation may improve, but for starters?

Agile is about common sense, and there seems to be a very important common sense item that needs to be considered: Does Agile even make sense for my team and project? The answer is not always yes, no matter what Agile-enthusiasts may say. In reality (clue common sense) the answer shouldn't come as a surprise: It depends!

You should follow me on twitter here.