Can Agile work for a team of experts?

My last article about considering Agile produced a lot of interesting feedback. Most of all, it prompted me to do a lot more research into Agile software development and specifically Scrum as a project management framework.

As has been mentioned before, many of these Agile ideas are just common sense. This is often pointed out by its critics, but is that really a bad thing? Since common sense is not as common as we would like it to be, Agile frameworks and methodologies codify their practices in rather specific rules on how to do things. Basically, these rules are designed to help (force?) a team to adopt a lot of common sense practices, which in traditional software engineering may have been forgotten. I have to say, I feel like I actually 'get' many of them and I might talk about some specifics here in future blog entries.

In the meantime, though, I'm still trying to wrap my head around one particular issue: Agile seems to work best - in fact it requires - that there are no 'silos' in the engineering organization: Particular knowledge should not be locked up in the head of one developer or a small group of them. Instead, knowledge should be shared across the team and some Agile practices, such as pair programming, for example, are specifically designed to address this. But in same cases those silos are unavoidable. Read on for an example of that. And in the moment you have knowledge silos in your team, many of the benefits of Agile seem to just evaporate.

Let me explain what I mean. For truly Agile development it must be possible to assign user stories and their related tasks to almost any team member. At least, there should be more than one team member who's able to take on a specific task.

If you have a team like this it allows a number of interesting things to happen:

1. Once a developer is done with a particular task they can merely grab the next highest priority item from the backlog. This makes it possible for the team to really deliver higher-priority items in order, thus producing most value to the product owner in each sprint.
2. You can use story point estimating, maybe utilizing surprising techniques such as planning poker, and it will actually make sense: Over the course of several sprints you get a feeling for the velocity of the team and even each individual developer. The nice thing about estimating in story points is that it is self-tuning and doesn't depend on the experience level of the individual developer. After a while you will know that John - a senior developer - can do about 20 story points per sprint, while Bob - fresh from university - may be able to do only 10. So, you can re-assign user stories without having to re-estimate them - well, you need to re-esitmate the associated tasks, but that is easier with these metrics - and still quickly get a good idea of how your team will be able to do in the sprint.
3. Every developer can just jump in and help others in the team. That is of course a given and is helpful in every kind of team, Agile or not. If a staff member goes on vacation or is sick the work can be picked up by whoever is available (see also point 1). Likewise, if a developer is held up with a previous task for longer than expected then the next task that he was supposed to tackle doesn't have to wait. If someone else is free or doesn't have a higher priority item to work on, that is.

Imagine you are in a team that develops web-applications, using Django or RoR or some such framework. There is a large number of user stories, all of them describing the various features that need to be implemented. All of your developers will have to work in pretty much the same environment and under the same constraints provided by the framework. In a situation like this, I can imagine Agile to work very well, since all developers essentially need to have the same abilities for the job anyway: Know the language, know the framework, know the infrastructure you use for developing and testing. That covers a lot of ground and establishes large areas of commonalities between developers. Everyone is much more likely to be able to take on any specific task during the sprint.

But now imagine a different team in a different company and suddenly we encounter trouble: Assume you are in a small startup, which doesn't just devlop a web-app, or work with a single framework or environment. Assume you are working on a project that hacks the Linux kernel for some innovative network monitoring solution, produces a lot of data that needs to be sliced and diced and requires a web-app to drive an interactive front end. You are a small startup with some angel investment, so you get yourself the smallest team you need to get the job done: An experienced networking/kernel hacker, a database expert, someone who knows Django or TurboGears (or a similar web-app framework), and a user-interface person taking care of the HTML/JS/CSS for the front-end.

So, that's your team. Now imagine a planning poker session for a specific feature: "Ok, on the count of three: What's your estimate for hacking the ipchains kernel module to add feature XYZ?" Guess who is going to put down a card with a number and who is going to put down a card with a question mark (meaning: "Don't know...")? Well, the kernel person may have a decent estimate, while everyone else - those who never hacked around in the kernel before - realistically will have no idea how complex that task is! You have just one person on the team who can do the job and can provide estimates. You may just as well estimate in hours then, and you will be back at much more traditional planning. You need to define exactly ahead of time who will do which tasks, since each person is so specialized that in effect they are the only ones being able to estimate and do it. Say hello to your Gantt charts.

I think this is an example where some of the best aspects of Agile seem to fall flat.

Maybe you will say: Well, you shouldn't have teams like this anyway, you should pair-program to spread the knowledge, you should assemble your team so that you don't have this formation of knowledge silos.

To those arguments I will have to say: VCs don't have a lot of patience and they don't hand you a lot of money to start with. You can only get a small team, and you can't turn an HTML/CSS/JS person into a Linux kernel expert just like that. Especially not when you deal with a very tight deadline and budget. You can try, but then who's doing the front end work, while you are sending that person through the Linux kernel 'apprenticeship'? No, you need subject matter experts that are as efficient as they can be and can crank out a piece of functionality with the least amount of fuss and delay. And all you can afford at that stage is one expert for each subject area.

If you have a team of experts, each expertly knowledgeabe in their own area with very little overlap: Does Agile still make sense?

As I said at the beginning, I can see a lot of value in many of the Agile practices and ideas. Yes, it's basically common sense, but I understand how by codifying it you can bring a lot of this common sense back into engineering organizations. But while I have learned to appreciate a lot about Agile since my [previous post], I have to stand by the assertion I made then: Agile works in some teams and environments, but there are also some cases where it simply makes much less sense. Some of its aspects might still be useful, but under some circumstances it loses a lot of value to the point where you have to ask yourself: Why bother with Agile at all then?

Agile seems to make sense only once your team can grow in size - or alternatively in the spread of knowledge across the team - to a point where most tasks could be effectively tackled by more than just one person on your team. If your work team consists of generalists, and your user stories and tasks are suitable to be tackled by those generalists. Note, with 'generalist' here I mean someone who is able to deal with all the aspects of your project. I'm not advocating that developers shouldn't have speciality areas.

This is just my impression. If you have experience working in Agile teams that just consist of a small group of specific and distinct subject area experts, please comment and let me know how you handled this problem. I would love to know.

In the meantime, I can imagine that non-homogeneous teams, with too many knowledge silos, may be the reason for many failed Agile projects: The team thinks it's agile, but when push comes to shove, they realize that actually they have been hand-cuffed by the insufficiently spread knowledge in the team. After a few sprints the situation may improve, but for starters?

Agile is about common sense, and there seems to be a very important common sense item that needs to be considered: Does Agile even make sense for my team and project? The answer is not always yes, no matter what Agile-enthusiasts may say. In reality (clue common sense) the answer shouldn't come as a surprise: It depends!

You should follow me on twitter here.

 

 

An entire web site in a single page

Update: I received nice feedback from some readers, explaining
why this one-page design is not very good for SEO. Thank you
very much! I appreciate your feedback. As a consequence of this,
I have now moved my site to a traditional multi-page design.
If you are still interested in seeing the original single-page
site, you can go here.

When I first put up my web site, it was all just a single, static page. Basically a small blurb about me, the services I offer, and so on. It looked kind of neat, with a very simple design, which I like. And with a small enough font and big enough screen, it really fit all into your browser without scrolling.

But then a friend told me that it doesn't really look professional enough with just a single page. People expect to see at least several pages on a 'real' web-site. So, something had to be done.

Now, I'm not a professional web-designer who can photoshop together wonderful logos and backgrounds and also work a site based on the best user/computer interaction strategies. I mostly just wanted to retain the very simple and straight forward design of my site. And it should load fast. And it should be friendly to search engines. And it should still look good in text based browsers, or browsers that don't have JavaScript enabled.

When you look at my site now, you see that it does use multiple pages. But then... not really. Your browser will actually just load a single HTML page, which contains the entire text of the site. The navigation links merely trigger some changes in the CSS for various text sections, so that different content is displayed when those links are clicked.

Using the 'display' style property for hiding elements on a page is nothing new and is used on many sites. It's just that normally you see it used to show or hide specific elements within a page, while site navigation itself is normally left to the real thing: Actual separate pages.

So, there are probably other sites which do it this way, but it was the first time for me that I tried this approach with a site, and I like how it turned out. That's why I'm sharing this.

If you look at the site without JavaScript enabled, you see all the content in the page, one 'page' after the other. The navigation links on the left still work. I'm thinking I might add a link back to the top enclosed in <noscript> under each section at some point.

One of the most visible benefits is that site navigation is so fast, it feels instantaneous. Nothing needs to be loaded when a navigation link is clicked. But that is not the only advantage...

If you look at the page in a text-based browser such as Lynx, you see that all the relevant information is there, easily readable on a single page.

Best of all, the 'crap-to-content' ratio for the page is very good. I you look at the HTML source, you see that there is a lot of relevant textual content, and very little markup to distract from it. I hope Google gives me brownie points for that one...

In fact, I would be very interested to hear what those of you with more SEO experience have to say about the site. Does this approach actually help with search engines, or would it make it worse?

Ok, so there is one issue I can see with this approach: Tracking of visitors is made a bit more complex. Maybe I can use a small bit of JavaScript to re-load the image of the tracker that I use when a user clicks one of the navigation links? Again, if you have some feedback there, I would really appreciate it.


You should follow me on twitter here.

 

 

Considering Agile, deciding against it

Some time ago, while managing a distributed software project, I spoke to a developer who was thinking about joining our team. This individual was a big proponent of Agile development. When he found out that in our particular project we were not using an Agile development model, he began to lose interest. He compared my position there to the position of the Tom Smykowski character from the movie Office Space. When I recently then came across this blog posting by Shannon JJ Behrens, I had to remember that particular incident...

In the movie Office Space, Tom is an older employee at Initech, whose job it is to “take requirement documents from the customers and bring them to the developers”. Not surprisingly, he is being characterized as 'useless' and let go. Similarly, in Agile development, direct communication between customers and developers is seen as the way to ensure that the deliverables align with customer requirements. No middle man is needed. Instead of extensive up-front design, requirements are captured in short and concise user stories.

I have to admit, I was somewhat taken aback by the comparison to the Smykowski caricature from the movie, but this episode certainly made me think about why we had chosen a non-Agile style for our project.

Let's try to answer this. Take a look at the graphic here (I don't remember where I saw it first, but it can be found on many different sites, click to see the larger version):

What do we not see anywhere? Design and architecture. The definition of the 'backlog' and the 'release plan' is probably the closest we get, but even that is feature focused. And in Agile, features always seem to be driven by customers.

For me it boils down to this: I believe that Agile is applicable in projects where not only a clear vision of the project has been communicated to the developers, but also where certain conventions and structures have already been established and where we are dealing mostly with explicit requirements, rather than implicit ones. Projects, where the 'how' is already known, and we are just dealing with the 'what'.

For example, to achieve a consistent UI you require some up-front planning, possibly a UI czar: One person - or at most a small number of people - who establish and downright dictate what the UI shall look and feel like. It may stifle the creativity of the developer, but it will ensure a much more consistent look for the UI. Imagine you have user stories describing the requirements for various dialogs in the application's interface. Different developers take on the implementation of those user stories. Unless there is some established, over-arching strategy and - dare I say it? - upfront design of the UI then the results are likely to appear disjointed or piecemeal.

Another example is error handling in the application. I remember one of my software engineering instructors a long time ago, who insisted that a project should have an error czar: Someone who establishes the standard by which errors should be handled within the code and who then diligently ensures that all new code complies with this standard. If this is not done and established ahead of time, how are the different developers supposed to produce consistent code that knows how to adhere to those standards and handles errors in a unified manner?

It appears to me that if you are developing an application for an existing framework then many of these up-front 'infrastructure' decisions have been made for you already. Consequently, you can focus more readily on explicit customer requirements and Agile is a much more worthwhile development strategy.

For example, if you develop a web-app in a framework such as Django or Rails then there are specific ways in which your database is going to be laid out. You also have specific conventions for how errors are handled, and so on and so forth. Thus, the design and architecture of much of your application is pre-determined by your choice of framework. You don't have to make those decisions any more. And later then, the more of the application has been written, the more such underlying decisions will have been made already. Agile works well in those situations. Since big-picture architectural decisions don't have to be made any more, we can focus on the addition of small features (or larger features broken down into many smaller ones), while all developers benefit from the established infrastructure of the project. The 'grab-bag' approach with focus on the individual user story becomes feasible.

However, if a vision for the project still has to be conjured up and the basic architecture has to be created and the UI look and feel needs to be designed and error handling or other standards have to be established... well, then Agile does not seem to be ideally suited for that. I have seen design and architecture by committee and design by chaos (everyone going off to implement things before anything was established). Neither one is a pretty sight. The best results are achieved if the architectural direction and basic design can be established by one or just a few, good individuals. Up-front. Well, somewhere after establishing the vision and collecting initial requirements, of course.

Realistically, I'm sure that most Agile projects will include some sort of up-front design and architecture, whether the participants call it that or not. Heck, even questions like: “Should we use Java or Python for this?” need to be answered ahead of time and are part of these implied up-front activities.

That particular project for which the developer from the beginning of this article interviewed was not at the stage yet where an Agile approach really would have worked. There were still too many unknowns at that time, which required up-front consideration. Thus, our project wasn't run as an Agile project. There is a time and place for everything, and our project was not at a time or place for Agile, yet.

However, projects can be transitioned from a more conventional approach to an Agile approach over time. The more we know, the more suitable Agile becomes. The less architecture and design decisions have to be made the better for Agile. But at the start of a brand new project, without anything established, some quite conventional up-front design and architecture is needed.


You should follow me on twitter here.

 

 

Google's Chrome OS is bad news for Linux

Google's Chrome OS has been announced today and - like any announcement from Google - is already widely being discussed. People usually focus on the supposed threat that this 'new' OS represents to Microsoft's dominance on the desktop. The commentary focuses on this Google vs. Microsoft aspect of the story. However, I think the news is actually worse for Linux than it is for Microsoft.

How can I say this, considering that Chrome is be based on a Linux kernel? Let me explain, with a 'historical' analogy.

In the 80s and 90s we had the so-called Unix Wars: Prominent Unix vendors like Sun, Digital, IBM, AT&T and many others were battling for supremacy in the server market. Guess who emerged victorious from those Unix Wars?

Microsoft's Windows NT.

Customers were sufficiently fed-up with the fragmentation and confusion in the Unix server market and finally were receptive to a message of simplicity: Your desktops already run Microsoft software, so why not your servers? It will all play nicely together ... just deal with a single vendor ... yada yada.

Now fast forward 15 to 20 years. While Unix is holding on to various diminishing pockets in the server market and Windows server-products have a stronghold in many enterprises, Linux has emerged as a very viable and popular choice for the server. On the desktop, however, Microsoft rules supreme now as it did then.

Back in the Unix Wars, Microsoft used the fragmentation amongst the competitors to become a dominant force in the market. Today, it enjoys the advantage that the fragmentation of potential opposition in the desktop market is handing to it. Nowhere is the fragmentation of Linux more apparent than on the desktop.

Ubuntu managed to become a strong player in the Linux desktop market, and for once it appeared as if things were finally coalescing. Ubuntu's popularity started to feed itself: Any issues or problems? Chances are someone solved it for Ubuntu already, a quick search on the Internet reveals answers to most common issues. Therefore, while Ubuntu might not be perfect for everyone or might have a few annoyances, for the most part you just don't go wrong settling on Ubuntu: So many others are running it, momentum has been forming.

Now Google comes along with Chrome OS. We now have another major player entering the Linux desktop OS market. Linux needs this like we need another hole in the head. Microsoft will read this news with glee: Through the opponent's fragmentation it managed to win the Unix wars. Now Linux's continued fragmentation will ensure that there isn't even going to be a war for the desktop.

So, I wish for Google Chome OS to fail. We don't need it. It doesn't help Linux. As suggested here, if Google wants an improved non-Microsoft desktop OS then please throw your considerable resources behind improvements in the existing Linux desktop distros. We will all be better off for it.


You should follow me on twitter here.

 

 

Setting the initial value for Django's ModelChoiceField

Recently, I worked with a Django form that utilized the ModelChoiceField. That is a convenient field that normally is rendered as an HTML select tag, which usually appears as a drop-down menu on a web-page.

A ModelChoiceField is specified as:

    class MyForm(forms.Form):
        my_field = forms.ModelChoiceField(queryset = MyModel.objects.all())

As you can see, this field is used to easily specify a drop-down for all items in a table (or whatever the queryset specifies). The model is represented as the output of its __unicode__() function. If you evaluate the form after it was posted, the value for the field is going to be an instance for the actual model whose unicode representation was selected.

The problems for me started when I tried to set an initial value for the field: In an 'edit' form, I wanted all fields to reflect whatever had been saved for a particular model instance, of course. As I said above, if you evaluate the form after it has been submitted you get an actual model instance. Naturally, you would think that initial values would be set in a symmetric manner, by specifying a model instance:

    form = MyForm(initial = { 'my_field' : some_model_instance })

Sadly, this doesn't work. And try as I might, I couldn't find an answer to this on the Internet either (more on that in a moment). So, after looking at the Django code, it finally dawned on me that you need to specify the ID of the model instance as the initial value:

    form = MyForm(initial = { 'my_field' : some_model_instance.id })

That works now. It's a bit unfortunate that the retrieval value (a model instance) and the initial value (the ID of a model instance) are of a different type. It's an inconsistency in the Django API, I think. But in the end, I probably should have at least tried that one a bit sooner.

The surprising thing is that I couldn't find any discussion of this anywhere on the Internet. I should mention here that I am using Yahoo as my default search engine. Shortly after I finally found the solution, it occured to me to try Google. And wouldn't you know it? Right there, third hit from the top, I had the answer.

Why did Yahoo not give me this result? Well, the answer was discussed on Google Groups. Is Yahoo not indexing those? Or is Google not letting them index it?

Either way, that small issue between Yahoo and Google cost me a few hours of frustration. So, I'm posting the solution here on a non-Google page, so that Yahoo users may also find the answer to that problem in the future.


You should follow me on twitter here.