tag:blogger.com,1999:blog-67488062712903171002024-02-21T09:09:12.134+13:00Brendel ConsultingA blog from Brendel Consulting: Custom software architecture, design, implementation and managementjbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.comBlogger27125tag:blogger.com,1999:blog-6748806271290317100.post-81102479924265602022012-04-18T13:26:00.000+12:002012-04-18T13:26:24.872+12:00Random primary keys for Django modelsThe automatically generated primary keys for Django models tend to be just sequentially increasing integer numbers. If you expose those IDs at any point to your users, for example in the URLs referring to particular IDs, then your users can oftentimes easily guess how many users your site has, or how many objects of a certain kind your database holds.<br />
<br />
You could follow the approach of never exposing those primary keys to the user, and always use an additional, randomly generated key instead whenever a user needs to refer to an object in your database. However, you then need to implement this random key creation, need to maintain two fields, etc. It would be nice if we didn't have to worry about this and could just use the primary keys of our models to refer to them, internally as well as externally.<br />
<br />
For this purpose, I have created a new base class, which you can use instead of <span style="font-family: 'Courier New', Courier, monospace;">models.Model</span><span style="font-family: inherit;"> when you create your own Django models. This base class is called </span><span style="font-family: 'Courier New', Courier, monospace;">RandomPrimaryIdModel</span><span style="font-family: inherit;"> . With this base class, your primary IDs will start to look random, similar to what you know from URL shorteners.</span><br />
<span style="font-family: inherit;"><br /></span><br />
<span style="font-family: inherit;">Here's an example. Let's say you have defined a Django model like this (in the example, the only change you have to make to your normal model definition is to replace the </span><span style="font-family: 'Courier New', Courier, monospace;">models.Model</span><span style="font-family: inherit;"> base class with </span><span style="font-family: 'Courier New', Courier, monospace;">RandomPrimaryIdModel</span><span style="font-family: inherit;">):</span><br />
<span style="font-family: inherit;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> from random_primary import RandomPrimaryIdModel</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> </span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> class MyModel(RandomPrimaryIdModel):</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> # Normally define your Django model</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> ...</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> for i in xrange(3):</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> m = MyModel(... parameters ...).save()</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> print m.id</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: inherit;">As output you might get something like this:</span><br />
<span style="font-family: inherit;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> Q68mfU</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">zjvsx3</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"> VNuL0Lp</span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
You can tune the key length as well as the characters that are used to construct the key. The docstring of the class is pretty extensive, so please have a look.<br />
<br />
The code for the new model base class is free to use for anyone and can be found in this <a href="https://github.com/jbrendel/django-randomprimary" target="_blank">Github repository</a>.<br />
<br />
Hopefully, this can be useful to you. I'd welcome any feedback or comment.<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span><br />
<span style="font-family: 'Courier New', Courier, monospace;"><br /></span>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-75113763547145855122012-01-24T08:04:00.004+13:002012-01-24T08:30:49.841+13:00Django ModelForms: Setting any field attributes via Meta classDjango's ModelForms are a convenient way to create forms from already existing models. However, if any customization on the form definition is desired then we often are forced to manually re-define the field in the ModelForm definition. For example, consider <a href="http://stackoverflow.com/questions/3436712/custom-error-messages-with-model-form" target="_blank">this discussion on Stack Overflow</a>. This is of course very unfortunate.<br />
<br />
Some things about the form fields already can be modified via the Meta class of the ModelForm, for example the widgets used to render the form fields. Unfortunately, setting of custom error messages via the Meta class does not seem to be possible.<br />
<br />
So I implemented a small change to the ModelForm class, which allows you to set arbitrary field attributes via the Meta class. For that, I introduced a new Meta class field, called "<span style="font-family: 'Courier New', Courier, monospace;">field_args</span>". It is used like this (note that we are deriving from a new base class, called <span style="font-family: 'Courier New', Courier, monospace;">ExtendedMetaModelForm</span>):<br />
<br />
<pre>class AuthorForm(ExtendedMetaModelForm):
class Meta:
model = Author
field_args = {
"first_name" : {
"error_messages" : {
"required" : "Please let us know what to call you!"
}
},
"notes" : {
"+error_messages" : {
"required" : "Please also enter some notes.",
"invalid" : "This is not a valid note."
},
"widget" : forms.Textarea(attrs={'cols': 70, 'rows': 15}),
}
}</pre><br />
As you can see, <span style="font-family: 'Courier New', Courier, monospace;">field_args</span> is a dictionary of dictionaries. Each dictionary within <span style="font-family: 'Courier New', Courier, monospace;">field_args</span> specifies the field attributes you wish to set for a given field. As you can see, you are not limited to the error messages. For instance, the "notes" field here receives a custom widget. Please note that the Meta class already allows you to set custom widgets. This example here is is just to show that you are free to use <span style="font-family: 'Courier New', Courier, monospace;">field_args</span> to set any field attribute.<br />
<br />
Another interesting feature is illustrated with the error messages for the "notes" field. As you can see we have a plus sign at the start of "+<span style="font-family: 'Courier New', Courier, monospace;">error_messages</span>". This merely is used as an indicator that we wish to merge the specified attributes to an already existing attribute, rather than fully replace it. This only is supported if both the newly defined value and the already existing value for the named field attribute are dictionary types. The advantage of using the "+" notation is that you do not have to fully specify all error messages - for example - if you only wish to customize a few of them. In our example above, the error messages for the <span style="font-family: 'Courier New', Courier, monospace;">first_name</span> field are fully replaced, while for the <span style="font-family: 'Courier New', Courier, monospace;">notes</span> field they are merely modified or appended to.<br />
<br />
The new <span style="font-family: 'Courier New', Courier, monospace;">ExtendedMetaModelForm</span> can be used anywhere you traditionally would have used <span style="font-family: 'Courier New', Courier, monospace;">forms.ModelForm</span>. The source code for this class is as follows:<br />
<br />
<pre>class ExtendedMetaModelForm(forms.ModelForm):
<span style="color: #274e13;"> """
Allow the setting of any field attributes via the Meta class.
"""</span>
def __init__(self, *args, **kwargs):
<span style="color: #274e13;"> """
Iterate over fields, set attributes from Meta.field_args.
"""</span>
super(ExtendedMetaModelForm, self).__init__(*args, **kwargs)
if hasattr(self.Meta, "field_args"):
<span style="color: #274e13;"> # Look at the field_args Meta class attribute to get
# any (additional) attributes we should set for a field.
</span> field_args = self.Meta.field_args
<span style="color: #274e13;"> # Iterate over all fields...
</span> for fname, field in self.fields.items():
<span style="color: #274e13;"> # Check if we have something for that field in field_args
</span> fargs = field_args.get(fname)
if fargs:
<span style="color: #274e13;"> # Iterate over all attributes for a field that we
# have specified in field_args
</span> for attr_name, attr_val in fargs.items():
if attr_name.startswith("+"):
merge_attempt = True
attr_name = attr_name[1:]
else:
merge_attempt = False
orig_attr_val = getattr(field, attr_name, None)
if orig_attr_val and merge_attempt and \
type(orig_attr_val) == dict and \
type(attr_val) == dict:
<span style="color: #274e13;"> # Merge dictionaries together
</span> orig_attr_val.update(attr_val)
else:
<span style="color: #274e13;"> # Replace existing attribute
</span> setattr(field, attr_name, attr_val)
</pre><br />
I hope this little code snippet here helps you to make your work with Django's model forms easier.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com1tag:blogger.com,1999:blog-6748806271290317100.post-49635868370356128772012-01-19T06:54:00.005+13:002012-01-19T07:03:42.630+13:00How to use django_extensions' runscript commandThe <a href="https://github.com/django-extensions/django-extensions" target="_blank">django_extensions</a> app provides many helpful additions for the Django developer, which are worth checking out. I highly recommend it to anyone working on a Django project. One such addition is <span style="font-family: 'Courier New', Courier, monospace;">runscript</span>, which allows you to run any script in the Django context, without having to manually import and configure your paths and Django settings. Unfortunately, there is very little documentation on how to correctly use it. This blog post here just quickly sums it all up:<br />
<div><br />
</div><div><b>1. Running a script</b></div><div><b><br />
</b></div><div>This is done like this:</div><div><br />
</div><div><span style="font-family: 'Courier New', Courier, monospace;"> ./manage.py runscript <scriptname></span></div><div><span style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div>You cannot specify the name of a specific Python file, though. Scripts have to be within modules in a particular location, which is explained next.</div><div><br />
</div><div><b>2. The location of scripts</b></div><div><b><br />
</b></div><div>The <span style="font-family: 'Courier New', Courier, monospace;">runscript</span><span style="font-family: inherit;"> command looks for modules called </span><span style="font-family: 'Courier New', Courier, monospace;">scripts</span><span style="font-family: inherit;">. These modules may be located in your project root directory, but may also reside in any of your apps directories. This allows you to maintain app-specific scripts. So, create the scripts directory (either at the root or within an apps directory), along with an </span><span style="font-family: 'Courier New', Courier, monospace;">__init__.py</span><span style="font-family: inherit;"> file:</span></div><div><span style="font-family: inherit;"><br />
</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> mkdir scripts</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> touch scripts/__init__.py</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><b>3. The script itself</b></div><div><b><br />
</b></div><div>Scripts are simple Python files, but they need to contain a <span style="font-family: 'Courier New', Courier, monospace;">run()</span> function. For example like this:</div><div><br />
</div><div><span style="font-family: 'Courier New', Courier, monospace;"> def run():</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> print "I am a script"</span></div><div><span style="font-family: inherit;"><br />
</span></div><div><span style="font-family: inherit;">Within such a script, you could import and use any of your models or other parts of your Django project. When you store the above code in a file called </span><span style="font-family: 'Courier New', Courier, monospace;">scripts/testscript.py</span><span style="font-family: inherit;"> then you can run it like so (note that this is without the </span><span style="font-family: 'Courier New', Courier, monospace;">.py</span><span style="font-family: inherit;"> extension):</span></div><div><span style="font-family: inherit;"><br />
</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> ./manage.py runscript testscript</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span style="font-family: inherit;"><b>4. Run multiple scripts at once</b></span></div><div><span style="font-family: inherit;"><b><br />
</b></span></div><div><span style="font-family: inherit;">A nice feature of </span><span style="font-family: 'Courier New', Courier, monospace;">runscript</span><span style="font-family: inherit;"> is that you can execute multiple scripts at the same time, if those scripts have the same name. Assume that you have different applications and wish to initialize data for them by running a script. Rather than putting all of this into a single script, you could have a </span><span style="font-family: 'Courier New', Courier, monospace;">scripts</span><span style="font-family: inherit;"> directory in your various apps directories, and within each of those have - for example - an </span><span style="font-family: 'Courier New', Courier, monospace;">init_data.py</span><span style="font-family: inherit;"> script, thus keeping each script smaller and easier to understand and maintain.</span></div><div><span style="font-family: inherit;"><br />
</span></div><div><span style="font-family: inherit;">You can see this in action if you specify more verbose output via the "</span><span style="font-family: 'Courier New', Courier, monospace;">-v 2</span><span style="font-family: inherit;">" option. </span></div><div><div><span style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> ./manage.py runscript -v 2 init_data</span></div></div><div><span style="font-family: 'Courier New', Courier, monospace;"><br />
</span></div><div><span style="font-family: inherit;">The output will look something like this:</span></div><div><span style="font-family: inherit;"><br />
</span></div><div><div><span style="font-family: 'Courier New', Courier, monospace;"> Check for django_extensions.scripts.init_data</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> Check for django.contrib.auth.scripts.init_data</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> ...</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> Check for one_of_my_apps.scripts.init_data</span></div><div><div><span style="color: #274e13; font-family: 'Courier New', Courier, monospace;"> Found script '</span><span style="color: #274e13; font-family: 'Courier New', Courier, monospace;">one_of_my_apps.</span><span style="color: #274e13; font-family: 'Courier New', Courier, monospace;">scripts.init_data' ...</span></div><div><span style="color: #274e13; font-family: 'Courier New', Courier, monospace;"> Running script 'one_of_my_apps.scripts.init_data' ...</span></div></div><div><span style="color: #351c75; font-family: 'Courier New', Courier, monospace;"> I am a script for one of my apps.</span></div><div><span style="font-family: 'Courier New', Courier, monospace;"> Check for some_other_app.scripts.init_data</span></div><div><div><div><span style="color: #38761d; font-family: 'Courier New', Courier, monospace;"> Found script '</span><span style="color: #38761d; font-family: 'Courier New', Courier, monospace;">some_other_app.</span><span style="color: #38761d; font-family: 'Courier New', Courier, monospace;">scripts.init_data' ...</span></div><div><span style="color: #38761d; font-family: 'Courier New', Courier, monospace;"> Running script 'some_other_app.scripts.init_data' ...</span></div></div><div><span style="color: #351c75; font-family: 'Courier New', Courier, monospace;"> I am a script for some other app.</span></div></div><div><div><span style="font-family: 'Courier New', Courier, monospace;"> Check for scripts.init_data</span></div><div><span style="color: #38761d; font-family: 'Courier New', Courier, monospace;"> Found script 'scripts.init_data' ...</span></div><div><span style="color: #38761d; font-family: 'Courier New', Courier, monospace;"> Running script 'scripts.init_data' ...</span></div></div><div><span style="color: #351c75; font-family: 'Courier New', Courier, monospace;"> I am a script at the root level.</span></div><div style="font-family: inherit;"><br />
</div></div><div><span style="font-family: inherit;">As you can see, </span><span style="font-family: 'Courier New', Courier, monospace;">runscript</span><span style="font-family: inherit;"> checks for a </span><span style="font-family: 'Courier New', Courier, monospace;">scripts</span><span style="font-family: inherit;"> module for every installed application, plus on the root level. If it finds a match, it runs the script and then keeps looking for more matches.</span></div><div><span style="font-family: inherit;"><br />
</span></div><div><span style="font-family: inherit;"><br />
</span></div>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com2tag:blogger.com,1999:blog-6748806271290317100.post-17724852645929738332011-08-28T16:41:00.000+12:002011-08-28T16:41:11.161+12:00Kiwi Pycon 2011Kiwi Pycon 2011, the annual Python conference in New Zealand has just been concluded. Two days in Wellington, full of inspiring talks and discussion. Thank you to the organizers and all the presenters. It's really nice to see other developers interested in Python.<br />
<br />
In New Zealand, the Python community is pretty small. If you look through the software related section in a bookstore, you will find very little about Python. And the Python job market... well, there really isn't much. That is probably one of the reasons why I mostly work with overseas clients.<br />
<br />
Hopefully, Python continues to make inroads with New Zealand organizations. It would benefit them and all of us here on this small, South Pacific island.<br />
<br />
jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-57641967088759264022011-08-17T09:48:00.002+12:002011-08-17T09:57:20.331+12:00Tips on using oDesk contractors as system administratorsSites like <a href="https://www.odesk.com/">oDesk</a>, <a href="http://www.elance.com/">eLance</a> and others give any business instant access to skilled, remote staff, often working at very competitive hourly prices. Basically, you can find outsourced workers (usually individuals, sometimes teams) for small to mid-sized projects you need to get done, while not having to deal with paperwork or recruiting. A lot has been written about the pitfalls of outsourcing and also on how to make such outsourced work succeed: Define the project very well, document the APIs against which the contractor should work, demand good documentation, etc. However, most of these tips and tricks have been written about software development and coding.<br />
<br />
But what about tasks such as system administration? You can find quite a few sysadmins on these sites. If you are an entrepreneur who wishes to focus on the business idea, or maybe even just on the software development, what is more tempting than to outsource those pesky aspects of system administration to someone else? Setting up a web-server with caching proxy, a database server, a CMS, a cluster of build and test servers, etc. After all, these might not be your core skills. Figuring all of this out yourself would take time, you might still not get it right and it certainly is quite distracting. So, in theory, these should make for perfect outsourced projects.<br />
<br />
There's just one problem.<br />
<br />
Consider that most of the system administration tasks I just mentioned require root or administrator access to your systems. Those systems later will hold your software, sometimes your source code or - even more important - your customer data, maybe logins, addresses or even credit card numbers.<br />
<br />
Now consider that the contractors you meet on oDesk are very often in different countries and that you will have no contract with them besides whatever automated agreement you strike up via oDesk. You won't meet them in person, there usually are no NDAs and sometimes you won't even talk to them on the phone.<br />
<br />
I have worked with quite a few oDesk contractors now and was lucky enough to meet many fine engineers and administrators. But if push comes to shove, you just don't know ahead of time who you are really going to get, and considering that any dispute would likely be across borders and legal systems, realistically you would have no legal recourse in case something goes wrong.<br />
<br />
While any issues or disputes are bad enough when it comes to software development, you at least have a deliverable you can review and examine relatively easily, or even throw away should you decide to do so. However, for system administration tasks, there are numerous ways in which a disgruntled or simply malicious or careless person with root access could compromise your system before it goes into production. Backdoors could have been installed, timed jobs could run unbeknownst to you, collecting and sending off important data from your system for some nefarious purposes, or some passwords and accounts could accidentally have been left open and unprotected.<br />
<br />
So, having used oDesk contractors for a number of projects, including system administration tasks, let me give you two recommendations:<br />
<br />
Firstly - and true for any type of task - choose contractors who have worked a lot on oDesk (or whichever other outsourcing site you use) and who are still active there. Look for the "recently worked hours" and recent projects. Look for contractors with a good reputation. The idea is to choose contractors who really work projects on those sites for a living and thus rely on a good reputation. That, of course, is still not a guarantee, but it is a good starting point.<br />
<br />
Secondly, if at all possible, do NOT give "system administration" tasks to contractors (which require root access to production systems). Instead, give "system administration automation" tasks. That's a subtle but important difference. However, it pays off in several ways. These are slightly arbitrary definitions, but they may serve to make a point:<br />
<br />
With "system administration" tasks I mean: Setting up a complete server for you, or installing and configuring an additional software package on your server. Basically, working directly on a production system. You give someone root on those machines and they have the keys to the kingdom, with all the issues we have described before. If you need to have those tasks done, you should probably look for someone you can meet in person, get a better feel for and develop a better trust relationship with.<br />
<br />
On the other hand, with "system administration automation" tasks I mean the development of scripts, which setup a server (or software package) for you in an automated fashion. How does this work? There are particular tools (for example <a href="http://projects.puppetlabs.com/projects/puppet">puppet</a> or <a href="http://wiki.opscode.com/display/chef/Home">chef</a>), or just a number of custom shell scripts, which can be used to automate the administration and setup of machines. Now the project description is different. While before you description may have read <i>"Configure a caching proxy for our application server..." </i>it now changes to <i>"Provide scripts, which automate the setup of a caching proxy for our application server..."</i>.<br />
<br />
What do you gain with this? On one hand, you can just give a throw-away system to the contractor on which to develop those scripts. Throw away servers are easy and cheap to get these days. You may use <a href="http://aws.amazon.com/ec2/">Amazon EC2</a>, <a href="http://www.rackspace.com/">Rackspace</a>, <a href="http://www.linode.com/index.cfm">Linode</a> or any number of other providers to get "dedicated looking" machines for just a few dollars per month or even just on an hourly basis. On those machines, the contractor can get a root login and can develop the automation scripts until they work. Once the project is completed you can test it easily: Get yourself another throw away server and run the finished scripts (which naturally have to come with documentation on how to run them). If your server ends up configured properly, the project was successful. You may specify scripts for automated tests and self-diagnosis as part of the deliverable as well. However, at no point did the contractor require root or administration rights on your actual production server.<br />
<br />
It's important to use the same or similar base machine image for the throw away server as for the production server. So for example, if you run your production server on Linode, with Ubuntu 10.04 LTS as base OS you should create a similar Linode instance with the same base OS for the contractor as throw away server to work on. Then likewise to test the deliverables before applying those scripts to your production server. Also, consider that if all your configuration setups for your production server are automated then it is easy to give a complete mock production server to the contractor as a throw away server: You can quickly bring the throw away server into the same state (or at least containing the same configurations and software packages) as your production server before the contractor starts work. Just be careful not divulge important passwords or SSH keys that way.<br />
<br />
A second advantage of requiring automation scripts is that the scripts themselves can act as a documentation of sorts of all the tasks and steps that are involved in completing this system administration task. Therefore, they will allow you and other contractors to more easily pick up where one contractor left off.<br />
<br />
Finally, if you did not automate the system administration task, you cannot easily reproduce the work in case you want to bring up an additional production server. So, having automated scripts helps you to scale more effectively, as your organization grows.<br />
<br />
Do you have any other tips and ideas on how to effectively work with remote contractors for system administration tasks? If so, please let us know in the comments.<br />
<br />
So, the next time you ask for a system administration task to be performed and before handing over the root password to your production server to the contractor, consider if it might not be better to instead phrase the project as a system administration automation task instead.<br />
<br />
<br />
You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.<br />
<br />
jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com2tag:blogger.com,1999:blog-6748806271290317100.post-28469662569848607232011-07-14T14:40:00.003+12:002011-08-13T10:39:35.703+12:00Automatically create a table of contents with jQueryRecently, I had to create an FAQ page. A good FAQ should have a table of contents (TOC) at the start, easily allowing you to see the various section of the FAQ and the questions in each section. Naturally, I didn't want to have to maintain the TOC manually, every time a question/answer item had to be moved, added or deleted. On the server side, we didn't have a CMS in place (the project doesn't really require this, the FAQ is the only page with a TOC).<br />
<br />
So, what to do? I could write scripts, running on the server, parsing a questions/answer file, generating the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">faq.html</span> file out of that. Or I could use one of any number of other server-side tools. But since I had just started to get into jQuery, I decided to try something slightly different. Why not dynamically create the TOC and the numbering of the questions when the page is loaded? In other words, the HTML of the page would only contain questions and answers, but no TOC and no fixed numbering of the questions; all of that would be added dynamically in the browser when the page was displayed.<br />
<div class="separator" style="clear: both; text-align: center;"></div><br />
<div class="separator" style="clear: both; text-align: center;"></div>This seemed to be as good an exercise as any to try out my fledgling jQuery skills. Well, it works! Here is the JavaScript:<br />
<br />
<script src="http://pastebin.com/embed_js.php?i=JQRD08Mq"></script><br />
<br />
You can see the result here, in this extremely <a href="http://brendel.com/pages/toc_test.html">simple test page</a>, which has no external dependencies, except the jQuery library. So, all you need can be seen in the source of that page. For illustration purposes, switch off JavaScript in your browser and reload the page. As you can see, it still looks readable, just the numbering and TOC is gone.<br />
<br />
If you examine the page source, you can see a blob of jQuery in the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ready()</span> function, but no TOC. Instead, you have the questions and answers, following a particular structure. The jQuery code examines this structure and dynamically creates the TOC, the numbering of each entry and the links.<br />
<br />
The structure you have to follow with your questions and answers is very simple:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5HikKrn0LAS4y_JPAshJ7QphvOw6L8gxWppEb795ehfCO9ZX5_mu3GR655ovOs1v_-720LFp6qt9ciW4V3GZq_S_QXFdUEMtZM8dfT7iec3KGnBNFD2Q_3KW94Nhcmu-WJxz03VBJAi1R/s1600/toc_structure.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi5HikKrn0LAS4y_JPAshJ7QphvOw6L8gxWppEb795ehfCO9ZX5_mu3GR655ovOs1v_-720LFp6qt9ciW4V3GZq_S_QXFdUEMtZM8dfT7iec3KGnBNFD2Q_3KW94Nhcmu-WJxz03VBJAi1R/s1600/toc_structure.png" /></a></div><br />
A <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">div</span> with id "<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">faq_toc</span>" is where the TOC will be created. All top-level sections (or 'chapters') of the FAQ need to be enclosed in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">div</span> elements with class "<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">level_1_elem</span>". In each of those sections, the individual questions and answers are enclosed in a simple <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">div</span> element. You can see that there are definitions for "<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">level_1_link</span>" and "<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">qlink</span>". These are the anchor tags, which allow linking to the individual answers and sections. These anchors do not contain any numbering - static or auto-generated. This means that even after you move questions around or add questions ahead of them, any links to those questions remain valid.<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">The result is a properly formatted TOC and numbered questions and answers, looking something like this:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzzIyLclYRyDDVDtjKZiBcEFOJq4SKhj-OTIq-nNI9Xs3-63xBJeHOXrDxv9v_CqyQ7fH1laAU-5xJa2t1VbM6fw-cIK3_h6UCLiLxBHIf8DQZACd2pLXXi7SzLhvl7_JFvlXdnEFTbzyZ/s1600/toc_image.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzzIyLclYRyDDVDtjKZiBcEFOJq4SKhj-OTIq-nNI9Xs3-63xBJeHOXrDxv9v_CqyQ7fH1laAU-5xJa2t1VbM6fw-cIK3_h6UCLiLxBHIf8DQZACd2pLXXi7SzLhvl7_JFvlXdnEFTbzyZ/s1600/toc_image.png" /></a></div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div>This, I think, is not bad at all. No need to add a CMS or anything complicated on the server side, if the client can generate the TOC for us so nicely.</div><div><br />
</div>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-17285364300211783402011-06-25T03:30:00.001+12:002011-09-22T14:28:54.378+12:00Wireless USB headsets on Linux: My experience with the Logitech G930USB wireless headsets can be a bit of a hit and miss affair on Linux. Stories abound of these devices failing to work, people not being able to use them for Skype, etc. I'm running Ubuntu 10.04 (Lucid Lynx) on my somewhat aging Dell Latitude D820 laptop, but recently decided to give one of those headsets a try: The Logitech G930. Here I am describing how it's performing for me.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ6XSj_V_me20aQzSQiTcWO4JkKWwFKW1YgaWQwlruPJa5JB-L6wlu2T2RD2yM5A9T6-1lsXGG-eSprtQ9JsNF-XcTtne5lP7hUkh1ub5KR-vVugiJN0DTF-BSPiCThCIJg08rFD2IyB_g/s1600/Screenshot-2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="295" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ6XSj_V_me20aQzSQiTcWO4JkKWwFKW1YgaWQwlruPJa5JB-L6wlu2T2RD2yM5A9T6-1lsXGG-eSprtQ9JsNF-XcTtne5lP7hUkh1ub5KR-vVugiJN0DTF-BSPiCThCIJg08rFD2IyB_g/s320/Screenshot-2.png" width="320" /></a></div>For many years, I have used cheap wired headsets with ordinary analog audio headphone/mic plugs. But I'm spending a lot of time on Skype, talking to my overseas colleagues and customers. Since it seems to help me think, I have the tendency to walk around during discussions, usually just in circles in my home-office. But even for this small distance from my desk, wired headsets get in the way. So, a comfortable fitting wireless headset would be great.<br />
<br />
While general USB support on Linux is excellent, wireless USB headsets historically were not very well supported, which is why I have been hesitant to try them out. I was looking around for a good RF-based headset, but what I found was either very pricey or just headphones (without a microphone). Then I came across some reports of people claiming to successfully use the Logitech G930 wireless USB headset on Linux. While this is marketed as a “gaming headset”, I figured that as long as it worked, I wouldn't care what it's called. So, with some trepidation, I decided to order this headset from Amazon. Pricing for this product can differ quite a bit, but you should expect to spend somewhere between US$90 to US$120.<br />
<br />
<b>The hardware</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsJ1SqkXHaxuS3rEN9dTMZM9VUCK-gTRLgVZ8JY7a1q7JGRtg6EhrGp9eGyDTJ5r8dvJO6S-npwvzQZ0_lQe2Hx0kOWiuhFfmGKf3SoaEtDcfvTsQdmdaW62hshJppolLrrzkK8EW9l4JS/s1600/7302.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhsJ1SqkXHaxuS3rEN9dTMZM9VUCK-gTRLgVZ8JY7a1q7JGRtg6EhrGp9eGyDTJ5r8dvJO6S-npwvzQZ0_lQe2Hx0kOWiuhFfmGKf3SoaEtDcfvTsQdmdaW62hshJppolLrrzkK8EW9l4JS/s320/7302.jpg" width="307" /></a></div>The headset arrived a few days later. In the packaging I found the headset itself, which feels like a solidly made device. It is very light and provides ample space around your ears, so it remains comfortable even for longer usage. There is also a USB connected base station – which conveniently also contains the charger cable for the headset itself – and a USB stick with the actual dongle for the wireless connection. You can see it all in this picture here (all images in this article – except screenshots – are courtesy of <a href="http://www.logitech.com/en-us/gaming/headsets/devices/7248">Logitech</a> or <a href="http://www.desktopreview.com/default.asp?newsID=1178">DesktopReview.com</a>).<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOJqlfDhrz8nK9MkSceF6iv8LW94e_6WFGPb9xe-Fb-VWWGXFKTqw0HnHBqnSILBRrVDcpVm3bxj75dWZrcLgM81QUtjJpQOpmiGCR8Zz1epjRP1oVM0ItzcYLgSAPG1H4ovoRW2wYt7gF/s1600/Screenshot.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="175" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjOJqlfDhrz8nK9MkSceF6iv8LW94e_6WFGPb9xe-Fb-VWWGXFKTqw0HnHBqnSILBRrVDcpVm3bxj75dWZrcLgM81QUtjJpQOpmiGCR8Zz1epjRP1oVM0ItzcYLgSAPG1H4ovoRW2wYt7gF/s320/Screenshot.png" width="320" /></a></div>The 'base station' is a purely passive device. Basically, a fancy USB cable or hub with a drum to tidily roll up your cables. You can see that it contains a single USB plug at the top. This allows the wireless dongle to be plugged in there, rather than into one of the ports of your computer. That is very welcome, since the dongle is quite long and would inevitably be bumped-in or bent. You avoid this danger by plugging it into that base station and placing it somewhere out of the way. The second, smaller cable from the base station is used to charge your headset. Fortunately, the headset works even while charging. The USB dongle may also be plugged straight into the computer, in which case the cable roll is not needed for operation.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjD_jD_AgytgmD7gNEVS6-UlgGKFDJ-au2TXtalrEQMk_08-BZi859Ahh0t71AnpaOyqWrDhHrhshgVe9CU35xN1xQw_GW4Kx1dyWVH-MYFXjh9lf5o6Nz5kGiZp7r9-bQ-4pUd_UoyD-M/s1600/Screenshot-3.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjD_jD_AgytgmD7gNEVS6-UlgGKFDJ-au2TXtalrEQMk_08-BZi859Ahh0t71AnpaOyqWrDhHrhshgVe9CU35xN1xQw_GW4Kx1dyWVH-MYFXjh9lf5o6Nz5kGiZp7r9-bQ-4pUd_UoyD-M/s320/Screenshot-3.png" width="270" /></a></div>The microphone feels solid and automatically mutes itself if it is flipped upwards. There is also a mute button on the left ear cup. When muted, a small red light at the end of the microphone serves as visual reminder to the wearer. A nice feature. The microphone beam actually is flexible so that it can be bent towards your mouth when talking. This probably also prevents it from breaking easily should the headset be dropped. The entire microphone connection and assembly feels solid and well done. Additional controls on the left ear cup are a roller for volume, an on-off button, a Dolby 7.1 surround button and three special effect G-buttons, which supposedly can be programmed with various functions.<br />
<br />
<b>The Linux experience</b><br />
<br />
There is some software that comes in the packaging, but as usual, it is Windows only. So, I ignored it and just plugged in the base station cable and USB dongle, switched on the headphones, waited for the green light on dongle and headset to indicate a good connection and anxiously wondered what would happened. At first nothing. Music output continued on my laptop's built-in speakers.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnn9K8NaxZ4UIQ0mrWpVMfTXsORM8hPhxvLi1WC2_HhbFBThxk1YMERsfF_aFKqaK5rD_OjFzeo8giOeictXK_ae7HZc90rA_PToIiJghuHvweGT-a1bteoxwhpoc8PI69OU5QYdh-pJUS/s1600/Screenshot-5.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnn9K8NaxZ4UIQ0mrWpVMfTXsORM8hPhxvLi1WC2_HhbFBThxk1YMERsfF_aFKqaK5rD_OjFzeo8giOeictXK_ae7HZc90rA_PToIiJghuHvweGT-a1bteoxwhpoc8PI69OU5QYdh-pJUS/s320/Screenshot-5.png" width="320" /></a></div>But in no time, I noticed that a new sound device had been identified: When you open the PulseAudio volume control application, you can see it in the Output and Input tabs: Logitech G930 Headset Analog Stereo for output and Logitech G930 Headset Analog Mono for input. I have to say, I was very happy to see that without any special software the device was correctly recognized. Compliments to Linux's great USB support.<br />
<br />
I selected the Logitech device in both the input and output tap, restarted my music player and there it was: Wireless sound in my new headset! Then I started Skype and made my first test call, which worked as well! The microphone was recognized and used without problem. I did not have to change any settings in Skype. In fact, the sound quality of the microphone is very good, better than with previous standard wired headsets. The built-in sound card in my laptop had always produced a slight crackle whenever I used the microphone. Annoying for others in conference calls or when I wanted to record voice-overs for screencasts. However, with this USB headset there was no crackle at all, just a nice and clear recording.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6jEmgvCDcIe7NdY9Z9A8hI6srUmR44UrlrLi4YxTLmU4LQL3OjMSsc3KErCDGWL1-_CM9LNwU5Njv7b70ZdEPvvb_eGpEj72FTlnjCNT3otgbg01WYqrlhRkvNO-pdbiSf9g_jVWYNalb/s1600/Screenshot-4.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6jEmgvCDcIe7NdY9Z9A8hI6srUmR44UrlrLi4YxTLmU4LQL3OjMSsc3KErCDGWL1-_CM9LNwU5Njv7b70ZdEPvvb_eGpEj72FTlnjCNT3otgbg01WYqrlhRkvNO-pdbiSf9g_jVWYNalb/s1600/Screenshot-4.png" /></a></div>The roller for volume adjustments works well. However, it does not change the headset volume alone, but actually the volume for the entire system. Using it has the same effect as using the ACPI volume buttons on my laptop, with the familiar volume notifier appearing in the upper right corner of my desktop when the roller is used.<br />
<br />
<b>Sound quality: Using an equalizer to get some bass</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div>While the sound is clear, initially I was a little disappointed about the lack of depth and bass for the G930. However, apparently for good reasons the Windows version of the Logitech software for this headset contains an equalizer, which would allow you to put more emphasis on the lower frequencies. So, I figured I should try this on Linux as well. All I needed was an equalizer. Sadly, there wasn't one in the repositories. Why PulseAudio wouldn't come bundled with an equalizer by default, or even have one available in the standard repos is beyond me. After some searching, however, I found Conn O Griofa's wonderful <a href="https://launchpad.net/~psyke83/+archive/ppa">equalizer application for PulseAudio</a>. Conn has his own repository. So, all you need to do is to execute these instructions:<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> sudo add-apt-repository ppa:psyke83/ppa</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> sudo apt-get update</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> sudo apt-get install pulseaudio-equalizer</span><br />
<br />
The PulseAudio Equalizer then becomes available under your <i>Applications > Sound</i> menu. Select the settings form you like, save and apply. This equalizer application makes a huge difference. While before the headset sounded “tinny” and flat, music now suddenly had great volume, very satisfying bass and depth. Here are my settings:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3DW16AyOsNofzKDF7K2k4UGyfWnD6oQjEtErNs-Wct1SlGXk16s3tT4cwkuZnYQCqMxaDViYUh8zn1ltbC_gv7AheMNzSUTYNf1SYPMNXKp9WSbiaMQWCAFwM8dOvD_TAjziM7Ifh_LDX/s1600/Screenshot-6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3DW16AyOsNofzKDF7K2k4UGyfWnD6oQjEtErNs-Wct1SlGXk16s3tT4cwkuZnYQCqMxaDViYUh8zn1ltbC_gv7AheMNzSUTYNf1SYPMNXKp9WSbiaMQWCAFwM8dOvD_TAjziM7Ifh_LDX/s400/Screenshot-6.png" width="400" /></a></div><br />
Notice that when active, the equalizer appears as its own output device, which has to be selected.<br />
<br />
It's a good idea to start the equalizer whenever you login. For that, go to <i>System > Preferences > Startup Applications</i>. Click 'Add' to enter a new startup-application. Enter “pulseaudio-equalizer enable” in the command field, as shown here. That way, the equalizer always starts and its settings are applied.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiN09RVTXi269Sb1T_UAKmdbGA6VsMxB6sLlWdlfHq6tbzrWUSZGranyMzz7LY7E9ZMB0LjOz3DmfUCg1HSr82I0G-p6H7rEvY_FYCpzh-NlOvQ2wdgghyphenhyphen4B6Wo9DMvxjEFJAUVm5omxc_I/s1600/Screenshot-8.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiN09RVTXi269Sb1T_UAKmdbGA6VsMxB6sLlWdlfHq6tbzrWUSZGranyMzz7LY7E9ZMB0LjOz3DmfUCg1HSr82I0G-p6H7rEvY_FYCpzh-NlOvQ2wdgghyphenhyphen4B6Wo9DMvxjEFJAUVm5omxc_I/s320/Screenshot-8.png" width="320" /></a></div>Please make sure to not turn up the low frequency too high. If you do, you will get some pretty nasty distortions or clipping on louder bass tones. However, my settings shown above are probably quite moderate, avoid those distortions and result in great sound with the Logitech G930 headset.<br />
<br />
<b>Wireless performance</b><br />
<br />
The range and quality of the wireless signal is impressive. I have walked out my room, down a pretty long corridor, with the signal having to traverse 'around' a TV, through a heavy door and pretty stable wall and along the corridor. Only then were the first distortions audible. The range is advertised as 40 feet, but I would think it's actually more than that.<br />
<br />
<b>When there is much light...</b><br />
<br />
Considering that the custom Windows software for the G930 cannot be used on Linux, it is no surprise that a few things do not work. Likewise, Linux itself – particularly PulseAudio – still has a few noticeable quirks. Here's a list of what I found so far:<br />
<br />
<ul><li>The programmable “G buttons” on the left ear cup don't do anything. On Windows, using Logitech's provided software, you can program them with various functions, such as voice morph, or special application specific shortcuts. It would be nice if I could program those on Linux as well, for example to pause my music player, or to pick up an incoming Skype call. Sadly, it's not meant to be, unless Logitech (or some enterprising soul) manages to release useful Linux software for this task.</li>
<li>The Dolby Surround Sound button has no effect. Since I don't use the headset for gaming, though, this doesn't concern me much.</li>
<li>When you adjust the volume, either with your computer's ACPI buttons or the roller on the side of the headset, sometimes a faint crackling is audible. It's not really very bothersome, since it's only there right when you change the volume, but I noticed it nevertheless.</li>
<li>The ear cups are nicely padded and thus muffle any ambient noise well. However, when you are in a VoIP call, you therefore also have no good feeling for the sound or volume of your own voice. It doesn't seem to be possible to redirect your microphone input to the headset speakers. This, however, seems to be more of a PulseAudio limitation than of the headset. I tried to switch-on the loopback module for PulseAudio, but there is too much of a delay to make this useful. So, when using VoIP, you have to be a bit mindful of your own voice. You can get used to it, though.</li>
<li>For some reason, I cannot switch sound devices in the middle of a song. In fact, when I want to switch to the external speakers, or back to the USB headset, I have to do that switch in the Input and Output device dialogs of the PulseAudio volume control application. And even then at first nothing changes. I actually have to restart my sound applications (Skype, gnome-player, etc.) before the change is recognized. Quite annoying. Again, this seems to be mostly a PulseAudio limitation, and doesn't appear to have anything to do with the Logitech headset per-se. However, after always using the standard, built-in sound card and cheap headphones where changing devices was merely a matter of pulling a plug, I certainly noticed that particular aspect. Pulling the plug on the USB dongle turned out to be not a good idea: Doing so at some point locked up my computer. So, switching the devices in software is the preferred way. A properly working and simple to use PulseAudio device chooser application would be great. There is a package with exactly this name in the Ubuntu repositories, but it does not do – at least not easily – what the name seems to promise.</li>
</ul><br />
<b>Conclusion</b><br />
<br />
For the most part the Logitech G930 on Ubuntu (10.04 and most likely also later versions) has been a very positive “works out of the box” experience for me. Great sound quality, comfortable to wear, good for music and VoIP, good build quality. I'm happy with my purchase. While there is room for improvement, I would certainly recommend the headphones for use on Linux already.<br />
<br />
You should <a href="http://twitter.com/BrendelConsult">follow me on Twitter...</a>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com17tag:blogger.com,1999:blog-6748806271290317100.post-78323076519751578782010-10-06T12:19:00.014+13:002010-10-07T05:07:06.681+13:00It's a long way to the next useful planetToday, a slight departure from the usual topic of this blog.<br />
<br />
Recently, there has been a lot of excitement about the discovery of <a href="http://en.wikipedia.org/wiki/Gliese_581_g">Gliese 581g</a>, a planet orbiting a star that is around 20 light years away from here in the constellation of Libra. What makes this extra-solar planet so interesting? It is believed to be rather Earth like in mass and one of the first so-called "<a href="http://en.wikipedia.org/wiki/Goldilocks_planet">Goldilocks</a>" planets: Located in the habitable zone around its star, not too far and cold, not too close and hot, but just right for life to possibly exist there.<br />
<br />
On a cosmic scale, 20 light years is pretty much nothing. For us humans, though, it's still very, very far away. For comparison: An Astronomical Unit (AU) is the average distance of Earth from the Sun, which is about 149.6 million kilometers. Light actually needs over 8 minutes to travel this distance! Now assume that our solar system has a diameter of roughly 80 AU or almost 12 billion (!) km, which is more or less the diameter of Pluto's orbit. A somewhat old-fashioned way to measure it I will admit, but old habits die hard. The Voyager spacecrafts, launched more than 30 years ago have traveled just over this distance now. Our solar system may be vast, but light can still cross it in just 11 hours.<br />
<br />
But Gliese 581g is over 20 light<b>years</b> away.<br />
<br />
Now how can we put this into perspective? How can we show how incredibly far away Gliese 581g is, compared to anything we can even remotely begin to comprehend? Well, I tried. Below is an extremely simple 'infographic'.<br />
<br />
At the top of the image you see a single-pixel dot in the middle of a square. That dot is the diameter of the solar system! Not the square, just the dot in its center. A single pixel representing all 80 AU, or 11 light hours; a distance it takes us around 30 years to travel with our current technology.<br />
<br />
Then start to scroll down. And scroll... and scroll...<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div style="text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="http://brendel.com/images/gliese.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://brendel.com/images/gliese.png" width="200" /></a></div><br />
</div>And here we are! 20 light years away from our solar system, away from our home. Hopefully, we remembered to turn the ship around at some point in order to slow down for our arrival.<br />
<br />
To recap, on this scale the diameter of the entire solar system is just 1 pixel! The image above is more than 15,000 pixels high. This means that the distance to Gliese 581g is 15,000 times more than the diameter of our solar system! The distance between Sun and Earth is 1/80th of a pixel. The diameter of Earth? Around 1/1,000,000 of a pixel.<br />
<br />
We have a long way to go.<br />
<br />
I would be happy if you would follow me on <a href="http://twitter.com/BrendelConsult">Twitter</a>. You might as well, after having traveled 20 light years...jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com1tag:blogger.com,1999:blog-6748806271290317100.post-7418735702168987192010-09-23T08:37:00.012+12:002010-09-29T13:05:57.847+13:00How to customize Django's default messagesEver tried to customize a message produced by one of Django's generic views or standard field validators? This is a problem I run into pretty often.<br />
<div><br />
</div><div><b><span class="Apple-style-span" style="font-size: large;">Custom validator messages</span></b></div><div><br />
</div><br />
<div>In some cases Django offers you some pretty straight forward options, such as specifying an <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">error_messages</span> dictionary as argument to a field validation:</div><small></small><br />
<small><pre>email = forms.EmailField(label = "Email address",
error_messages = {'required' : "Please enter an email address.",
'invalid' : "Malformed address. Please correct."})
</pre></small><br />
<div><b><span class="Apple-style-span" style="font-size: large;">Hard-coded messages</span></b></div><div><br />
</div><div>Sometimes, however, easy options like this do not exist. For example, take the generic <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">password_reset</span> view offered by <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">django.contrib.auth</span>. If you look at the source code for this view function, you find that by default it uses <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">PasswordResetForm</span> - also contained in the same package - as its standard form. The problem is that this form has a few hard-coded messages in it, so there is no obvious way to modify those if you just want to continue to use the standard views and forms.</div><div><br />
</div><div>The <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">password_reset</span> view gives you the option of specifying your own form - with the messages you want - by setting the <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">password_reset_form</span> parameter. For example, by doing this in your <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">url.py</span> file:</div><small></small><br />
<small><pre>(r'^password/reset/$', 'django.contrib.auth.views.password_reset',
{ 'template_name' : 'my_pw_reset_form.html',
'email_template_name' : 'my_pw_reset.txt',
'password_resset_form' : MyForm }),</pre></small><br />
<div>I don't know about you, but the thought of essentially copying an entire form implementation just so that I can change some message text is not very attractive to me. Some of these forms consist of more than just field definitions and I don't want to be in the business of creating and maintaining 'mini-forks' of various Django components.</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: large;">A better way</span></b></div><div><br />
</div><div>It seems that there should be a better way and I think there is: Let's use Django's built-in translation system for the task.</div><div><br />
</div><div>Django's translation is capable of taking any specially marked text in any of your files (including Python source code) and looking up a translated string for it, based on the locale of your site or user. For example, in your Python source code, you could write this:</div><small></small><br />
<small><pre>from django.utils.translation import ugettext_lazy as _
msg = _("Here is some text")</pre></small><br />
<div>The <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">_()</span> function here is the special marker and also performs the on-the-fly translation at runtime. In templates you use the <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">{% trans ... %}</span> template tag. You can read more about it <a href="http://docs.djangoproject.com/en/dev/topics/i18n/internationalization/">here</a>.</div><br />
<div>Fortunately, the hard-coded messages in the standard Django views and templates are prepared for translation. Therefore, the idea now is simple: Even if otherwise you are not interested in translation and internationalization, you simply use this mechanism to provide 'custom translations' for the selected messages you are interested in!</div><div><br />
</div><div>In my <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">settings.py</span> file, the <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">LANGUAGE_CODE</span> setting is 'en-us'. You may have a different setting, but that doesn't matter. While in your settings file, make sure that <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">USE_I18N</span> is set to <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">True</span> and that you include <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">django.contrib.messages.middleware.MessageMiddleware</span> in your <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">MIDDLEWARE_CLASSES</span>.</div><div><br />
</div><div>In your project directory, make sure you have a 'locale' directory (create it if it's missing) and then run this command:</div><small></small><br />
<small><pre>% django-admin.py makemessages -l en</pre></small><br />
<div>This utility examines all your files for strings that are marked for translation. In this case, I am asking it to create translation files for 'en' (English), but you can define whatever was specified in your <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">LANGUAGE_CODE</span> setting.</div><div></div><br />
<div>Have a look at the <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">locale/</span><i><span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">[local-id]</span></i><span class="Apple-style-span">/</span><locale-id><span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">LC_MESSAGES/django.po</span> file that was created (<i>[locale-id]</i> <locale-id>in my case is 'en', it may be different for you). If you don't have any of your own messages marked for translation, this file will be mostly empty. A <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">.po</span> file contains messages - that are recognized at runtime - and their translations that need to be applied. It's quite straight forward.</locale-id></locale-id></div><div><br />
</div><div>Now with an editor open the standard <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">.po</span> file for your locale that comes with a Django install. Django may be in your Python install's site-packages directory. The path therefore should be something like this:</div><small></small><br />
<small><pre>site-packages/django/conf/locale/[locale-id]/LC_MESSAGES/django.po</pre></small><br />
<div>In that file, search for the message you would like to customize. For our password-reset example, I don't like the standard message it displays when the email address cannot be found. In the standard <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">django.po</span> we find this message represented like this:</div><small></small><br />
<small><pre>#: contrib/auth/forms.py:110
msgid ""
"That e-mail address doesn't have an associated user account. Are you sure "
"you've registered?"
msgstr ""
</pre></small><br />
<div>This shows us that currently no translation is provided for this message (<span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">msgstr</span> is empty). If you don't have an English language locale then you probably already see a translation specified right here.</div><div><br />
Now instead of editing the message in the standard <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">.po</span> file that comes with Django - I don't like editing standard files like that - copy these lines into your own <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">django.po</span> file in your project directory. Use <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">msgstr</span> to define whatever text you want to be displayed instead. Single line messages can be written right into the two quotes behind <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">msgstr</span>, multi-line messages are written the way you see it done for <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">msgid</span>. In our example, I just say:</div><small></small><br />
<small><pre>msgstr "Unknown email. Are you sure you have registered with that address?"</pre></small><br />
<div>So, you now have provided a second translation for your default locale. Since Django looks first in your project's directory for translations of individual messages, it finds this translation before it falls back on the default provided by Django itself.</div><small></small><br />
<small><pre>% django-admin.py compilemessages</pre></small><br />
<div>Now you can restart your server and you should see your customized default message in action!</div><div><br />
</div><div><b><span class="Apple-style-span" style="font-size: large;">Maintenance</span></b></div><div><br />
</div><div>An annoying habit of the <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">makemessages</span> utility is that it comments out your custom messages out of your <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">.po</span> file every time it is run. The utility is pretty good in not losing any changes you make in that file, but every translation you add for which <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">makemessages</span> does not find the text in your project's files is going to be commented out. And since you are providing a translation for a standard Django message - and Django most likely is not part of your project - this will apply to the custom messages we are talking about here.</div><div><br />
</div><div>You could just manually remove all the comment identifiers ("<span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">#~ </span>"). However, since I'm lazy, I wrote a little script, which not only removes those comments for me, but also calls compilemessages. Just create a file called <span class="Apple-style-span" style="font-family: 'courier new'; font-size: small;">translate.sh</span> in your project directory with the following content:</div><small></small><br />
<small><pre>#!/bin/bash
echo "### Translating all messages..."
django-admin.py makemessages -a
echo "### Removing commented-out manual messages..."
find locale -name 'django.po' -exec sed s/^\#\~\ // -i {} \;
echo "### Compiling messages..."
django-admin.py compilemessages
echo "### Done!"</pre></small><br />
<div>Once you give execute permissions to that file, you can get all your project's messages discovered and compiled, while also retaining any of your customizations for Django's default messages, simply by running this command.</div><div><br />
</div><div>Pretty convenient, isn't it?</div><div><br />
</div><div>If you liked this little article, please consider <a href="http://twitter.com/BrendelConsult">following me on Twitter</a>.</div>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-50228579364144875582010-08-03T11:08:00.008+12:002010-08-06T06:30:50.676+12:00Easy to use, flexible HTTP access method in PythonPython comes with 'batteries included' as they like to say. This means that libraries for most tasks you need to tackle are already part of a standard Python install. In order to issue HTTP requests, the most common libraries mentioned are <span class="Apple-style-span" style="font-family:'courier new';">urllib</span> and <span class="Apple-style-span" style="font-family:'courier new';">urllib2</span>. They offer the very convenient <span class="Apple-style-span" style="font-family:'courier new';">urlopen()</span> function, which make it a snap to quickly retrieve online resources.<div><br /></div><div>One thing that always bothered me, however, was the inflexibility of those methods. Want to use PUT instead of POST? Out of luck. Need authentication? Deal with pesky URLopener classes. Want timeouts for your requests? Use global socket options or use Python 2.6. Want to specify some custom headers? Deal with more classes. And so on.</div><div><br /></div><div>All I want is a simple function like <span class="Apple-style-span" style="font-family:'courier new';">urlopen()</span>, which can do all of these things for me and which also works on Python 2.5 (I'm often using Jython, which still is at 2.5). Of course, there is always the low level <span class="Apple-style-span" style="font-family:'courier new';">httplib</span>, on top of which the two urllibs actually build. And while using that library requires a few more steps, you can use it to write yourself your powerful, simple version of <span class="Apple-style-span" style="font-family:'courier new';">urlopen()</span>.</div><div><br /></div><div>So, for <a href="http://restx.mulesoft.org/">RESTx</a> - an open source project to simply and easily create <a href="http://restx.mulesoft.org/">RESTful web services</a> - I wrapped all the functionality I wanted into a convenient, compact function that uses <span class="Apple-style-span" style="font-family:'courier new';">httplib</span> directly. Currently, it only supports HTTP's basic authentication, but hopefully that can be extended at some point in the future.</div><div><br /></div><div>Here it is then for your enjoyment. Let me know if you think this is useful. Oh, and while you are here, please <a href="http://twitter.com/BrendelConsult">follow me on Twitter</a>.</div><div><br /></div><div><b><i>Update:<span class="Apple-style-span" style="font-weight: normal; font-style: normal;"> Several people pointed out httplib2 to me, which offers a lot of the convenience I was looking for. However, it is not available in Jython, or Python 2.5 in general.</span><br /></i></b><pre><br />import httplib<br />import urllib<br />import socket<br />import urlparse<br /><br />def http_access(method, url, data=None, headers=None, timeout=None, credentials=None):<br />"""<br />Access an HTTP resource with GET or POST.<br /><br />@param method: The method for the HTTP request: GET, POST, etc.<br />@type method: string<br /><br />@param url: The URL to access.<br />@type url: string<br /><br />@param data: If present it specifies the data for a POST or PUT request.<br />@type data: Data to be sent or None.<br /><br />@param headers: A dictionary of additional HTTP request headers or None.<br />@type headers: dict<br /><br />@param timeout: Timeout for the request in seconds, or None. Specified as<br /> floating point value, so you can set sub-second timeouts.<br />@type timeout: float<br /><br />@param credentials: A username/password tuple to support basic HTTP authentication.<br />@type credentials: tuple<br /><br />@return: Code and response data tuple.<br />@rtype: tuple<br /><br />"""<br />(scheme, host_port, path, params, query, fragment) = urlparse.urlparse(url)<br />allpath = url[url.index(host_port)+len(host_port):]<br />host, port = urllib.splitport(host_port)<br /><br />if not headers:<br /> headers = dict()<br /><br />if credentials:<br /> accountname, password = credentials<br /> headers["Authorization"] = "Basic " + base64.encodestring('%s:%s' % (accountname, password))[:-1]<br /><br />if scheme == 'https':<br /> conn = httplib.HTTPSConnection(host, port)<br />else:<br /> conn = httplib.HTTPConnection(host, port)<br /><br />conn.request(method, allpath, data, headers)<br />conn.sock.settimeout(timeout)<br /><br />try:<br /> resp = conn.getresponse()<br /> code = resp.status<br /> data = resp.read()<br />except socket.timeout, e:<br /> return 408, httplib.responses[408]<br /><br />return code, data<br /><br /><br /><br />if __name__ == '__main__':<br /># Some usage example: Getting data in JSON format from a ficticious server with<br /># a timeout of 0.5 seconds.<br />status, data = http_access("GET", "http://localhost:8001", data = None,<br /> headers = { "Accept" : "application/json" }, timeout=0.5 )<br /><br />print "@@@ HTTP response code: ", status<br />print "@@@ Received data: ", data<br /><span class="Apple-style-span" style="font-family:Georgia, serif;font-size:130%;"><span class="Apple-style-span" style=" white-space: normal;font-size:16px;"><span class="Apple-style-span" style="font-family:monospace;font-size:100%;"><span class="Apple-style-span" style=" white-space: pre;font-size:13px;"><br /></span></span></span></span></pre></div><div><br /></div>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com3tag:blogger.com,1999:blog-6748806271290317100.post-65231515053600951072010-07-01T05:44:00.005+12:002010-08-05T11:17:15.361+12:00A new, simpler way to create RESTful resources<div>Lately, I have been working on a new project called <a href="http://restx.mulesoft.org/">RESTx</a>. A new, simpler way to do data integration and data publishing. Fully open source and licensed under GPLv3, fully RESTful, super simple to install and use. The download is small - just over 200k - and installation is deliberately quick and straight forward; a single command sets it all up for you. So I would like to invite you to have a look and check it out. I'd love to get your feedback and opinion on it.</div><div><br /></div><div><b>What is it about?</b></div><div>RESTx gives developers the ability to quickly write custom data access and integration logic as <i>components</i>. Writing those components is very simple, the API is compact to the point where you could probably explain it in just 5 minutes. At the same time it gives you all the freedom you'd get in a custom program or script. You can even chose the language for writing components (<a href="http://restx.mulesoft.org/documented-java-component-example">Java</a> and <a href="http://restx.mulesoft.org/documented-python-component-example">Python</a> are currently supported, more to come).</div><div><br /></div><div>Any component parameters are exposed, so that users can post new parameter sets to the component in order to create a new <i><a href="http://restx.mulesoft.org/about-restful-resources">RESTful resource</a><span class="Apple-style-span" style="font-style: normal;"> or <a href="http://restx.mulesoft.org">RESTful web service</a> in just seconds, simply by filling out a form in a browser.</span></i> The RESTful resources are then accessed via a simple URL, suitable for access by end users in a browser, sharing, building bocks for mashups, and so on. The ability to create RESTful resources without any coding allows users to quickly build their own data sources without having to wait for IT to provide them.</div><div><br /></div><div>There are a few key ideas behind RESTx, and they mostly center around being nice and <a href="http://restx.mulesoft.org/need-simplicity">simple</a>:</div><div><br /></div><div><b>Be nice to developers</b></div><div><b></b>Why fiddle with 300 lines of obscure XML if you can express the same idea in just 3 lines of concise, easy to read code? This is one of the core concepts for developers: Convention over configuration and sane defaults. If you need custom integration or data access logic, it will be much quicker for you to create a new component and express this concisely in a language you know, rather than work your way through yet another XML dialect.</div><div><br /></div><div>For component development, RESTx currently supports <a href="http://restx.mulesoft.org/documented-java-component-example">Java</a> and <a href="http://restx.mulesoft.org/documented-python-component-example">Python</a>, more languages to be added soon. Writing custom data integration and access code is absolutely simple. The entire API can easily be explained in 5 minutes. If you think proper frameworks are too heavy and think you're faster just with a quick adhoc hack... don't! RESTx is simpler and quicker than adhoc hacking, but gives you all the advantages of a proper platform.</div><div><br /></div><div><b>Be nice to users</b></div><div>Allow users to create their own RESTful data resource without coding by sending a new set of parameters to components on the RESTx server. That turns into an easy to share and use URL, totally hiding the component and parameters.</div><div><br /></div><div>Users can discover all the available components (for which they can provide parameters to create a resource) and all existing resources (which they can use), simply by following links. Components and resources explain themselves: Each of them has human readable documentation strings and all their parameters and provided services can be discovered.</div><div><br /></div><div><b>Be nice to search engines, web browsers and client applications</b></div><div>Everything on the server (components and resources) can be discovered just by following links. You get human AND machine readable descriptions, which is great for discovery by search engines.</div><div><br /></div><div>RESTx automatically represents data in a way that matches the client request. For example, if you access a resource via a web browser, you get the information in HTML. If you access it with another client application, you can request to see the very same data in JSON, for example. In fact, all interactions with the server take place via a simple RESTful API, which allows the usage as well as creation of resources via simple, JSON encoded requests.</div><div><br /></div><div><br /></div><div>A few links you might be interested in:</div><div><ul><li><a href="http://blogs.mulesoft.org/introducing-restx-a-new-simpler-way-to-integrate-and-publish-data/">My blog post, announcing the RESTx release</a> with a bit more information.</li><li>A <a href="http://restx.mulesoft.org/quick-start-guide">quick start guide</a>, if you want to be up and running in 5 minutes.</li><li>A few words about <a href="http://restx.mulesoft.org/about-restful-resources">RESTful resources</a>.</li></ul></div><div>Please enjoy! Hopefully, you will find this project useful. I am looking forward to your feedback.</div>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-68400395638185924752010-03-30T13:25:00.003+13:002010-03-30T14:02:22.598+13:00Mixed Jython/Python development and Google AppEngine odditiesRecently, I worked on a Python project, which I developed as a stand-alone WSGI application. I also experimented with running this app in <a href="http://www.jython.org/">Jython</a>. In case you don't know: Jython is a neat Python implementation, which runs in the Java VM. The advantage of this is that you have access to all the usual Java infrastructure, framework and libraries, while being able to rapidly develop your program in the Python language. I ended up designing my application so that it could run both as Python as well as Jython project.<div><br /></div><div>Shortly afterwards, I moved my WSGI application to Google's <a href="http://code.google.com/appengine/">AppEngine</a>, which is is a hosted environment for Python and Java apps. AppEngine can run pure WSGI applications quite easily, so this shouldn't be too much of a problem. Using Google's local development server (<i>dev_appserver.py</i>), I was able to run my application on my desktop in the emulated Google environment without problem.</div><div><br /></div><div>I then used the provided command line utility (<i>appcfg.py</i>) to move my app to Google's actual servers... and nothing worked. I received a "500 Internal Server Error". Looking at the logs I found that my application was not able to import any of my own modules. While standard library items were imported without an issue, any module I had provided myself resulted in an import error.</div><div><br /></div><div>What had happened? It took me a long time to figure this out, so I thought I share it with you: When running the Python application via Jython, a bunch of <i>.class</i> files were created, which is normal. However, when the application was uploaded via <i>appcfg.py</i>, those class files were uploaded as well (files ending in <i>*.pyc</i> and a few other extensions are ignored, but apparently anything else is considered to be part of your application).</div><div><br /></div><div>So then, for reasons that aren't quite clear, the presence of <i>.class</i> files in your Python application seems to be really confusing for Google AppEngine.</div><div><br /></div><div>The lesson? If you are developing an application that can run locally in Jython as well as on Google's AppEngine, make sure to clear out your <i>*.class</i> files before uploading. It can save you hours of grief.</div><div><br /></div><div><br /></div><div><div>Oh, and by the way, you should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.</div><div><br /></div></div><div><br /></div><div><br /></div>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-3071803602946801302010-03-24T11:54:00.001+13:002010-03-24T14:03:28.895+13:00This blog has moved<br /> This blog is now located at http://blog.brendel.com/.<br /> You will be automatically redirected in 30 seconds, or you may click <a href='http://blog.brendel.com/'>here</a>.<br /><br /> For feed subscribers, please update your feed subscriptions to<br /> http://blog.brendel.com/feeds/posts/default.<br /> jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-81023622707336426432009-09-15T10:26:00.010+12:002009-10-28T05:18:12.320+13:00Telecommuting for software developersI telecommute a lot, and it's a pretty extreme case of telecommuting as well: I'm based in New Zealand, while my clients usually are in North America, Europe or Asia. Occasionally I meet some of them in person, but often I never get a chance to shake their hands. In this article here I want to talk a little about how I tend to make telecommuting work for me and my clients.<br /><br /><span style="font-style: italic;">Telecommuting and modern development processes: A conflict?</span><br /><br />While telecommuting has become much more acceptable in the last few years, many prospective clients or employers still shy away from it. Their concerns are understandable: How can someone so far away be successfully integrated into a team? How can that person really participate in the development process, brain-storming or design sessions, code reviews or high-pressure debugging sessions?<br /><br />In fact, there are two opposing trends at work here: On one hand we have more and better technology available for telecommuting. On the other hand, however, the more we talk about iterative and agile processes in software development, the more it is felt that in-office presence is required, since these processes rely on low-latency, high-bandwidth and frequent communication.<br /><br />The question then for every telecommuter and their employer/client is this: How can we best adapt the way we work and utilize available technology to make telecommuting work even with the small-team, interactive and iterative, communication-heavy approach that is often preferred today?<br /><br />Making telecommuting work requires some preparation and a little bit of flexibility from both sides. Undoubtedly, there are a lot of nuances, which you will miss out on, no matter how good the tele-presence technology at your disposal might be. I'm not saying that you can make telecommuting 100% equivalent to on-site presence. However, I do believe that telecommuting can be made to work very efficiently, and usually much more so than most people would guess.<br /><br />In fact, I have telecommuted not only as a developer, but even as the engineering manager of teams, doing everything from architecture and design to the daily ins and outs of project management. It can work, and it can work well if both sides are willing to work together.<br /><br />Here now a list of my favourite tools, methods and infrastructure components, which help a telecommuting software developer to integrate and work well with a team.<br /><br /><span style="font-style: italic;">Almost like being there in person</span><br /><br />Let me address some of the key concerns I hear a lot. For each of these concerns I try to suggest a solution.<br /><br /><ul><li><span style="font-weight: bold;">Concern:</span><span style="font-weight: bold; font-style: italic;"> "</span><span style="font-style: italic;">We have testing/development/documentation resources in our network. A telecommuter doesn't have all of this available and thus cannot be effective.</span>"<br /><br />Access to important development resources, such as build-servers, test-databases, in-house wikis and other services can easily be provided through a VPN (Virtual Private Network). Good open-source VPN software is available, so aside from the initial setup work, cost should not be a concern. OpenVPN is popular and is available in the standard repos for most Linux distributions. It can be downloaded for Windows, or in its source form <a href="http://openvpn.net/index.php/open-source/downloads.html">here</a>.<br /><br />In many cases, shared resourced like this can also be hosted 'in the cloud', or simply as a cheap VPS. Properly secured with SSH keys and logins for each developer it will do just fine. Wikis, test suites, databases, and so on can be hosted there.<br /><br />Very important, of course, is the use of a version control system. There are some VCSs now specifically designed for distributed teams, for example Git. But I have also worked successfully for clients that used SVN, even though our team was distributed. If you are not dealing with projects the size of the Linux kernel and also have a relatively small team, you won't have to convert your project to a new VCS, the old one most likely will do just fine.<br /></li></ul><ul><li><span style="font-weight: bold;">Concern:</span> <span style="font-style: italic;">"In an office, people can just pop their head over the cubicle wall to ask a quick question. With telecommuting there is a lot of communication overhead and long turn-around time."<br /><br /></span>Fortunately, that's why we have instant messaging. A plethora of IM clients is available for free and most everyone these days has an account with at least one of them somewhere. Quickly typing a question is just as effective as 'popping their head over the cubicle wall', and possibly less distracting.<br /><br />In addition there is always the phone and VoIP (see next paragraph for more on that).<br /><br /></li><li><span style="font-weight: bold;">Concern:</span> <span style="font-style: italic;">"With the team in the same office they all see each other on a daily basis. Because of that they have a more personal, closer relationship and form a better team."</span><br /><br />I can't say that presence in the same office always makes for good relationships, but I understand the concern. Luckily, for a few years already, we have a service available to us which solves this problem, again for free. Of course, I'm talking about <span style="font-style: italic;">Skype</span>, which offers voice, chat and video conferencing. Personally, I would much prefer a capable open-source solution, but time and again Skype's painless setup and ease-of-use has proven just too simple and effective to argue with.<br /><br />Whichever route you go: Seeing each others' faces and hearing their voices on a daily basis is not a problem anymore these days. The services are free, run on Windows, Linux and Macs and web-cams are cheap.<br /><br /></li><li><span style="font-weight: bold;">Concern:</span><span style="font-style: italic;"> "We often have brain-storming sessions or our daily scrum standup meeting in the conference room..."<br /><br /></span>The easiest solution is to place a laptop with a web-cam in the conference room. Even something cheap like an Asus EEE PC will do. No, I'm serious! I have worked in teams where we did exactly that. We had a laptop in the conference room and a web-cam. The laptop was useful for occasional presentations anyway, and I was able to 'be there' during discussions and meetings.<br /><br /></li><li><span style="font-weight: bold;">Concern:</span> <span style="font-style: italic;">"How can a code-review meeting be conducted? Don't tell me you can read the code that's projected on the wall when 'being present' via a web-cam from the other side of the room..."<br /><br /></span>There are some really nice tools available for code-reviews in distributed teams. For example <a href="http://codestriker.sourceforge.net/">CodeStriker</a>, which is simple but very effective. It's open source, so again it's free. Sure, that's not a meeting where everyone piles into the same room, but having experienced both types of code review, I can attest to the fact that a code review via CodeStriker is much more pleasant than a sleep-inducing code review meeting with everyone in the same room.<br /><br /></li><li><span style="font-weight: bold;">Concern:</span> <span style="font-style: italic;">"We do a lot of pair-programming / pair-debug-sessions. That's just not possible in a distributed team."<br /><br /></span>Of course it is! There might even be specialized tools available for this. However, the simplest one I have found is straight-forward screen sharing. VNC is the usual, free and open source standard. But there are also commercial screen-sharing services, such as GoToMeeting.com. You start a Skype session or phone call (voice is sufficient here) with each other and then one of the pair shares their screen with the other one. You can even set up VNC so that you can control the screen remotely, allowing you to 'grab the mouse' and point at something you are talking about.<br /><br />A number of free and/or open-source VNC clients and servers are available or are already in the standard repos for most Linux distributions. In Ubuntu, for example, screen-sharing comes pre-installed already. You just chose 'Remote Desktop' from your preferences, or 'Remote Desktop Viewer' from your Internet applications. That's how simple it is these days.<br /><br />Obviously, this is made easier when a VPN has been setup, since otherwise you need to figure out how to get the necessary network traffic through your firewall(s).<br /><br />Even simpler than VNC is just a plain old 'screen' session in a terminal. That works great on any *nix server to which one or more team member can login to. The big advantage of a shared 'screen' session is that it has much lower bandwidth requirements than VNC and thus works better for any shared work that only requires textual information.<br /><br /></li><li><span style="font-weight: bold;">Concern:</span><span style="font-style: italic;"> "We need to be able to collaborate in real time on notes/documents/drawings</span>.<span style="font-style: italic;"> In short: We need a whiteboard!"</span><br /><br />There are several solutions I am using for this. One very effective and simple one is <a href="http://etherpad.com/">EtherPad</a>, which is a straight-forward way for multiple people to simultaneously collaborate on the same document. You actually see the individual letters appear as the other person types, contributions from different people are color coded and you have a time-slider at the top to re-visit the evolution of the document. You can also export the documents as PDF or Word file.<br /><br />For diagrams and drawings, <a href="http://creately.com/">Creately</a> offers similar features. Just like EtherPad, you can get started for free. Of course, there are a number of similar offerings out there, EtherPad and Creately are not the only one.<br /><br />In fact, remember what I said about pair-programming, using VNC? You can take the same approach for diagramming and sketches, allowing you to simultaneously work on the same drawing application (Visio or the drawing functionality in most presentation of word-processing packages).<br /><br /></li><li><span style="font-weight: bold;">Concern:</span><span style="font-style: italic;"> "We use Post-It notes for our Scrum board...</span>"<br /><br />Online Scrum boards are offered by more than one provider. <a href="http://scrumy.com/">Scrumy</a> is one of those services, giving you the 'look and feel' of Post-It notes. Many people still like to represent their Scrum board in spreadsheets or Wikis, though.<br /><br /></li><li><span style="font-weight: bold;">Concern:</span><span style="font-style: italic;"> "All these technical solutions are fragile or take too long to set up when you need them."<br /><br /></span>The best advise I can give here is to prepare ahead of time. When you start your telecommuting workday, make sure nothing has 'fallen over' and you can still connect via the VPN. If you have a fickle sound system on your desktop then make a Skype test call (for example) to confirm that it all still works. Make sure to have a how-to on the company wiki that explains the use of VNC or whichever other screen-sharing service you use and occasionally go with your colleagues through a test session at least. Always send any visuals or supporting documentation out to all meeting attendees sufficiently ahead of the actual meeting<span style="font-style: italic;"><span style="font-style: italic;"></span></span>.<br /><br /></li><li><span style="font-weight: bold;">Concern: </span><span style="font-style: italic;">"You are in a completely different time-zone, so even with all those communication tools, you still won't be able to interact with the team."<br /><br /></span>Addressing this concern requires a bit of flexibility by the telecommuting worker. Most work environments, especially for software development, won't be strictly 9 to 5. Instead, they may have 'core hours' (10 am to 3 pm, for example). It helps when you as the telecommuter can offer the client/employer that you will at least be at your computer during those core hours they have in their office, no matter which time-zone you are in.<br /><br />Case in point: When I work for clients on the US west-coast, I tend to get up around 4:30 am, which is about the time the team there gets into the office in the morning. I also arranged my work week to go from Tuesdays through Saturdays, in order to compensate for the one day that New Zealand is ahead of the US.<br /><br />As a telecommuter we ask for some flexibility from the client, therefore we can offer this much flexibility in return.<br /><br /></li><li><span style="font-weight: bold;">Concern:</span> <span style="font-style: italic;">"When a team is housed in the same office you get valuable informal interactions, watercooler talk, which you just can't get when telecommuting.</span><br /><br />Indeed, this is one of the toughest things to deal with. Short of roaming the corridors with one of <a href="http://www.engadget.com/2009/01/09/anybots-rolls-out-qa-the-telegenic-telepresence-robot/">these</a> (a bit pricey), or <a href="http://www.instructables.com/id/Sparky_DIY_Web_Based_Telepresence_Robot/">these</a> (somewhat cheaper), the best solution appears to be to (a) take full advantage of all the other means of communication, (b) encourage the team to involve the telecommuting worker(s) when the discussion turns to work-related issues and (c) to install a thin laptop/desktop in the kitchen or other common gathering area as well, start Skype and set it to auto-answer, which allows the telecommuter to 'visit' that area whenever they want to.<br /><br />The cheapest and probably most effective solution, however, is to continually keep the team as well as the telecommuting worker in mind. It is easy to 'forget' about the other side at times. Staying in contact can take a little bit of effort if you are engrossed in a programming task, for example. Occasionally, it is good to make a point out of calling someone else, rather than 'just' sending an email or IM, efficient as those solutions may be.<br /></li></ul>So much for my thoughts on this. Hopefully this list can help you or your clients to consider telecommuting as a valid option. If you have any additional ideas or suggestions how to enable a telecommuting software developer, please let me know in the comments.<br /><br />Oh, and by the way, you should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com3tag:blogger.com,1999:blog-6748806271290317100.post-15106752974879450092009-08-07T08:27:00.004+12:002009-08-07T09:07:52.760+12:00Search engine comparison for software developersFor today's software developer it is difficult to imagine that there once was a time where we did not have instant access to information on the Internet to help us do our work: Answers to questions, howto-guides, reference material, documentation... whatever we need, it is usually just a quick search away.<br /><script src="/consulting/js/thumbnailviewer.js" type="text/javascript">/***********************************************<br /> * Image Thumbnail Viewer Script-Dynamic Drive (www.dynamicdrive.com)<br /> * This notice must stay intact for legal use.<br /> * Visit http://www.dynamicdrive.com/ for full source code<br /> ***********************************************/<br /></script><br />I can't count how many times I simply searched for something like "Django ... such and such ..." to instantly find the relevant documentation or reports of other people's experience while working on a Django project, for example. The amount of time this saved me is immeasurable. I have a friend who - as a software developer - is not allowed to access the Internet while at work! The mind boggles. How much does this cost his employer in lost productivity?<br /><br />Anyway, clearly the access to search engines while developing software should be considered essential. If during a job interview I'd ask a candidate something like: "How do you define a C function pointer?" and the answer would be: "I've done it before, but can't recall the exact syntax now... I'd just google it!" ... well, I would consider that to be a perfectly valid answer, to be honest. One more reason why these types of questions are quite meaningless in interviews. But that's a different topic.<br /><br />So, the question then is: What is the best search engine for a software developer to use? Many people default to Google, obviously, but personally, I don't like to let one company have all the information about me, so I like to mix it up a little. The difference in search results can be quite startling, though. With Bing as the new arrival on the scene, let's perform a completely subjective and un-scientific experiment and ask four of the major search engines the same set of questions and see what they tell us.<br /><br />For this test, I have chosen Bing, Google, Yahoo and Ask. Sadly Yahoo search will soon stop to exist when they start using Bing as a search provider, but for sentimental reasons and while it is still available, I thought I include them. Especially since Yahoo was my default search engine for quite some time now.<br /><br />I have chosen four questions, which obviously cover only a tiny fraction of software development related areas:<br /><ol><li><span style="font-style: italic;">"Django IntegerField model"</span>: Here I would like to see links to the Django project documentation on how to define an IntegerField in a model. It should be noted that 'IntegerField' also exists as part of forms, so the search engine should not confuse these two.</li><li><span style="font-style: italic;">"Define C function pointer"</span>: Our 'interview question' from above. The top results should lead to pages that very clearly explain how to define this thing.</li><li><span style="font-style: italic;">"TCP checksum calculation"</span>: I would like to see links to pages that explain how the algorithm works that computes TCP checksums.</li><li><span style="font-style: italic;">".Net class inheritance"</span>: I'm not an MS developer at all, so I'm not even quite sure whether this question makes much sense, but I thought I try this one, just on a hunch.</li></ol>Ok, let's start...<br /><br /><span style="font-weight: bold;">The test</span><br /><br />Question 1. <span style="font-style: italic;">"Django IntegerField model"</span><br /><br /><a href="http://farm3.static.flickr.com/2638/3796322008_6e8a6d0bf3_b.jpg" rel="thumbnail"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 240px; height: 127px;" src="http://farm3.static.flickr.com/2638/3796322008_6e8a6d0bf3_m.jpg" border="0" /></a>Here are the results from all four search engines, side by side. In all images here, you can see Bing in the first column, followed by Google, then Yahoo and finally Ask in the last column.<br /><br /><ul><li><span style="font-style: italic;">Bing:</span> The first results are essentially useless, leading (ironically) to Google code pages, followed by some obscure changeset in the Django code itself. Sure, you can see an IntegerField used there. However, the link to the Django docs, which I was looking for, is only at the very bottom of the page.</li><li><span style="font-style: italic;">Google:</span> The results are highly relevant (only peeve might be that they are pointing to docs for the SVN version, not the latest stable release version, but that's really minor). A link to the older Django version's docs is in third place, which is useful. But as we can see, the ads are not quite as relevant.</li><li><span style="font-style: italic;">Yahoo:</span> The results are not bad. However, I didn't like the fact that it pointed me to docs for a much older version of Django in the top result. They could have done better here.</li><li><span style="font-style: italic;">Ask:</span> The top results are good. However, I am disturbed by those ads, which appear right under the top result. They could have shown them on the side. But on Ask the side space is taken up by "Related Searches", which in this particular case were completely irrelevant.</li></ul>Ranking: Google wins hands down. Ask in second place, closely followed by Yahoo. Bing is dead last.<br /><br />Question 2. <span style="font-style: italic;">"Define C function pointer"</span><br /><br /><a href="http://farm3.static.flickr.com/2577/3795503485_d6fb3d7d4e_b.jpg" rel="thumbnail"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 240px; height: 127px;" src="http://farm3.static.flickr.com/2577/3795503485_d6fb3d7d4e_m.jpg" border="0" /></a>Here are the results for this question.<br /><br /><ul><li><span style="font-style: italic;">Bing</span>: Again disappointing. The top result is about a function pointer validation function... which would be relevant if obviously I didn't need to learn first how to define a function pointer. But since I am looking only for how to define a pointer these results don't help me much. The third one is even about how to call C functions from Forth!?<br /></li><li> <span style="font-style: italic;">Google:</span> This is exactly what I was looking for: A tutorial! Top link, good job Google!</li><li><span style="font-style: italic;">Yahoo:</span> Not bad, with the tutorial in second place. For some reason, the Wikipedia entry comes first. Both Yahoo and Google generally give quite some weight to Wikipedia pages, which I am normally fine with. In this particular case, the tutorials are really what I'm looking for, though.</li><li><span style="font-style: italic;">Ask:</span> Again a very good top result, just like Google, marred by irrelevant annoying ads, followed by other good results (for example how to define a function-pointer parameter for a function, which is very nice).</li></ul>Ranking: Ask wins this one based on the result quality, even though their ads bother me. Google is a very close second, Yahoo in third, but still close. Bing is way off the mark again.<br /><br />Question 3. <span style="font-style: italic;">"TCP checksum calculation"</span><br /><br /><a href="http://farm3.static.flickr.com/2622/3796323244_caff1ae572_b.jpg" rel="thumbnail"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 240px; height: 127px;" src="http://farm3.static.flickr.com/2622/3796323244_caff1ae572_m.jpg" border="0" /></a>The results for this one, side by side.<br /><br /><ul><li><span style="font-style: italic;">Bing:</span> Very disappointing results at the top, leading to small discussion threads and postings here and there. Not at all what I was looking for.</li><li><span style="font-style: italic;">Google:</span> Excellent results, especially the third link ("TCP Checksum code"). That page contains a textual description of the algorithm as well as actual sample code. Perfect. The further links on the page remain relevant as well. The PDF in fourh place is actually quite good.</li><li><span style="font-style: italic;">Yahoo:</span> The top three results are the same as Google, so that's very good. The relevancy of the remaining links trails off quickly, though.</li><li><span style="font-style: italic;">Ask:</span> The top three results are the same as Google and Yahoo. The remaining links remain more relevant than Yahoo, but not quite as good as Google, though.</li></ul>Ranking: Google in first, followed by Ask and Yahoo. Bing again in last place.<br /><br />Question 4. <span style="font-style: italic;">".Net class inheritance"</span><br /><br /><a href="http://farm4.static.flickr.com/3425/3796322718_c25448f9fb_b.jpg" rel="thumbnail"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 240px; height: 127px;" src="http://farm4.static.flickr.com/3425/3796322718_c25448f9fb_m.jpg" border="0" /></a>Here is a question for Microsoft's Bing to shine! It should know all about that, right? Well, here are the results...<br /><br /><ul><li><span style="font-style: italic;">Bing:</span> Oh the humanity! What were they thinking? Again, the top link goes to some forum page, the second one is a blog. That's a recurring and unsuccessful theme with Bing. The third one contains some sample code, but is basically someone making the case for multiple-inheritance, which apparently is missing in .Net.</li><li><span style="font-style: italic;">Google:</span> Hits the mark again with the top link pointing straight to the relevant page of Microsoft's own online documentation. Why couldn't Bing come up with this? And the second link is great, too: A tutorial for class inheritance, which is always nice to see.</li><li><span style="font-style: italic;">Yahoo:</span> Good results in the top two spots, with a useful page about interfaces included, which Google didn't show. But the third result is disappointing: Apparently, Yahoo was confused by the domain name for that result, which ended in ".net".</li><li><span style="font-style: italic;">Ask:</span> The ads are disturbing as usual, but the top results are the same as Google's, which is good. Not visible in this screenshot, because too much valuable space was used by those ads, there is also a good tutorial link further down, which Google didn't show.</li></ul>Ranking: Ask by a slim margin (if we manage to ignore those ads), Google in close second, followed by Yahoo a bit further back. Bing is again at the very end with a smattering of useless results.<br /><br />I sure hope that Microsoft's own developers are free to use Google or Ask at work.<br /><br /><span style="font-weight: bold;">Conclusions</span><br /><br />Let's start at the back.<br /><br />Maybe they are still trying to sort out issues and will improve over time. However, at the moment Bing is just completely hopeless when it comes to these kinds of queries. I guess they are much more consumer oriented. As it stands, though, for questions software developers need to ask for their work, Bing quite plainly ... sucks!<br /><br />Yahoo still tends to be good and delivers useful results. But while the top results are usually ok, the relevance of further results quickly drops. Of course, that will all end once they switch to Bing. At that moment they will get much, much worse. What a sad loss for the Internet to see Yahoo search go.<br /><br />Ask and Google are actually ranking equally well here in this test. That may come as a surprise to many. The biggest annoyance about Ask is the placement of the ads right after the top result. And their "Related Searches" links usually don't provide any value at all for these types of searches. But the quality of the links is about as good as Google. The ads on both Ask and Google are close to irrelevant in both cases.<br /><br />A message to Ask: Please change the placement of your ads or at least distinguish them visually a bit more!<br /><br />So, take your pick: Google or Ask. I will start using Ask, because as I said earlier, Google is on enough pages already by means of their ad network and they don't need to know everything about me. And with Yahoo ranking behind Ask and soon completely fading away, Ask remains the only real alternative for search results that are relevant to software developers.<br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com4tag:blogger.com,1999:blog-6748806271290317100.post-87984686993394227162009-07-29T06:26:00.008+12:002009-07-30T06:50:18.921+12:00Can Agile work for a team of experts?My last article about <a href="http://www.brendel.com/consulting/blog/2009/07/considering-agile-deciding-against-it.html">considering Agile</a> produced a lot of interesting feedback. Most of all, it prompted me to do a lot more research into Agile software development and specifically <a href="http://en.wikipedia.org/wiki/Scrum_%28development%29">Scrum</a> as a project management framework.<br /><br />As has been mentioned before, many of these Agile ideas are just common sense. This is often <a href="http://lukehalliwell.wordpress.com/2008/11/16/the-agile-disease/">pointed out</a> by its critics, but is that really a bad thing? Since common sense is not as common as we would like it to be, Agile frameworks and methodologies codify their practices in rather specific rules on how to do things. Basically, these rules are designed to help (force?) a team to adopt a lot of common sense practices, which in traditional software engineering may have been forgotten. I have to say, I feel like I actually 'get' many of them and I might talk about some specifics here in future blog entries.<br /><br />In the meantime, though, I'm still trying to wrap my head around one particular issue: Agile seems to work best - in fact it requires - that there are no 'silos' in the engineering organization: Particular knowledge should not be locked up in the head of one developer or a small group of them. Instead, knowledge should be shared across the team and some Agile practices, such as <a href="http://en.wikipedia.org/wiki/Pair_programming">pair programming</a>, for example, are specifically designed to address this. But in same cases those silos are unavoidable. Read on for an example of that. And in the moment you have knowledge silos in your team, many of the benefits of Agile seem to just evaporate.<br /><br />Let me explain what I mean. For truly Agile development it must be possible to assign user stories and their related tasks to almost any team member. At least, there should be more than one team member who's able to take on a specific task.<br /><br />If you have a team like this it allows a number of interesting things to happen:<br /><ol><li> Once a developer is done with a particular task they can merely grab the next highest priority item from the backlog. This makes it possible for the team to really deliver higher-priority items in order, thus producing most value to the product owner in each sprint.</li><li> You can use <a href="http://en.wikipedia.org/wiki/Story_points">story point</a> estimating, maybe utilizing surprising techniques such as <a href="http://en.wikipedia.org/wiki/Planning_poker">planning poker</a>, and it will actually make sense: Over the course of several sprints you get a feeling for the velocity of the team and even each individual developer. The nice thing about estimating in story points is that it is self-tuning and doesn't depend on the experience level of the individual developer. After a while you will know that John - a senior developer - can do about 20 story points per sprint, while Bob - fresh from university - may be able to do only 10. So, you can re-assign user stories without having to re-estimate them - well, you need to re-esitmate the associated tasks, but that is easier with these metrics - and still quickly get a good idea of how your team will be able to do in the sprint.</li><li> Every developer can just jump in and help others in the team. That is of course a given and is helpful in every kind of team, Agile or not. If a staff member goes on vacation or is sick the work can be picked up by whoever is available (see also point 1). Likewise, if a developer is held up with a previous task for longer than expected then the next task that he was supposed to tackle doesn't have to wait. If someone else is free or doesn't have a higher priority item to work on, that is.</li></ol>Imagine you are in a team that develops web-applications, using Django or RoR or some such framework. There is a large number of user stories, all of them describing the various features that need to be implemented. All of your developers will have to work in pretty much the same environment and under the same constraints provided by the framework. In a situation like this, I can imagine Agile to work very well, since all developers essentially need to have the same abilities for the job anyway: Know the language, know the framework, know the infrastructure you use for developing and testing. That covers a lot of ground and establishes large areas of commonalities between developers. Everyone is much more likely to be able to take on any specific task during the sprint.<br /><br />But now imagine a different team in a different company and suddenly we encounter trouble: Assume you are in a small startup, which doesn't just devlop a web-app, or work with a single framework or environment. Assume you are working on a project that hacks the Linux kernel for some innovative network monitoring solution, produces a lot of data that needs to be sliced and diced and requires a web-app to drive an interactive front end. You are a small startup with some angel investment, so you get yourself the smallest team you need to get the job done: An experienced networking/kernel hacker, a database expert, someone who knows Django or TurboGears (or a similar web-app framework), and a user-interface person taking care of the HTML/JS/CSS for the front-end.<br /><br />So, that's your team. Now imagine a planning poker session for a specific feature: <span style="font-style: italic;">"Ok, on the count of three: What's your estimate for hacking the ipchains kernel module to add feature XYZ?"</span> Guess who is going to put down a card with a number and who is going to put down a card with a question mark (meaning: <span style="font-style: italic;">"Don't know..."</span>)? Well, the kernel person may have a decent estimate, while everyone else - those who never hacked around in the kernel before - realistically will have no idea how complex that task is! You have just one person on the team who can do the job and can provide estimates. You may just as well estimate in hours then, and you will be back at much more traditional planning. You need to define exactly ahead of time who will do which tasks, since each person is so specialized that in effect they are the only ones being able to estimate and do it. Say hello to your Gantt charts.<br /><br />I think this is an example where some of the best aspects of Agile seem to fall flat.<br /><br />Maybe you will say: Well, you shouldn't have teams like this anyway, you should pair-program to spread the knowledge, you should assemble your team so that you don't have this formation of knowledge silos.<br /><br />To those arguments I will have to say: VCs don't have a lot of patience and they don't hand you a lot of money to start with. You can only get a small team, and you can't turn an HTML/CSS/JS person into a Linux kernel expert just like that. Especially not when you deal with a very tight deadline and budget. You can try, but then who's doing the front end work, while you are sending that person through the Linux kernel 'apprenticeship'? No, you need subject matter experts that are as efficient as they can be and can crank out a piece of functionality with the least amount of fuss and delay. And all you can afford at that stage is one expert for each subject area.<br /><br />If you have a team of experts, each expertly knowledgeabe in their own area with very little overlap: Does Agile still make sense?<br /><br />As I said at the beginning, I can see a lot of value in many of the Agile practices and ideas. Yes, it's basically common sense, but I understand how by codifying it you can bring a lot of this common sense back into engineering organizations. But while I have learned to appreciate a lot about Agile since my [previous post], I have to stand by the assertion I made then: Agile works in some teams and environments, but there are also some cases where it simply makes much less sense. Some of its aspects might still be useful, but under some circumstances it loses a lot of value to the point where you have to ask yourself: Why bother with Agile at all then?<br /><br />Agile seems to make sense only once your team can grow in size - or alternatively in the spread of knowledge across the team - to a point where most tasks could be effectively tackled by more than just one person on your team. If your work team consists of generalists, and your user stories and tasks are suitable to be tackled by those generalists. <span style="font-style: italic;">Note, with 'generalist' here I mean someone who is able to deal with all the aspects of your project. I'm not advocating that developers shouldn't have speciality areas.</span><br /><br />This is just my impression. If you have experience working in Agile teams that just consist of a small group of specific and distinct subject area experts, please comment and let me know how you handled this problem. I would love to know.<br /><br />In the meantime, I can imagine that non-homogeneous teams, with too many knowledge silos, may be the reason for many failed Agile projects: The team thinks it's agile, but when push comes to shove, they realize that actually they have been hand-cuffed by the insufficiently spread knowledge in the team. After a few sprints the situation may improve, but for starters?<br /><br />Agile is about common sense, and there seems to be a very important common sense item that needs to be considered: Does Agile even make sense for my team and project? The answer is not always yes, no matter what Agile-enthusiasts may say. In reality (clue common sense) the answer shouldn't come as a surprise: It depends!<br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com10tag:blogger.com,1999:blog-6748806271290317100.post-47097228553499790012009-07-16T09:21:00.005+12:002009-07-29T07:13:13.327+12:00An entire web site in a single page<div style="text-align: right;"><span style="font-style: italic;"><span style="font-weight: bold;">Update:</span></span> I received nice feedback from some readers, explaining<br />why this one-page design is not very good for SEO. Thank you<br />very much! I appreciate your feedback. As a consequence of this,<br />I have now moved my site to a traditional multi-page design.<br />If you are still interested in seeing the original single-page<br />site, you can <a href="http://www.brendel.com/consulting/single_page.html">go here</a>.<br /><br /></div>When I first put up <a target="_blank" href="http://www.brendel.com/consulting/">my web site</a>, it was all just a single, static page. Basically a small blurb about me, the services I offer, and so on. It looked kind of neat, with a very simple design, which I like. And with a small enough font and big enough screen, it really fit all into your browser without scrolling.<br /><br />But then a friend told me that it doesn't really look professional enough with just a single page. People expect to see at least several pages on a 'real' web-site. So, something had to be done.<br /><br />Now, I'm not a professional web-designer who can photoshop together wonderful logos and backgrounds and also work a site based on the best user/computer interaction strategies. I mostly just wanted to retain the very simple and straight forward design of my site. And it should load fast. And it should be friendly to search engines. And it should still look good in text based browsers, or browsers that don't have JavaScript enabled.<br /><br />When you look at <a target="_blank" href="http://www.brendel.com/consulting/">my site</a> now, you see that it does use multiple pages. But then... not really. Your browser will actually just load a single HTML page, which contains the entire text of the site. The navigation links merely trigger some changes in the CSS for various text sections, so that different content is displayed when those links are clicked.<br /><br />Using the 'display' style property for hiding elements on a page is nothing new and is used on many sites. It's just that normally you see it used to show or hide specific elements within a page, while site navigation itself is normally left to the real thing: Actual separate pages.<br /><br />So, there are probably other sites which do it this way, but it was the first time for me that I tried this approach with a site, and I like how it turned out. That's why I'm sharing this.<br /><br />If you look at the site without JavaScript enabled, you see all the content in the page, one 'page' after the other. The navigation links on the left still work. I'm thinking I might add a link back to the top enclosed in <noscript> under each section at some point.<br /><br />One of the most visible benefits is that site navigation is so fast, it feels instantaneous. Nothing needs to be loaded when a navigation link is clicked. But that is not the only advantage...<br /><br />If you look at the page in a text-based browser such as Lynx, you see that all the relevant information is there, easily readable on a single page.<br /><br />Best of all, the 'crap-to-content' ratio for the page is very good. I you look at the HTML source, you see that there is a lot of relevant textual content, and very little markup to distract from it. I hope Google gives me brownie points for that one...<br /><br />In fact, I would be very interested to hear what those of you with more SEO experience have to say about the site. Does this approach actually help with search engines, or would it make it worse?<br /><br />Ok, so there is one issue I can see with this approach: Tracking of visitors is made a bit more complex. Maybe I can use a small bit of JavaScript to re-load the image of the tracker that I use when a user clicks one of the navigation links? Again, if you have some feedback there, I would really appreciate it.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.<span style="text-decoration: underline;"></span>jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com1tag:blogger.com,1999:blog-6748806271290317100.post-35453835489393018332009-07-11T16:38:00.016+12:002009-07-16T08:58:48.956+12:00Considering Agile, deciding against itSome time ago, while managing a distributed software project, I spoke to a developer who was thinking about joining our team. This individual was a big proponent of Agile development. When he found out that in our particular project we were not using an Agile development model, he began to lose interest. He compared my position there to the position of the Tom Smykowski character from the movie <a href="http://www.imdb.com/title/tt0151804/">Office Space</a>. When I recently then came across <a href="http://jjinux.blogspot.com/2009/07/agile-programming-im-stuck-in-middle.html">this blog posting</a> by Shannon JJ Behrens, I had to remember that particular incident...<br /><script src="/consulting/js/thumbnailviewer.js" type="text/javascript">/***********************************************<br /> * Image Thumbnail Viewer Script-Dynamic Drive (www.dynamicdrive.com)<br /> * This notice must stay intact for legal use.<br /> * Visit http://www.dynamicdrive.com/ for full source code<br /> ***********************************************/<br /></script><br /><img style="margin: 5px 5px 5px 0px; float: left; width: 211px; height: 240px; border: none;" src="http://farm3.static.flickr.com/2572/3708420949_8c487f2d5c_m.jpg" border="0" />In the movie Office Space, Tom is an older employee at Initech, whose job it is to “take requirement documents from the customers and bring them to the developers”. Not surprisingly, he is being characterized as 'useless' and let go. Similarly, in Agile development, direct communication between customers and developers is seen as the way to ensure that the deliverables align with customer requirements. No middle man is needed. Instead of extensive up-front design, requirements are captured in short and concise user stories.<br /><br />I have to admit, I was somewhat taken aback by the comparison to the Smykowski caricature from the movie, but this episode certainly made me think about why we had chosen a non-Agile style for our project.<br /><br /><a href="http://farm3.static.flickr.com/2491/3708341443_b3c0625872_o.jpg" rel="thumbnail"><img style="margin: 0pt 0pt 10px 10px; float: right; cursor: pointer; width: 211px; height: 240px;" src="http://farm3.static.flickr.com/2491/3708341443_f3f12d1ba3_m.jpg" border="0" /></a>Let's try to answer this. Take a look at the graphic here (I don't remember where I saw it first, but it can be found on many different sites, click to see the larger version):<br /><br />What do we not see anywhere? Design and architecture. The definition of the 'backlog' and the 'release plan' is probably the closest we get, but even that is feature focused. And in Agile, features always seem to be driven by customers.<br /><br />For me it boils down to this: I believe that Agile is applicable in projects where not only a clear vision of the project has been communicated to the developers, but also where certain conventions and structures have already been established and where we are dealing mostly with explicit requirements, rather than implicit ones. Projects, where the 'how' is already known, and we are just dealing with the 'what'.<br /><br />For example, to achieve a consistent UI you require some up-front planning, possibly a <span style="font-style:italic;">UI czar</span>: One person - or at most a small number of people - who establish and downright dictate what the UI shall look and feel like. It may stifle the creativity of the developer, but it will ensure a much more consistent look for the UI. Imagine you have user stories describing the requirements for various dialogs in the application's interface. Different developers take on the implementation of those user stories. Unless there is some established, over-arching strategy and - dare I say it? - upfront design of the UI then the results are likely to appear disjointed or <a href="http://elasticprocess.com/content/clumsy-manifesto">piecemeal</a>.<br /><br />Another example is error handling in the application. I remember one of my software engineering instructors a long time ago, who insisted that a project should have an <span style="font-style:italic;">error czar</span>: Someone who establishes the standard by which errors should be handled within the code and who then diligently ensures that all new code complies with this standard. If this is not done and established ahead of time, how are the different developers supposed to produce consistent code that knows how to adhere to those standards and handles errors in a unified manner?<br /><br />It appears to me that if you are developing an application for an existing framework then many of these up-front 'infrastructure' decisions have been made for you already. Consequently, you can focus more readily on explicit customer requirements and Agile is a much more worthwhile development strategy.<br /><br />For example, if you develop a web-app in a framework such as Django or Rails then there are specific ways in which your database is going to be laid out. You also have specific conventions for how errors are handled, and so on and so forth. Thus, the design and architecture of much of your application is pre-determined by your choice of framework. You don't have to make those decisions any more. And later then, the more of the application has been written, the more such underlying decisions will have been made already. Agile works well in those situations. Since big-picture architectural decisions don't have to be made any more, we can focus on the addition of small features (or larger features broken down into many smaller ones), while all developers benefit from the established infrastructure of the project. The 'grab-bag' approach with focus on the individual user story becomes feasible.<br /><br />However, if a vision for the project still has to be conjured up and the basic architecture has to be created and the UI look and feel needs to be designed and error handling or other standards have to be established... well, then Agile does not seem to be ideally suited for that. I have seen design and architecture by committee and design by chaos (everyone going off to implement things before anything was established). Neither one is a pretty sight. The best results are achieved if the architectural direction and basic design can be established by one or just a few, good individuals. Up-front. Well, somewhere after establishing the vision and collecting initial requirements, of course.<br /><br />Realistically, I'm sure that most Agile projects will include some sort of up-front design and architecture, whether the participants call it that or not. Heck, even questions like: “Should we use Java or Python for this?” need to be answered ahead of time and are part of these implied up-front activities.<br /><br />That particular project for which the developer from the beginning of this article interviewed was not at the stage yet where an Agile approach really would have worked. There were still too many unknowns at that time, which required up-front consideration. Thus, our project wasn't run as an Agile project. There is a time and place for everything, and our project was not at a time or place for Agile, yet.<br /><br />However, projects can be transitioned from a more conventional approach to an Agile approach over time. The more we know, the more suitable Agile becomes. The less architecture and design decisions have to be made the better for Agile. But at the start of a brand new project, without anything established, some quite conventional up-front design and architecture is needed.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com7tag:blogger.com,1999:blog-6748806271290317100.post-7139953610566282702009-07-09T08:06:00.003+12:002009-07-16T09:01:26.084+12:00Google's Chrome OS is bad news for Linux<a href="http://googleblog.blogspot.com/2009/07/introducing-google-chrome-os.html">Google's Chrome OS</a> has been announced today and - like any announcement from Google - is already widely being discussed. People usually focus on the supposed threat that this 'new' OS represents to Microsoft's dominance on the desktop. The commentary focuses on this <span style="font-style: italic;">Google vs. Microsoft</span> aspect of the story. However, I think the news is actually worse for Linux than it is for Microsoft.<br /><br />How can I say this, considering that Chrome is be based on a Linux kernel? Let me explain, with a 'historical' analogy.<br /><br />In the 80s and 90s we had the so-called <span style="font-style: italic;">Unix Wars</span>: Prominent Unix vendors like Sun, Digital, IBM, AT&T and many others were battling for supremacy in the server market. Guess who emerged victorious from those Unix Wars?<br /><br />Microsoft's Windows NT.<br /><br />Customers were sufficiently fed-up with the fragmentation and confusion in the Unix server market and finally were receptive to a message of simplicity: Your desktops already run Microsoft software, so why not your servers? It will all play nicely together ... just deal with a single vendor ... yada yada.<br /><br />Now fast forward 15 to 20 years. While Unix is holding on to various diminishing pockets in the server market and Windows server-products have a stronghold in many enterprises, Linux has emerged as a very viable and popular choice for the server. On the desktop, however, Microsoft rules supreme now as it did then.<br /><br />Back in the Unix Wars, Microsoft used the fragmentation amongst the competitors to become a dominant force in the market. Today, it enjoys the advantage that the fragmentation of potential opposition in the desktop market is handing to it. Nowhere is the fragmentation of Linux more apparent than on the desktop.<br /><br />Ubuntu managed to become a strong player in the Linux desktop market, and for once it appeared as if things were finally coalescing. Ubuntu's popularity started to feed itself: Any issues or problems? Chances are someone solved it for Ubuntu already, a quick search on the Internet reveals answers to most common issues. Therefore, while Ubuntu might not be perfect for everyone or might have a few annoyances, for the most part you just don't go wrong settling on Ubuntu: So many others are running it, momentum has been forming.<br /><br />Now Google comes along with Chrome OS. We now have another major player entering the Linux desktop OS market. Linux needs this like we need another hole in the head. Microsoft will read this news with glee: Through the opponent's fragmentation it managed to win the Unix wars. Now Linux's continued fragmentation will ensure that there isn't even going to be a war for the desktop.<br /><br />So, I wish for Google Chome OS to fail. We don't need it. It doesn't help Linux. As suggested <a href="http://www.zdnet.com.au/insight/software/soa/No-thanks-Google-we-ve-got-Ubuntu/0,139023769,339297306,00.htm">here</a>, if Google wants an improved non-Microsoft desktop OS then please throw your considerable resources behind improvements in the existing Linux desktop distros. We will all be better off for it.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-45576718464242753642009-07-04T12:09:00.008+12:002009-07-16T09:04:32.152+12:00Setting the initial value for Django's ModelChoiceFieldRecently, I worked with a Django form that utilized the ModelChoiceField. That is a convenient field that normally is rendered as an HTML select tag, which usually appears as a drop-down menu on a web-page.<br /><br />A ModelChoiceField is specified as:<pre><code> class MyForm(forms.Form):<br /> my_field = forms.ModelChoiceField(queryset = MyModel.objects.all())</code></pre><br />As you can see, this field is used to easily specify a drop-down for all items in a table (or whatever the queryset specifies). The model is represented as the output of its __unicode__() function. If you evaluate the form after it was posted, the value for the field is going to be an instance for the actual model whose unicode representation was selected.<br /><br />The problems for me started when I tried to set an initial value for the field: In an 'edit' form, I wanted all fields to reflect whatever had been saved for a particular model instance, of course. As I said above, if you evaluate the form after it has been submitted you get an actual model instance. Naturally, you would think that initial values would be set in a symmetric manner, by specifying a model instance:<pre><code> form = MyForm(initial = { 'my_field' : some_model_instance })</code></pre><br />Sadly, this doesn't work. And try as I might, I couldn't find an answer to this on the Internet either (more on that in a moment). So, after looking at the Django code, it finally dawned on me that you need to specify the ID of the model instance as the initial value:<pre><code> form = MyForm(initial = { 'my_field' : some_model_instance.id })</code></pre><br />That works now. It's a bit unfortunate that the retrieval value (a model instance) and the initial value (the ID of a model instance) are of a different type. It's an inconsistency in the Django API, I think. But in the end, I probably should have at least tried that one a bit sooner.<br /><br />The surprising thing is that I couldn't find any discussion of this anywhere on the Internet. I should mention here that I am using Yahoo as my default search engine. Shortly after I finally found the solution, it occured to me to try Google. And wouldn't you know it? Right there, third hit from the top, I had the answer.<br /><br />Why did Yahoo not give me this result? Well, the answer was discussed on Google Groups. Is Yahoo not indexing those? Or is Google not letting them index it?<br /><br />Either way, that small issue between Yahoo and Google cost me a few hours of frustration. So, I'm posting the solution here on a non-Google page, so that Yahoo users may also find the answer to that problem in the future.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com3tag:blogger.com,1999:blog-6748806271290317100.post-8408289182553121812009-06-26T08:29:00.010+12:002009-07-16T09:05:29.233+12:00Pitfalls of lazy evaluation in Django's ORMIt's well documented that query sets in Django's ORM are <a href="http://docs.djangoproject.com/en/dev/topics/db/queries/#id3">lazily</a> evaluated. The database is not accessed until you actually reference query set results. For example:<code><pre> my_set = MyModel.objects.all() # No database access<br /> print my_set[0] # DB accessed and model instance created<br /></pre></code>The thing one needs to be aware of here is that any access to the result array will always cause a new database access and object creation for model instances:<code><pre> my_set = MyModel.objects.all() # No database access<br /> print my_set[0] # DB access and model instantiation...<br /> print my_set[0] # ... and the same again!<br /></pre></code>It is not uncommon to retrieve a set of results via a query set and then perform some operations on them, for example in a loop. It is tempting to view query sets just as if they were arrays - lists of in-memory objects - and use them in a similar manner. In-memory arrays tend to be quite efficient, query sets not so much: If you have to run over such a loop several times, you will end up retrieving the same object from the database multiple times as well.<br /><br />If you need to iterate over the result set more than once - or just access the same element a few times - it is therefore more efficient to make copies of your individual results. That way, the query and the creation of model instance objects is done only once:<code><pre> my_set = [ m for m in MyModel.objects.all() ] # DB accessed for all<br /> print my_set[0] # No DB access!<br /></pre></code>In effect, the list you created in the last code snippet becomes a cache for your query results. You could also have just assigned a single result-set element to a variable for the same effect.<br /><br />Fortunately, when passing a query set to a template it appears as if this caching is taken care of automatically for us. Making copies of query set results therefore is important when more complex operations are necessary on result sets within the view function.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-11691381204848602742009-06-26T06:50:00.002+12:002009-06-26T06:52:55.770+12:00Now on Twitter as wellSo, I thought I try my hand in this thing that everyone has been talking about and got myself a Twitter account. You can now <a href="http://twitter.com/BrendelCons">follow me there</a>, if you are so inclined.<br /><br />I'm still undecided on whether this is going to be enlightening or merely another efficient way of wasting time, but we will see.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-25623074695744296382009-06-14T20:00:00.012+12:002009-07-30T06:49:22.422+12:00Read-optimize your source codeWhen you design a software system, database or data structures, you take into consideration its most common use case. For example, you organize your data differently when it is read frequently, but only rarely written to. You <span style="font-style: italic;">read optimize</span> your data.<br /><br />I am convinced that the same applies to source code. In almost all cases, source code will be read much more often than it is written. Who writes the source code? You. A function is developed over some limited time, and once it is done it changes rarely, unless being refactored or modified to accommodate some changed requirements.<br /><br />But who has to read your source code?<br /><br /><ul><li>Well, for starters, your colleagues who have to integrate with or use your code.<br /></li><li>New hires who join your team and try to find their way around.</li><li>The maintenance (or continued engineering) programmers.</li><li>Those who come after you, or inherit your code as part of their responsibility</li><li>You.<br /></li></ul>Your code is read around the time it is written and integrated and possibly for many, many years after you have long forgotten about your code.<br /><br />We can see that over the life-time of some piece of code, it is very likely to be read much more often then written or modified. Consequently, source code needs to be <span style="font-style: italic;">read optimized</span> just like we might read optimize a data structure or database if called for.<br /><br />What does it take to read optimize source code? Here are the key points:<br /><br /><ul><li>Meaningful variable and function names.</li><ul><li>The compiler doesn't care whether the function name is 3 characters or 30 characters long. However, a colleague reading the code will be very grateful if the function is called <code>get_daily_rainfall_average()</code>, rather than <code>dra()</code>. Who cares that it takes two seconds more to type it? Many modern IDEs are going to do that for you anyway.<br /></li></ul><li>Thoughtful source code documentation.</li><ul><li>Explaining in one sentence what some code does and in possibly many more sentences <span style="font-style: italic;">why</span> it does it and why it's needed. The <span style="font-style: italic;">why</span> is often much more important then the how: Explain the rational behind your design or implementation decisions. Everyone can see that a variable is increased in a for-loop, that doesn't have to be documented. But why the for-loop is needed in the first place is much more interesting and illuminating to someone who is new to the code.</li><li>Comments like this should be there for each module, class, function and the more complex code blocks within a function. Keep the comments close to the code they refer to. If they are all in the function or module header, they are often forgotten when the internals of code are changed, 15 pages further down.</li></ul><li>A legible and consistent coding style.</li><ul><li>We can endlessly argue about which style of parenthesis is the right one, but what's more important is that you remain consistent throughout your project.</li><li>If you are new to a team, use the coding style they use, even if it's not your favourite one.</li><li>Unless absolutely needed for performance reasons, don't try to optimize your doubly-nested loop into a fancy single line statement, exploiting even the most esoteric features of the language. Instead, break it up into a more easily understandable set of spelled out loops.</li><li>In general, don't optimize your code until you know you have to! Not only does it waste time if it turns out that the code really isn't performance critical, it is often also much more difficult to read and maintain.<br /></li><li>If you can do what you have to do with simple and often used language features then use those.</li><li>Use white-space to make code less 'dense' and increase legibility. Many developers have pretty large screens these days, white space doesn't cost money.</li><li>Align code. For example, if you have several variable assignments, align the '=' operators directly underneath each other. It's astonishing how much more legible that block of code becomes. If you have to line-break a long function call, indent the arguments of the second line to align with the start of the arguments in the first line. That's just a little example, but there are many cases like this in most programs, where a bit of thoughtful alignment can make a difference.<br /></li></ul></ul>It is often astonishing what amount of resistance some developers put up against even those few, simple rules. I have heard arguments, such as:<br /><br /><ul><li>"I don't want to write comments, because then I will have to scroll more to see the code."</li><li>"Writing comments in the code takes time."</li><li>Formatting the code nicely takes time, especially when I need to change a few things, in which case I then have to re-format the code.</li><li>"Code changes, and before you know it the comments are obsolete."</li></ul>I have absolutely no patience for the first three arguments. There might be some extremely rare situations where an emergency fix needs to be rushed out and short-cuts need to be taken. But for the most part, those arguments are bogus. The only one even remotely credible is the last one about comments getting obsolete, but this can be addressed in a straight forward manner: Keep the comments close to the code, and do code reviews where readability and correctness of the comments are stressed as well.<br /><br />In fact, I claim that if you don't take those rules to heart in your own source code then you are either unprofessional, lazy, not a team-player, or all of the above. If you as a software developer take pride in your professionalism and quality of your work then you have to consider that it is not only the achieved functionality for which you are being paid: The code you produce in almost all cases becomes property of your employer. Therefore, the code itself also becomes a product you deliver, and actually your most important product.<br /><br />How useful and usable your code is for the team who has to work with it is what really determines its value in the long run.<br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com23tag:blogger.com,1999:blog-6748806271290317100.post-72212303390162428262009-06-12T15:54:00.006+12:002009-07-16T09:06:46.114+12:00A Django ORM bug when using PostgreSQLNormally, an ORM such as the one used in Django is expected to 'do the right thing' with the database. I should be able to set up my models, and use the ORM without having to give any further thought to what it does behind the scenes. Switching between different databases should be as simple as changing two or three lines in my settings.py file: Use PostgreSQL instead of sqlite, for example.<br /><br />At least in theory. It can be quite jarring to come across situations where this is not the case. And that is exactly what happened to me today.<br /><br />Imagine a simple Django model:<pre><code> >>> class Foo(models.Model):<br /> >>> date = models.DateField()<br /></code></pre>Now assign a somewhat unusual date to a model instance:<pre><code> >>> f = Foo()<br /> >>> f.date = datetime.date(1,2,3)<br /> >>> f.date.year<br /> 1<br /> >>> f.date.month<br /> 2<br /> >>> f.date.day<br /> 3<br /></code></pre>You can see that we are talking about a date a long time ago. Nevertheless, a valid date as we can confirm by querying the individual elements of the date. Now save the instance:<pre><code> >>> f.save()<br /></code></pre>If you are using sqlite and do a <code>select * from foo;</code> in your database shell, you will see the following as the output:<pre><code> 1-02-03<br /></code></pre>If you load a Django object from that table, you will find everything in order and as expected.<br /><br />However, now try the same with PostgreSQL as database. After doing the very same thing in Python, take a look at what's written in the database table:<pre><code> 2003-01-02 <br /></code></pre>Oops!<br /><br />Now that shouldn't happen!<br /><br />The interesting thing is that as soon as you specify a year that has at least 3 digits, PostgreSQL stores the correct date. Do <code>f.date = datetime.date(101,2,3)</code> and after you save the model in the database it will have stored what we expect to see:<pre><code> 0101-02-03<br /></code></pre>Also interesting, if you specify a year with two digits and a value more than 12, you get an exception. For example <code>f.date = datetime.date(99,2,3)</code>:<pre><code> Traceback (most recent call last):<br /> .....<br /> DataError: date/time field value out of range: " 99-02-03"<br /> HINT: Perhaps you need a different "datestyle" setting.<br /></code></pre>I checked the DATESTYLE setting for my database, and indeed, it was ISO, which defines a date as YYYY-MM-DD. In theory, therefore, that <span style="font-weight:bold;">should</span> work. But obviously, something is not quite lining up.<br /><br />To summarize, while it all works fine with sqlite, in PostgreSQL I get:<ul><li>A wrong date stored for years between 1 and 12<br /><li>An exception for years between 13 and 99<br /><li>Correct behaviour for years 100 or up<br /></ul>So, in case you need to work with really small dates in your Django app, you might want to keep this in mind.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com0tag:blogger.com,1999:blog-6748806271290317100.post-52373901883556829952009-06-01T13:05:00.006+12:002009-07-16T09:15:44.397+12:00How to do "count" with "group by" in DjangoFaced with the 'age old' problem of having to do a <code>group by</code> query in Django, I finally came across a <a href="http://www.eflorenzano.com/blog/post/secrets-django-orm/">solution</a>.In short, you use the lower-level query interface directly:<br /><pre><code> q = MyModel.objects.all()<br /> q.query.group_by = ['field_name']<br /></code></pre>When you then iterate over 'q', you get the results grouped by the value of whatever was in 'field_name'. Great! So far, so good.<br /><br />Well, a word of caution at this point: Using the low-level query API is not really something Django wants you to do, apparently. In fact, try this with sqlite and it works. Try it with PostgreSQL and it does not (all sorts of error messages). So, your mileage may vary, depending on which database you are using. Ok, let's assume that your database is fine with this...<br /><br />The only challenge for me now was that I had to do a <code>count</code> for this.<br />If you form your query using the usual methods of Django's ORM, you will be disappointed. For example, this here will <span style="font-weight: bold;">not</span> work:<br /><pre><code> q = MyModel.objects.all()<br /> q.query.group_by = ['field_name']<br /> q.count()<br /></code></pre>It appears as if this returns only the count of the first group, not a list of counts, as you would expect.<br /><br />The solution is to wade a bit deeper into the low-level query API. We can instruct the query to add a count-column. In fact, this results in merely a single column being returned, just like <code>COUNT(*)</code>. It goes like this:<br /><pre><code> q = MyModel.objects.all()<br /> q.query.group_by = ['field_name']<br /> q.query.add_count_column()<br /></code></pre>Since this returns counts, rather than complete objects, we now need to get the individual group counts as a list of values. We add one more line:<br /><pre><code> q.values_list('id')<br /></code></pre>The <code>values_list()</code> function gives you not instantiated objects but instead the tuples of values for each object. The tuples contain only the fields you specify by name in the call to <code>values_list()</code>. Except, we have manually added the count column with the <code>add_count_column()</code> function. That count column is always returned first. So, what you get as result is something like this:<br /><pre><code> [ (3,1), (19,8), ... ]<br /></code></pre>The first value of each tuple is the count, the second value is the value of the 'id' field of the last occurrence of the grouped model in the table. If you specify something other than 'id', you get the same thing: First the count and then the value of that other field.<br /><br />But that's not what we want, right? We want a list of counts. We could manually extract the first element in each tuple, but Django offers us a shortcut:<br /><pre><code> q.values_list('id', flat=True)<br /></code></pre>Setting <code>flat=True</code> tells the <code>values_list()</code> function to just return the first element of each tuple in a plain list (not a list of tuples). And since the first element now is the count column, we finally get what we want: A list of the counts for each group.<br /><br />Note that we could have specified any other fields in the call to <code>values_list()</code>, not just 'id'. Because we specified <code>flat</code> those fields are ignored. It seems, though, that at least one field needs to be specified here, even though it won't be considered as part of the output.<br /><br /><br />You should follow me on twitter <a href="http://twitter.com/BrendelConsult">here</a>.jbrendelhttp://www.blogger.com/profile/09547073366771090616noreply@blogger.com1