Mixed Jython/Python development and Google AppEngine oddities

Recently, I worked on a Python project, which I developed as a stand-alone WSGI application. I also experimented with running this app in Jython. In case you don't know: Jython is a neat Python implementation, which runs in the Java VM. The advantage of this is that you have access to all the usual Java infrastructure, framework and libraries, while being able to rapidly develop your program in the Python language. I ended up designing my application so that it could run both as Python as well as Jython project.

Shortly afterwards, I moved my WSGI application to Google's AppEngine, which is is a hosted environment for Python and Java apps. AppEngine can run pure WSGI applications quite easily, so this shouldn't be too much of a problem. Using Google's local development server (dev_appserver.py), I was able to run my application on my desktop in the emulated Google environment without problem.

I then used the provided command line utility (appcfg.py) to move my app to Google's actual servers... and nothing worked. I received a "500 Internal Server Error". Looking at the logs I found that my application was not able to import any of my own modules. While standard library items were imported without an issue, any module I had provided myself resulted in an import error.

What had happened? It took me a long time to figure this out, so I thought I share it with you: When running the Python application via Jython, a bunch of .class files were created, which is normal. However, when the application was uploaded via appcfg.py, those class files were uploaded as well (files ending in *.pyc and a few other extensions are ignored, but apparently anything else is considered to be part of your application).

So then, for reasons that aren't quite clear, the presence of .class files in your Python application seems to be really confusing for Google AppEngine.

The lesson? If you are developing an application that can run locally in Jython as well as on Google's AppEngine, make sure to clear out your *.class files before uploading. It can save you hours of grief.

Oh, and by the way, you should follow me on twitter here.

 

 

Search engine comparison for software developers

For today's software developer it is difficult to imagine that there once was a time where we did not have instant access to information on the Internet to help us do our work: Answers to questions, howto-guides, reference material, documentation... whatever we need, it is usually just a quick search away.

I can't count how many times I simply searched for something like "Django ... such and such ..." to instantly find the relevant documentation or reports of other people's experience while working on a Django project, for example. The amount of time this saved me is immeasurable. I have a friend who - as a software developer - is not allowed to access the Internet while at work! The mind boggles. How much does this cost his employer in lost productivity?

Anyway, clearly the access to search engines while developing software should be considered essential. If during a job interview I'd ask a candidate something like: "How do you define a C function pointer?" and the answer would be: "I've done it before, but can't recall the exact syntax now... I'd just google it!" ... well, I would consider that to be a perfectly valid answer, to be honest. One more reason why these types of questions are quite meaningless in interviews. But that's a different topic.

So, the question then is: What is the best search engine for a software developer to use? Many people default to Google, obviously, but personally, I don't like to let one company have all the information about me, so I like to mix it up a little. The difference in search results can be quite startling, though. With Bing as the new arrival on the scene, let's perform a completely subjective and un-scientific experiment and ask four of the major search engines the same set of questions and see what they tell us.

For this test, I have chosen Bing, Google, Yahoo and Ask. Sadly Yahoo search will soon stop to exist when they start using Bing as a search provider, but for sentimental reasons and while it is still available, I thought I include them. Especially since Yahoo was my default search engine for quite some time now.

I have chosen four questions, which obviously cover only a tiny fraction of software development related areas:

1. "Django IntegerField model": Here I would like to see links to the Django project documentation on how to define an IntegerField in a model. It should be noted that 'IntegerField' also exists as part of forms, so the search engine should not confuse these two.
2. "Define C function pointer": Our 'interview question' from above. The top results should lead to pages that very clearly explain how to define this thing.
3. "TCP checksum calculation": I would like to see links to pages that explain how the algorithm works that computes TCP checksums.
4. ".Net class inheritance": I'm not an MS developer at all, so I'm not even quite sure whether this question makes much sense, but I thought I try this one, just on a hunch.

Ok, let's start...

The test

Question 1. "Django IntegerField model"

Here are the results from all four search engines, side by side. In all images here, you can see Bing in the first column, followed by Google, then Yahoo and finally Ask in the last column.

• Bing: The first results are essentially useless, leading (ironically) to Google code pages, followed by some obscure changeset in the Django code itself. Sure, you can see an IntegerField used there. However, the link to the Django docs, which I was looking for, is only at the very bottom of the page.
• Google: The results are highly relevant (only peeve might be that they are pointing to docs for the SVN version, not the latest stable release version, but that's really minor). A link to the older Django version's docs is in third place, which is useful. But as we can see, the ads are not quite as relevant.
• Yahoo: The results are not bad. However, I didn't like the fact that it pointed me to docs for a much older version of Django in the top result. They could have done better here.
• Ask: The top results are good. However, I am disturbed by those ads, which appear right under the top result. They could have shown them on the side. But on Ask the side space is taken up by "Related Searches", which in this particular case were completely irrelevant.

Ranking: Google wins hands down. Ask in second place, closely followed by Yahoo. Bing is dead last.

Question 2. "Define C function pointer"

Here are the results for this question.

• Bing: Again disappointing. The top result is about a function pointer validation function... which would be relevant if obviously I didn't need to learn first how to define a function pointer. But since I am looking only for how to define a pointer these results don't help me much. The third one is even about how to call C functions from Forth!?
  
• Google: This is exactly what I was looking for: A tutorial! Top link, good job Google!
• Yahoo: Not bad, with the tutorial in second place. For some reason, the Wikipedia entry comes first. Both Yahoo and Google generally give quite some weight to Wikipedia pages, which I am normally fine with. In this particular case, the tutorials are really what I'm looking for, though.
• Ask: Again a very good top result, just like Google, marred by irrelevant annoying ads, followed by other good results (for example how to define a function-pointer parameter for a function, which is very nice).

Ranking: Ask wins this one based on the result quality, even though their ads bother me. Google is a very close second, Yahoo in third, but still close. Bing is way off the mark again.

Question 3. "TCP checksum calculation"

The results for this one, side by side.

• Bing: Very disappointing results at the top, leading to small discussion threads and postings here and there. Not at all what I was looking for.
• Google: Excellent results, especially the third link ("TCP Checksum code"). That page contains a textual description of the algorithm as well as actual sample code. Perfect. The further links on the page remain relevant as well. The PDF in fourh place is actually quite good.
• Yahoo: The top three results are the same as Google, so that's very good. The relevancy of the remaining links trails off quickly, though.
• Ask: The top three results are the same as Google and Yahoo. The remaining links remain more relevant than Yahoo, but not quite as good as Google, though.

Ranking: Google in first, followed by Ask and Yahoo. Bing again in last place.

Question 4. ".Net class inheritance"

Here is a question for Microsoft's Bing to shine! It should know all about that, right? Well, here are the results...

• Bing: Oh the humanity! What were they thinking? Again, the top link goes to some forum page, the second one is a blog. That's a recurring and unsuccessful theme with Bing. The third one contains some sample code, but is basically someone making the case for multiple-inheritance, which apparently is missing in .Net.
• Google: Hits the mark again with the top link pointing straight to the relevant page of Microsoft's own online documentation. Why couldn't Bing come up with this? And the second link is great, too: A tutorial for class inheritance, which is always nice to see.
• Yahoo: Good results in the top two spots, with a useful page about interfaces included, which Google didn't show. But the third result is disappointing: Apparently, Yahoo was confused by the domain name for that result, which ended in ".net".
• Ask: The ads are disturbing as usual, but the top results are the same as Google's, which is good. Not visible in this screenshot, because too much valuable space was used by those ads, there is also a good tutorial link further down, which Google didn't show.

Ranking: Ask by a slim margin (if we manage to ignore those ads), Google in close second, followed by Yahoo a bit further back. Bing is again at the very end with a smattering of useless results.

I sure hope that Microsoft's own developers are free to use Google or Ask at work.

Conclusions

Let's start at the back.

Maybe they are still trying to sort out issues and will improve over time. However, at the moment Bing is just completely hopeless when it comes to these kinds of queries. I guess they are much more consumer oriented. As it stands, though, for questions software developers need to ask for their work, Bing quite plainly ... sucks!

Yahoo still tends to be good and delivers useful results. But while the top results are usually ok, the relevance of further results quickly drops. Of course, that will all end once they switch to Bing. At that moment they will get much, much worse. What a sad loss for the Internet to see Yahoo search go.

Ask and Google are actually ranking equally well here in this test. That may come as a surprise to many. The biggest annoyance about Ask is the placement of the ads right after the top result. And their "Related Searches" links usually don't provide any value at all for these types of searches. But the quality of the links is about as good as Google. The ads on both Ask and Google are close to irrelevant in both cases.

A message to Ask: Please change the placement of your ads or at least distinguish them visually a bit more!

So, take your pick: Google or Ask. I will start using Ask, because as I said earlier, Google is on enough pages already by means of their ad network and they don't need to know everything about me. And with Yahoo ranking behind Ask and soon completely fading away, Ask remains the only real alternative for search results that are relevant to software developers.

You should follow me on twitter here.

 

 

Google's Chrome OS is bad news for Linux

Google's Chrome OS has been announced today and - like any announcement from Google - is already widely being discussed. People usually focus on the supposed threat that this 'new' OS represents to Microsoft's dominance on the desktop. The commentary focuses on this Google vs. Microsoft aspect of the story. However, I think the news is actually worse for Linux than it is for Microsoft.

How can I say this, considering that Chrome is be based on a Linux kernel? Let me explain, with a 'historical' analogy.

In the 80s and 90s we had the so-called Unix Wars: Prominent Unix vendors like Sun, Digital, IBM, AT&T and many others were battling for supremacy in the server market. Guess who emerged victorious from those Unix Wars?

Microsoft's Windows NT.

Customers were sufficiently fed-up with the fragmentation and confusion in the Unix server market and finally were receptive to a message of simplicity: Your desktops already run Microsoft software, so why not your servers? It will all play nicely together ... just deal with a single vendor ... yada yada.

Now fast forward 15 to 20 years. While Unix is holding on to various diminishing pockets in the server market and Windows server-products have a stronghold in many enterprises, Linux has emerged as a very viable and popular choice for the server. On the desktop, however, Microsoft rules supreme now as it did then.

Back in the Unix Wars, Microsoft used the fragmentation amongst the competitors to become a dominant force in the market. Today, it enjoys the advantage that the fragmentation of potential opposition in the desktop market is handing to it. Nowhere is the fragmentation of Linux more apparent than on the desktop.

Ubuntu managed to become a strong player in the Linux desktop market, and for once it appeared as if things were finally coalescing. Ubuntu's popularity started to feed itself: Any issues or problems? Chances are someone solved it for Ubuntu already, a quick search on the Internet reveals answers to most common issues. Therefore, while Ubuntu might not be perfect for everyone or might have a few annoyances, for the most part you just don't go wrong settling on Ubuntu: So many others are running it, momentum has been forming.

Now Google comes along with Chrome OS. We now have another major player entering the Linux desktop OS market. Linux needs this like we need another hole in the head. Microsoft will read this news with glee: Through the opponent's fragmentation it managed to win the Unix wars. Now Linux's continued fragmentation will ensure that there isn't even going to be a war for the desktop.

So, I wish for Google Chome OS to fail. We don't need it. It doesn't help Linux. As suggested here, if Google wants an improved non-Microsoft desktop OS then please throw your considerable resources behind improvements in the existing Linux desktop distros. We will all be better off for it.


You should follow me on twitter here.

 

 

Setting the initial value for Django's ModelChoiceField

Recently, I worked with a Django form that utilized the ModelChoiceField. That is a convenient field that normally is rendered as an HTML select tag, which usually appears as a drop-down menu on a web-page.

A ModelChoiceField is specified as:

    class MyForm(forms.Form):
        my_field = forms.ModelChoiceField(queryset = MyModel.objects.all())

As you can see, this field is used to easily specify a drop-down for all items in a table (or whatever the queryset specifies). The model is represented as the output of its __unicode__() function. If you evaluate the form after it was posted, the value for the field is going to be an instance for the actual model whose unicode representation was selected.

The problems for me started when I tried to set an initial value for the field: In an 'edit' form, I wanted all fields to reflect whatever had been saved for a particular model instance, of course. As I said above, if you evaluate the form after it has been submitted you get an actual model instance. Naturally, you would think that initial values would be set in a symmetric manner, by specifying a model instance:

    form = MyForm(initial = { 'my_field' : some_model_instance })

Sadly, this doesn't work. And try as I might, I couldn't find an answer to this on the Internet either (more on that in a moment). So, after looking at the Django code, it finally dawned on me that you need to specify the ID of the model instance as the initial value:

    form = MyForm(initial = { 'my_field' : some_model_instance.id })

That works now. It's a bit unfortunate that the retrieval value (a model instance) and the initial value (the ID of a model instance) are of a different type. It's an inconsistency in the Django API, I think. But in the end, I probably should have at least tried that one a bit sooner.

The surprising thing is that I couldn't find any discussion of this anywhere on the Internet. I should mention here that I am using Yahoo as my default search engine. Shortly after I finally found the solution, it occured to me to try Google. And wouldn't you know it? Right there, third hit from the top, I had the answer.

Why did Yahoo not give me this result? Well, the answer was discussed on Google Groups. Is Yahoo not indexing those? Or is Google not letting them index it?

Either way, that small issue between Yahoo and Google cost me a few hours of frustration. So, I'm posting the solution here on a non-Google page, so that Yahoo users may also find the answer to that problem in the future.


You should follow me on twitter here.