Kiwi Pycon 2011

Kiwi Pycon 2011, the annual Python conference in New Zealand has just been concluded. Two days in Wellington, full of inspiring talks and discussion. Thank you to the organizers and all the presenters. It's really nice to see other developers interested in Python.

In New Zealand, the Python community is pretty small. If you look through the software related section in a bookstore, you will find very little about Python. And the Python job market... well, there really isn't much. That is probably one of the reasons why I mostly work with overseas clients.

Hopefully, Python continues to make inroads with New Zealand organizations. It would benefit them and all of us here on this small, South Pacific island.

 

 

Tips on using oDesk contractors as system administrators

Sites like oDesk, eLance and others give any business instant access to skilled, remote staff, often working at very competitive hourly prices. Basically, you can find outsourced workers (usually individuals, sometimes teams) for small to mid-sized projects you need to get done, while not having to deal with paperwork or recruiting. A lot has been written about the pitfalls of outsourcing and also on how to make such outsourced work succeed: Define the project very well, document the APIs against which the contractor should work, demand good documentation, etc. However, most of these tips and tricks have been written about software development and coding.

But what about tasks such as system administration? You can find quite a few sysadmins on these sites. If you are an entrepreneur who wishes to focus on the business idea, or maybe even just on the software development, what is more tempting than to outsource those pesky aspects of system administration to someone else? Setting up a web-server with caching proxy, a database server, a CMS, a cluster of build and test servers, etc. After all, these might not be your core skills. Figuring all of this out yourself would take time, you might still not get it right and it certainly is quite distracting. So, in theory, these should make for perfect outsourced projects.

There's just one problem.

Consider that most of the system administration tasks I just mentioned require root or administrator access to your systems. Those systems later will hold your software, sometimes your source code or - even more important - your customer data, maybe logins, addresses or even credit card numbers.

Now consider that the contractors you meet on oDesk are very often in different countries and that you will have no contract with them besides whatever automated agreement you strike up via oDesk. You won't meet them in person, there usually are no NDAs and sometimes you won't even talk to them on the phone.

I have worked with quite a few oDesk contractors now and was lucky enough to meet many fine engineers and administrators. But if push comes to shove, you just don't know ahead of time who you are really going to get, and considering that any dispute would likely be across borders and legal systems, realistically you would have no legal recourse in case something goes wrong.

While any issues or disputes are bad enough when it comes to software development, you at least have a deliverable you can review and examine relatively easily, or even throw away should you decide to do so. However, for system administration tasks, there are numerous ways in which a disgruntled or simply malicious or careless person with root access could compromise your system before it goes into production. Backdoors could have been installed, timed jobs could run unbeknownst to you, collecting and sending off important data from your system for some nefarious purposes, or some passwords and accounts could accidentally have been left open and unprotected.

So, having used oDesk contractors for a number of projects, including system administration tasks, let me give you two recommendations:

Firstly - and true for any type of task - choose contractors who have worked a lot on oDesk (or whichever other outsourcing site you use) and who are still active there. Look for the "recently worked hours" and recent projects. Look for contractors with a good reputation. The idea is to choose contractors who really work projects on those sites for a living and thus rely on a good reputation. That, of course, is still not a guarantee, but it is a good starting point.

Secondly, if at all possible, do NOT give "system administration" tasks to contractors (which require root access to production systems). Instead, give "system administration automation" tasks. That's a subtle but important difference. However, it pays off in several ways. These are slightly arbitrary definitions, but they may serve to make a point:

With "system administration" tasks I mean: Setting up a complete server for you, or installing and configuring an additional software package on your server. Basically, working directly on a production system. You give someone root on those machines and they have the keys to the kingdom, with all the issues we have described before. If you need to have those tasks done, you should probably look for someone you can meet in person, get a better feel for and develop a better trust relationship with.

On the other hand, with "system administration automation" tasks I mean the development of scripts, which setup a server (or software package) for you in an automated fashion. How does this work? There are particular tools (for example puppet or chef), or just a number of custom shell scripts, which can be used to automate the administration and setup of machines. Now the project description is different. While before you description may have read "Configure a caching proxy for our application server..." it now changes to "Provide scripts, which automate the setup of a caching proxy for our application server...".

What do you gain with this? On one hand, you can just give a throw-away system to the contractor on which to develop those scripts. Throw away servers are easy and cheap to get these days. You may use Amazon EC2, Rackspace, Linode or any number of other providers to get "dedicated looking" machines for just a few dollars per month or even just on an hourly basis. On those machines, the contractor can get a root login and can develop the automation scripts until they work. Once the project is completed you can test it easily: Get yourself another throw away server and run the finished scripts (which naturally have to come with documentation on how to run them). If your server ends up configured properly, the project was successful. You may specify scripts for automated tests and self-diagnosis as part of the deliverable as well. However, at no point did the contractor require root or administration rights on your actual production server.

It's important to use the same or similar base machine image for the throw away server as for the production server. So for example, if you run your production server on Linode, with Ubuntu 10.04 LTS as base OS you should create a similar Linode instance with the same base OS for the contractor as throw away server to work on. Then likewise to test the deliverables before applying those scripts to your production server. Also, consider that if all your configuration setups for your production server are automated then it is easy to give a complete mock production server to the contractor as a throw away server: You can quickly bring the throw away server into the same state (or at least containing the same configurations and software packages) as your production server before the contractor starts work. Just be careful not divulge important passwords or SSH keys that way.

A second advantage of requiring automation scripts is that the scripts themselves can act as a documentation of sorts of all the tasks and steps that are involved in completing this system administration task. Therefore, they will allow you and other contractors to more easily pick up where one contractor left off.

Finally, if you did not automate the system administration task, you cannot easily reproduce the work in case you want to bring up an additional production server. So, having automated scripts helps you to scale more effectively, as your organization grows.

Do you have any other tips and ideas on how to effectively work with remote contractors for system administration tasks? If so, please let us know in the comments.

So, the next time you ask for a system administration task to be performed and before handing over the root password to your production server to the contractor, consider if it might not be better to instead phrase the project as a system administration automation task instead.


You should follow me on twitter here.