Planet Bazaar

2 year cadence for major releases: some progress

2010-03-15T09:08:45+00:00

Even though the idea of formal alignment between the freezes of Debian and Ubuntu didn’t hold, there has been some good practical collaboration between the maintainers of key subsystems. There are real benefits to this, because maintainers have a much more fruitful basis for sharing patches when they are looking at the same underlying version.

Harmonization for Ubuntu 10.04 LTS and Debian Squeeze

I think this is where we stand now:

	Ubuntu	Debian	RHEL	SLES
Kernel	2.6.32 + drm-33	2.6.32 + drm-33	2.6.32	2.6.32
GCC	4.4	4.4
Python	2.6	2.6
OpenOffice.org	3.2	3.2
Perl	5.10.1	5.10.1
Boost	1.40	1.40

I’m sure there are inaccuracies, please help me keep this up to date, sabdfl on freenode is the best way to reach me. The RHEL and SLES numbers are third-hand, so up-to-date information would be appreciated.

The actual release dates of Ubuntu LTS and Debian will vary of course, because of different priorities. And there’s no requirement that the same base version be used for every major component – there may well be differences allowing for different approaches. But where we do have it, we’ll be able to collaborate much more effectively on bug fixes for key upstream pieces. If a lot of distributions pick the same base upstream version, it greatly increases the value of extended shared maintenance and point releases of that upstream.

Why every two years?

Two years is a compromise between those who want 1 year releases for better support of cutting-edge hardware and those who want 7 year releases so their software stack doesn’t change before their job description does .

A whole-year multiple has several advantages. It means we can schedule the processes that are needed for collaboration at the same time of year whenever we need them – unlike 1.5 or 2.5 year cycles. Three years was felt to be too long for hardware support. Two years is perceived to be the Goldilocks Cadence – just right.

What are the criteria for choosing a common base version?

In both the Ubuntu and Debian cases, we’ll be making a release that we support for many years. So be looked for versions of key upstreams that will pass the test of time. Sometimes, that means they can’t be too old, because they’ll be completely obsolete or unmaintainable in the life of the release. And sometimes that means they can’t be too young. In general, it would be better to be reviewing code that is already out there. But there are also lots of upstreams that do a credible job of release management, so we could commit to shipping a version that is not yet released, based on the reputation of the community it’s coming from.

What if there’s no agreement on a particular kernel, or X or component-foo?

We will almost certainly diverge on some components, and that’s quite OK. This is about finding opportunities to do a better job for upstreams and for users, not about forcing any distro to make a particular choice. If anyone feels its more important to them to use a particular version than another, they’ll do that.

Open invitations

It’s really helpful to have upstreams and other distributions participate in this process.

If you’re an upstream, kick off a thread in your mailing list or forums about this. Upstreams don’t need to do anything different if they don’t want to, we’ll still just make the best choices we can. But embracing a two year cadence is the best way you have to be sure which versions of your software are going to be in millions of hands in the future – it’s a great opportunity to influence how your users will experience your work.

Of course, we’d also like to have more distributions at the table. There’s no binding commitment needed – collaboration is opportunistic. But without participating in the conversation one can’t spot those opportunities! If you represent a distribution and are interested, then please feel free to contact me, or Matt Zimmerman, or anyone on the Debian release management team about it.

I think this is a big win for the free software community. Many upstreams have said “we’d really like to help deliver a great stable release, but which distro should we arrange that around?” Upstreams should not have to play favourites with distributions, and it should be no more work to support 10 distributions as to support one. If we can grow the number of distributions that embrace this cadence, the question becomes moot – upstreams can plan around that cycle knowing that many distributions will deliver their work straight to users.

The Bazaar Package Importer

2010-03-14T18:55:00+00:00

The Bazaar package importer is a service that we run to allow people to use Bazaar for Ubuntu development by importing any source package uploads in to bzr. It's not something that most Ubuntu developers will interact with directly, but is of increasing importance.

I've spent a lot of time working in the background on this project, and while the details have never been secret, and in fact the code has been available for a while, I'm sure most people don't know what goes on. I wanted to rectify that, and so started with some wiki documentation on the internals. This post is more abstract, talking about the archtecture.

While it has a common pattern of requirements, and so those familiar with the architecture of job systems will recognise the solution, the devil is in the details. I therefore present this as a case-study of one such system that can be used to constrast other similar sytstems as an aid to learning how differing requirements affect the finished product.

The Problem

For the Ubuntu Distributed Development initative we have a need for a process that imports packages in to bzr on an ongoing basis as they are uploaded to Ubuntu. This is so that we can have a smooth transition rather than a flag day where everyone switches. For those that are familiar with them think Launchpad's code imports but with Debian/Ubuntu packages as the source, rather than a foreign VCS.

This process is required to watch for uploads to Debian and Ubuntu and trigger a run to import that upload to the bzr branches, pushing the result to LP. It should be fast, though we currently have a publication delay in Ubuntu that means we are used to latencies of an hour, so it doesn't have to be greased lightning to get acceptance. It is more important to be reliable, so that the bzr branches can be assumed to be up to date, that is crucial for acceptance.

It should also keep an audit trail of what it thinks is in the branches. As we open up write access to the resulting branches to Ubuntu developers we can not rely on the content of the branches not being tampered with. I don't expect this will ever be a problem, but I wanted to ensure that we could at least detect tampering, even if we couldn't know exactly what had happened by keeping private copies of everything.

The Building Blocks

The first building block of the solution is the import script for a single package. You can run this at any time and it will figure out what is unimported, and do the import of the rest, so you can trigger it as many times as you like without worrying that it will cause problems. Therefore the requirement is only to trigger it at least once when there has been an upload since the last time it was run, which is a nicer requirement than "exactly once per upload" or similar.

However, as it may import to a number of branches (both lucid and karmic-security in the case of a security upload, say), and these must be consistent on Launchpad, only one instance can run at once. There is no way to do atomic operations on sets of branches on Launchpad, therefore we use locks to ensure that only one process is running per-package at any one time. I would like to explore ways to remove this requirement, such as avoiding race conditions by operating on the Launchpad branches in a consistent manner, as this would give more freedom to scale out.

The other part of the system is a driver process. We use separate processes so that any faults in the import script can be caught in the supervisor process, with the errors being logged. The driver process picks a package to import and triggers a run of the script for it. It uses something like the following to do that:

write_failure(package, "died")
try:
    import(package)
except:
    write_failure(packge, stderr)
finally:
    remove_failure(package)

write_failure creates a record that the package failed to import with a reason. This provides a list of problems to work through, and also means that we can avoid trying to import a package if we know it has failed. This ensures that previous failures are dealt with properly without giving them a chance to corrupt things later.

Queuing

I said that the driver picks a package and imports it. To do this it simply queries the database for the highest priority job waiting, dispatching the result, or sleeping if there are no waiting jobs. It can actually dispatch multiple jobs in parallel as it uses processes to do the work.

The queue is filled by a couple of other processes triggered by cron. This is useful as it means that further threads are not required, and there is less code running in the monitor process, and so less chance that bugs will bring it down.

The first process is one that checks for new uploads since the last check and adds a job for them, see below for the details. The second is one that looks at the current list of failures and retries some of them automatically, if the failure looks like it was likely to be transient, such as a timeout error trying to reach Launchpad. It only retries after a timeout of a couple of hours has elapsed, and also if that package hasn't failed in that same way several times in a row (to protect against e.g. the data that job is sending to LP causing it to crash and so give timeout errors.)

It may be better to use an AMQP broker or a job server such as Gearman for this task, rather that just using the database. However, we don't really need any of the more advanced features that these provide, and already have some degree of loose-coupling, so using fewer moving parts seems sensible.

Reacting to new uploads

I find this to be a rather neat solution, thanks to the Launchpad team. We use the API for this, notably a method on IArchive called getPublishedSources(). They key here is the parameter "created_since_date". We keep track of this and pass it to the API calls to get the uploads since the last time we ran, and then act on those. Once we processed them all then we update the stored date and go around again.

This has some nice properties, it is a poll interface, but has some things in common with an event-based one. Key in my eyes is that we don't have to have perfect uptime in order to ensure we never miss events.

However, I am not convinced that we will never get a publication that appears later than one that we have dealt with, but that reports an earlier time. If this happens we would never see it. The times we use always come from LP, so don't require synchronised clocks between the machine where this runs and the LP machines, but it could still happen inside LP. To avoid this I subtract a delta when I send the request, so assuming the skew would not be greater than that delta we won't get hit. This does mean that you repeatedly try and import the same things, but that is just a mild inefficiency.

Synchronisation

There is a synchronisation point when we push to Launchpad. Before and after this critical period we can blow away what we are doing with no issues. During it though we will have an inconsistent state of the world if we did that. Therefore I used a protocol to ensure that we guard this section.

As we know locking ensures that only one process runs at a time, meaning that the only way to race is with "yourself." All the code is written to assume that things can go down at any time as I said, the supervisor catches this and marks the failures, and even guards against itself dying. Therefore when it picks back up and restarts the jobs that it was processing before dying it needs to ensure that it wasn't in the critical section.

To do this we use a three-phase commit on the audit data to accomany the push. When we are doing the import we track the additions to the audit data separately from the committed data. Then if we die before we reach the critical section we can just drop it again, returning to the inital state.

The next phase marks in the database that the critical section has begun. We then start the push back. If we die here we know we were in the critical section and can restart the push. Only once the push has fully completed do we move the new audit data in to place.

The next step cleans up the local branches, dying here means we can just carry on with the cleanup. Finally the mark that we are in the critical section is removed, and we are back to the start state, indicating that the last run was clean, and any subsequent run can proceed.

All of this means that if the processes go down for any reason, they will clean up or continue as they restart as normal.

Dealing with Launchpad API issues

The biggest area of operational headaches I have tends to come from using the Launchpad API. Overall the API is great to have, and generally a pleasure to use, but I find that it isn't as robust as I would like. I have spent quite some time trying to deal with that, and I would like to share some tips from my experience. I'm also keen to help diagnose the issues further if any Launchpad developers would like so that it can be more robust off the bat.

The first tip is: partition the data. Large datasets combined with fluctuating load may mean that you suddenly hit a timeout error. Some calls allow you to partition the data that you request. For instance, getPublishedSources that I spoke about above allows you to specify a distro_series parameter. Doing

distro.main_archive.getPublishedSources()

is far far more likely to timeout than

for s in distro.series:
    distro.main_archive.getPublishedSources(distro_series=s)

in fact, for Ubuntu, the former is guaranteed to timeout, it is a lot of data.

This is more coding, and not the natural way to do it, therefore it would be great if launchpadlib automatically partioned and recombined the data.

The second tip is: expect failure. This one should be obvious, but the API doesn't make it clear, unlike something like python-couchdb. It is a webservice, so you will sometimes get HTTP exceptions, such as when LP goes offline for a rollout. I've implemented randomized exponential backoff to help with this, as I tend to get frequent errors that don't apparently correspond to service issues. I very frequently see 502 return codes, on both edge and production, which I believe means that apache can't reach the appservers in time.

Summary

Overall, I think this architecture is good, given the synchronisation requirements we have for pushing to LP, without those it could be more loosely coupled.

The amount of day-to-day hand-holding required has reduced as I have learnt about the types of issues that are encountered and changed the code to recognise and act on them.

Dry Rub Barbeque Trout

2010-03-14T16:23:00+00:00

Made this up after buying a nice piece of locally caught freshwater trout. I think that it would be even better if you were to hot-smoke it. Apply the rub between two and twelve hours before cooking.

Mix up the following then rub on to the flesh of the fish (enough for four servings):

1 tbsp sea/rock salt.

1 tbsp black peppercorns crushed.

1 tbsp ground cumin.

1 tbsp ground coriander.

2 tsp caraway seed.

2 tsp dried tarragon.

2 tsp dried thyme.

2 tsp chilli powder.

Zest of one lemon.

To drizzle on top when cooked melt some butter in a pan, add the juice of the lemon you used above, a pinch of salt, one crushed clove of garlic, and a handful of chopped coriander. Simmer for a couple of minutes.

Enjoy!

Input overload

2010-03-13T23:05:19+00:00

During the last few months, despite filtering and thread-scoring, I'm having trouble keeping up with the continuous flood of emails that come my way. I'm now spending way too much of my time dealing with both email and other background noise (hello, web 2.0 services). To cope with this, I've now dropped off a couple dozen mailing lists, unsubscribed from a similar number of RSS feeds and left a few IRC channels. I'm slowly working working my way through the backlog of emails and merge requests that I still have to deal with. If you need me to participate in a mailing list discussion, please CC me.

cp: Agalloch - Our Fortress Is Burning, pt. 1

launchpadlib gotchas

2010-03-12T14:45:16+00:00

I've shown you how to get started with launchpadlib and have shown a slightly more complex launchpadlib script.

Before I cry The power is yours! and return to my home within the earth, I'm going to warn you about the things that can trip you up when using launchpadlib.

Bugs

launchpadlib has bugs. There are also bugs in lazr.restful and lazr.restfulclient – two libraries that are core to launchpadlib's behaviour.

Welcome to software engineering.

Documentation

It's on the wiki, and it's good, but it could always be better. There's a page of as well as a guide on launchpadlib.

The reference documentation isn't written for Python programmers. It's written for REST programmers. Actually, it's not written at all but rather auto-generated from our source code. Sometimes this can be confusing, and I frequently find myself consulting the Launchpad source code to get things done with the API.

Error messages

I'm told this has got better with recent releases, but often when you get an error in launchpadlib, it looks like an HTTP error and you have very little help on how to debug it. Unfortunately, I don't have an example ready.

If you come across an error like this, file a bug and head straight to #launchpad-dev on freenode to get help.

Potato programming

It's really easy to write code with launchpadlib that does this:

  for thing in bunch_of_things:
    thing.do_something_on_launchpad()

Code like this is really slow. It will do one round-trip per thing, which can be quite expensive. Twisted folks sometimes call this potato programming.

Exposure

Not all of the code within Launchpad is exposed through the API. We have to expose things manually and we haven't done it all yet. Sorry.

If you come across something that you want, then please file a bug and tag it with api.

In general, exposing something of the API is really easy or almost impossible. If the thing you want falls into the first category, you can probably patch Launchpad yourself.

Testing

Testing launchpadlib apps is hard. You do not want your unit tests to run against launchpad.net and running your own instance of Launchpad simply to run unit tests is masochistically stupid.

I think the situation here has improved recently too, but I haven't heard much about it or explored it myself.

Conclusion

There you have it, all of the gotchas for writing code with launchpadlib. As you can see, it's not really any worse than writing for any Python library – I'm just being up-front with you because I like you.

If any of these gotchas no longer apply, please correct me and I will shout your good news from the rooftops.

Until then, happy hacking.

Have you tried lptools?

2010-03-12T04:25:36+00:00

Have you tried lptools?

It's not at all officially associated with the Launchpad project, of course, but it's got a few nice things that you might want to look at, including:

a milestone manipulator
a code review notifier, and
a milestone to iCal exporter

One of these days, I'd like for there to be an official, awesome Launchpad command-line client. As it is, I'm happy with the world of people making extensions to meet their needs.

For other extensions, check out the lpx project or our list of clients.

Book Review : Designing Social Interfaces

2010-03-12T00:05:45+00:00

I just finished reading Designing Social Interfaces. The book is written by the creators of the Yahoo! Pattern Library. I had no knowledge of any such pattern library previous to reading this book, but I've really been exploring the idea of creating user experiences on the web that will bring in users and keep them happy (and not confused). I think a lot of this interest has to do with Canonical's current focus on design and usability.

The book is an O'Reilly book, but once it's opened, it certainly doesn't feel like an O'Reilly book. The pages are slicker than normal O'Reilly books (and I own many of them). They're also in full color. This is great for a user experience design book, since it's often hard to look at screenshots of web sites when they're in black and white.

I soon realized that I was fascinated with the book itself. When it comes to web design, there are a few staples (Don't Make Me Think comes to mind), the books don't seem to be specific enough. They usually just show good examples, but they don't show the why of the good examples. Designing User Interfaces starts with the why, which is something I can really appreciate.

As a developer, I also have a tendency to think in "design patterns." By presenting the interfaces as "patterns," it was easy for me to start thinking up my own variations of those design patterns to work with my existing experiences and products. This is probably what sealed the fate of this book. It made me excited to re-design and re-think interfaces I've implemented (including Launchpad).

I would highly recommend this book for anyone doing front end work, whether it's just implementing existing designs or creating your own designs. I've noticed that I now critique web interfaces with an eye I didn't have before I read this book.

launchpadlib powerup

2010-03-11T05:59:39+00:00

In my last post, I introduced launchpadlib and demonstrated a very simple script that uses it. In this post, I'd like to build on that a bit and show you how to do something actually interesting.

In particular, I want to show you how to search for bugs, teach you a bit about Launchpad's internal data model and help you help yourself when it comes to figuring out Launchpad APIs.

The script at lp:~jml/+junk/bugstats is designed to tell you how good you are at filing bugs. It uses a very simple metric: out of the bugs that you've filed, how many actually have been fixed.

$ ./bugstats.py ubuntu jml
jml is 22.22% successful on bugs in Ubuntu
$ ./bugstats.py launchpad-code jml
jml is 47.63% successful on bugs in Launchpad Code

To do that, we need to:

get the "project" and person referred to on the command line
search for all fixed bugs filed by that person
search for all bugs in total by that same person
count them both
divide them
print them!

I say "project", but I really should say "pillar", which is the Launchpad technical term for a project (e.g. "bzr"), distribution (e.g. "ubuntu") or project group (e.g. "gnome"). A pillar is anything in first part of Launchpad URL that isn't a person.

We get the pillar and person like this:

   pillar = launchpad.projects[pillar_name]
  reporter = launchpad.people[reporter_name]

Pretty easy, huh? Now, how do we search for bug tasks?

The first port of call is to go to the Launchpad API reference page. I'm going to look for the string 'reporter', since that's the one thing I definitely know I want to find.

Eventually, I found the searchTasks method (named operation) that's on pillars and takes a bug_reporter parameter and a status parameter. It returns a collection of bug_tasks, which are the objects that represent the rows in the table you see at the top of a bug page.

I can find the bugtasks for the bugs I've reported that have been fixed by doing:

   fixed_bugtasks = pillar.searchTasks(
      bug_reporter=reporter, status=['Fix Released'])

It took me a while to figure out exactly how to spell "Fix Released". I ended up using trial and error.

Similarly, I can all the bugtasks for bugs I've filed by doing:

   total_bugtasks = pillar.searchTasks(
      bug_reporter=reporter,
      status=[
          "New",
          "Incomplete",
          "Invalid",
          "Won't Fix",
          "Confirmed",
          "Triaged",
          "In Progress",
          "Fix Committed",
          'Fix Released'])

I cheated a bit for that one and looked at the launchpad code to get a list of all bug statusus. The default for searchTasks is to only return open bugs.

Once we've got the collections of bug tasks, we need to get their counts. In an ideal world, it would be len(total_bugtasks), but sadly bug 274074 means that len is really, really slow here.

Instead, I wrote this helper function:

def length(collection):
  # XXX: Workaround bug 274074. Thanks wgrant.
  return int(collection._wadl_resource.representation['total_size'])

With that, I can calculate & print my success rate at filing bugs:

   percentage = 100.0 * length(fixed_bugtasks) / length(total_bugtasks)
  print "%s is %.2f%% successful on bugs in %s" % (
      reporter.display_name, percentage, pillar.display_name)

Next up on the API, I'll talk about some of the gotchas and what you can do about them.

Get started with launchpadlib

2010-03-10T21:43:38+00:00

In my spare time, I sometimes talk to people about how they can get started with launchpadlib hacking.

launchpadlib is the Python client-side library that talks to Launchpad's own REST API. It turns out that customize scripted control of a bug-tracker-code-hosting-translation-distribution-building-cross-project-collaboration thing is actually quite handy.

If you want to get started hacking with launchpadlib, and you have Ubuntu, then install 'python-launchpadlib' now. I'm pretty sure you can also get it from PyPI.

You can check that it works by running:

$ python
>>> import launchpadlib
>>> launchpadlib.__version__
'1.5.1'

I'll be assuming you're running 1.5.1 or later.

I've written a very simple launchpadlib application that you can get with 'bzr branch lp:~jml/+junk/bugstats'. Each revision shows a meaningful launchpadlib script. You can get at the old revisions with 'bzr revert -r1' or 'bzr revert -r2' or '-r3'.

Here's what the simplest launchpadlib script that I could think of looks like:

import os
import sys
from launchpadlib.launchpad import Launchpad, STAGING_SERVICE_ROOT

APP_NAME = 'jml-bug-stats'
CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache')
SERVICE_ROOT = STAGING_SERVICE_ROOT

launchpad = Launchpad.login_with(APP_NAME, SERVICE_ROOT, CACHE_DIR)
print launchpad.bugs[1].title

(Adapted from r2 of the above branch).

A few points.

We use STAGING_SERVICE_ROOT, which means that we're pointing
at Launchpad's staging service,
just in case we screw up any data.

We give the application a name, when you run the application, launchpadlib
opens up a browser window letting you decide how far the application
can act on your behalf.
We provide a cache directory. Credentials, among other things, get stored
here.
We then login and get an object that represents a Launchpad instance
Once we've got it, we look at the collection of bugs, get Bug #1 and then
print the title

Very simple. To learn how to write this application, I looked at the main Launchpad API help page, the
examples page and the reference documentation. You'll notice that I had to translate the reference documentation from REST-speak into Python-speak.

Already you have enough to go exploring with the Launchpad API and think of cool things to do. A bunch of people are already doing cool stuff and there are many projects that use launchpadlib.

Next up, I hope to show you some more complex things you can do with the API.

Monitor & keyboard

2010-03-08T23:27:54+00:00

I want to get a shiny new monitor and keyboard. The monitor is for coding, writing docs, answering emails and watching films. The keyboard is for same. They'll both plug into my Thinkpad X200 for now.

I'm tempted to get an Apple Cinema Display (but at such cost!). I don't know what keyboard I want. I think that the DAS keyboard I have isn't it.

Thoughts?

Curiosities for March 2010

2010-03-08T05:37:52+00:00

I bought lemonade and a granola bar from a sidewalk lemonade stand yesterday. Yes, spring is here. This (hopefully) means less computery time and more outsidery time. Yes, I just made up the word "outsidery." Coining new words should be part of my curiosities this month...

As for actual things I want to explore:

Jetpack - Last month I switched back to Firefox from Chrome because Chrome just didn't hack it for me as a web developer. One thing that Chrome had that I really liked was the extension mechanism. Basically. it was just some javascript. Jetpack is an extension that provides similar support for Firefox. After diving into Firefox extensions a few times and needing a lifeguard to get out, I thought I'd explore Jetpack to create simpler Firefox extensions.

Android SDK - I offered to help a friend create an Android App. No pay was involved, but I have been provided a more recent Android phone to replace my Android Dev Phone, so I actually have something of an obligation to complete and app for Android as opposed to just poking around in it.

Light: the new look of Ubuntu

2010-03-04T19:26:13+00:00

Jono Bacon, Alan Pope, and many others have written, yesterday we published a new visual story and style for Ubuntu. The core design work was lead by Marcus Haslam, Otto Greenslade and Dominic Edmunds, who are the three visual artists leading our efforts in the Canonical Design team. Once we had the base ideas in place we invited some anchor members of the Ubuntu Art community to a design sprint, to test that the concept had the legs to work with the full range of forums, websites, derivatives and other pieces of this huge and wonderful project. And apparently, it does!

Here are some additional thoughts.

Embracing both Ubuntu and Canonical

One of the real challenges for us has been to find a branding and design strategy which spans the spectrum of audiences, forums and dialogues that we cover. With Ubuntu, it’s my specific dream to find a constructive blend of commercial and community interests, not only for Canonical but for other companies. That has made our design and branding work difficult – the distinctive look of Ubuntu lent itself well to pure community messaging, but it was hard to do a brochure for Canonical data center services for Ubuntu on servers. We have not only Ubuntu, but also Kubuntu and an important range of derivatives that all have a role in our ecosystem.

So we spent a lot of time trying to distill the requirements down into a set of three dimensions:

We found a set of ideas which each represent those spectrums, and which work together.

For example, we identified a palette which includes both a fresh, lively Orange, and a rich, mature Aubergine, which work together. The use of Aubergine indicates Commercial involvement of one form or another, while Orange is a signal of community engagement. The Forums will use the Orange elements more strongly, and a formal product brochure, with descriptions of supporting services, would use more of the Aubergine.

On the consumer/enterprise spectrum, we took inspiration from the aerospace industry, and identified a texture of closely spaced dots. When you see more of that, it means we’re signalling that the story is more about the enterprise, less of that, and it’s more about the consumer. Of course, there are cross-overs, for example when we are talking about the corporate desktop, where we’ll use that closely space dot texture as a boundary area, or separator. We also identified shades of Aubergine that are more consumer, or more enterprise – the darker shades mapping to a stronger emphasis on enterprise work.

And on the end-user / engineer spectrum, we took inspiration from graph paper and engineering blue prints. When you see widely spaced patterns of dots, or outline images and figures, that’s signalling that the content is more engineering-oriented than end-user oriented.

And finally, we found a number of themes which enhanced and echoed those ideas. We use a warm gray supporting colour to give shape to pages and documents, and we built on the dots and circles to create a whole style for figures, illustrations and pictograms.

The beauty of this is that we can now publish content that spans the full range, and we generally know when we start the design process what sorts of visual cues we want to be signalling. Instead of having these different mental domains fight with one another, we can now convey quite subtle collaboration between community and corporate, or work which is aimed at engineers and developers from enterprises as opposed to developers working with consumers. Time will tell how it shapes up, but for now I’m celebrating the milestone and the efforts of the team that pulled it together. There’s something there for everyone who wants to participate in the great hubbub of Ubuntuness that is our shared experience of free software.

So, for example, here’s a conference banner. The strong use of Aubergine suggests that it’s more corporate messaging (Canonical is heavily involved). Orange is used here more as a highlight. The Aubergine is darker, and there’s quite a lot of the fine dot pattern. Below the image is a set of scales showing where on those spectra this work is pitched.

As another example, here’s a brochure with an emphasis on end-users who are thinking about adopting Ubuntu’s cloud infrastructure. Again, the fine dot patterns suggests a more enterprise focus, as does the use of the dark aubergine. You can see the circle metaphor used in the quote callout.

And here’s a similar brochure, but with a more developer or engineering oriented focus: note the use of the graph-paper theme with wide spaced dots, and outline shapes.

Finally, here’s an example of a brochure and CD cover for Ubuntu:

As you can see the idea is to signal a mix of both community and Canonical involvement in the message, addressing consumer audiences with a mix of developers and end-users.

A new Ubuntu font

We have commissioned a new font to be developed both for the logo’s of Ubuntu and Canonical, and for use in the interface. The font will be called Ubuntu, and will be a modern humanist font that is optimised for screen legibility. It will be published under an open font license, and considered part of the trade dress of Ubuntu, which will limit its relevance for software interfaces outside of Ubuntu but leave it free for use across the web and in printed documents.

It will take a few months for the font to be finalised, initial elements will be final in the next week which will be sufficient for the logo and other bits and pieces, but I expect to see that font widely used in 10.10. The work has been commissioned from world-renowned fontographers Dalton Maag, who have expressed excitement at the opportunity to publish an open font and also a font that they know will be used daily by millions of people.

Initial coverage will be Western, Arabic, Hebrew and Cyrillic character sets, but over time we may be able to extend that to being a full Unicode font, with great kerning and hinting for print and screen usage globally. We are considering an internship program, to support aspiring fontographers from all corners of the world to visit London and work with Dalton Maag to extend the font to their own regional glyph set.

The critical test of the font is screen efficiency and legibility, and its character and personality are secondary to its fitness for that purpose. Nevertheless, our hope is that the font has a look that is elegant and expresses the full set of values for both Canonical and Ubuntu: adroitness, accountability, precision, reliability, freedom and collaboration. We’ll publish more as soon as we have it.

A good start

It’s been an exciting process, but I have the sense that we are just getting started. The language will get richer, we will find new things that we want to communicate, and new treatments and visual themes that resonate well with these starting points. We’ll find new ways to integrate this on the web, and on the desktop (look out for the two new themes, Radiance and Ambiance). I hope we’ll see the language being used to good effect across everything we do, both commercial and community oriented. There’s a range of expression here that should be useful to artists across the spectrum. Let me know how it works for you.

Free Software is a Pyramid Scheme

2010-03-04T00:29:04+00:00

Last week, I was asked to meet with a family member for lunch. I went to venue only to find that the family member was joined by other people. Those other people wanted me to get involved in network marketing. People that aren't in network marketing often refer to network marketing as a "pyramid scheme." Yeah, one of those, you know, where you alienate your family and friends trying to get them to not only buy a product but to have them sell the product as well. I decided to decline the offer mostly because I'm a geek, not a salesman. I can't schmooze people; I'm direct and to the point.

On my bike ride home from that appointment though, I realized that I am a salesman, and I think I'm a pretty good one. The only difference is that I can only "sell" the things I truly believe in. One of those things is Free Software.

In 1999, I was a sophomore/junior in high school in a small podunk town. I don't remember a lot of my life then, but I do remember the computer lab high school. It had a T1 (lightning fast at the time) and hardly anyone at the school knew anything about computers. I guess you could say I was the man with one eye in the land of the blind.

With all that unused bandwidth, I had no choice but to start pirating software hand over fist. I stumbled over a piece of software called "Red Hat Linux" which sounded really geeky cool, and I had access to downloading all 8 cds (or whatever it was). I was such a pirate for downloading all those cds. Man, I was cool.

Installing it was a pain, eventually leading to me getting a book called "Linux for Dummies" before I really knew what to do. I played around a bit, but didn't really see the appeal of Linux over Windows. There was nothing like Visual Basic (a habit I'm ashamed to admit I developed during high school). I also wasn't a fan of the terminal.

2001 came and I found myself buying a copy of Mandrake 8.1 from Best Buy. It was sexier than Red Hat, and seemed to have everything I wanted. Unfortunately, I couldn't get my sound card to work, and every time I wanted to install something that wasn't an rpm from the cd, I had to go build all sorts of dependencies from source. It was hopeless, but I trudged on for the geek cred.

That same year, I discovered Debian because someone else I knew was using it. Debian was cool, because it had this apt-get thing that would help you install things magically. At this point, I took off. Installing things was so easy that I was always trying out new software. I was showing it to other people. I'd see people using a piece of Windows software and start talking about the Free version I got from apt, and how much better it was.

As time past, I started seeing the benefit of Free Software in everything. I started talking to people about how they needed to embrace this idea of Free Software. There was light in my eyes when I told people they didn't need to be slaves to software vendors; they had choice in how their computer worked.

Over the years, I've helped many people convert their lifestyles to use more Free Software in some capacity, whether it be through the use of OpenOffice, or the big jump to Linux. It's been rewarding and eductional for me, and it's even better when those that I've helped turn around to help others.

So, if I put this blog post to cheesy music on a DVD, sat you down and made you watch it, would you join the Free Software Pyramid Scheme? What if I promised you that if you worked as hard as I did, you would make as much money as I have convincing people to use Free Software?

Back from PyCon

2010-03-03T20:59:13+00:00

I had a great time at PyCon 2010. The best part was definitely the Twisted sprint, which gave me a long overdue opportunity to do some actual coding.

Well, actually, I ended up spending a lot of the time organizing a release and writing process documents, so I didn't escape management as much as I would have liked.

Now I'm back, planning on doing the Twisted 10.0 release and working with didrocks to get better Launchpad integration with Quickly, as my contribution to Project Awesome Opportunity.

Bazaar Explorer’s Killer Feature

2010-03-01T06:33:10+00:00

Bazaar Explorer 1.0.0 hits the streets yesterday. It’s now well entrenched in my Top 5 applications along with Firefox, Thunderbird and gvim – I truly love using it. Some user docs are coming but in the meantime, here’s a quick look at my favorite feature … project-specific tools.

The first time I opened a repository called “bzr”, Explorer guessed that I was working on the core Bazaar project and asked if I wanted to use the bzr “Hat”. I replied Yes. Now, every time I open a branch in that repository, my toolbox magically gets a Bazaar Development section as shown below.

The first 3 actions take me to the Open Questions, New Bugs and Active Code Reviews for bzr on Launchpad. That lets me reply to support questions, triage new issues and review merge proposals. Clicking on Servers pops up a menu of bzr-related web sites I visit from time to time, the PQM bot that merges proposed changes to our mainline (if and only if all tests pass) and our Hudson continuous integration server (that runs our regression test suite on lots of different operating systems).

Likewise, Docs pops up a menu of documentation I commonly use …

BTW, you don’t need to be using a shared repository for Explorer to propose using a Hat for given location and its subdirectories. Opening a branch or checkout with the right name will work as well. If you don’t want to use a Hat for a location, that’s equally fine. Explorer will remember that fact and not ask you again.

This feature isn’t only for “bzr” development either. Explorer ships with around 20 hats for a variety of popular open source projects on Launchpad including MySQL, Inkscape, Gwibber, Mailman, GNOME Do and Launchpad itself. You can easily create your own as well by simply adding a hats/project-name/tools.xml file under your explorer configuration directory (e.g. ~/.bazaar/explorer). Here’s a sample tools.xml file:

<folder title="Tools"> <separator/> <folder title="QBzr Development"> <toolset name="lp-devel" project="qbzr" /> </folder> </folder>
It might just be me but I find project-specific tools mega cool: fast access to the places I need to visit, anytime I’m working on a particular project. Explorer is and will remain an easy-to-use GUI application suitable for casual users of version control. My grand vision though is for it to evolve into a collaboration-centric, BYO editor IDE, suitable for hard-code developers like myself to spend most of their day in. Keeping the easy-vs-powerful balance right won’t be easy but I’m confident we can do it. Give it a try and let us know how it works for you!

Ian ClatworthyThe ToolboxServers for bzrDocs for bzr

February 2010 Exploration Report

2010-03-01T02:48:44+00:00

Earlier this month, I posted about some technologies that I was curious about. Here's my report:

Tahoe-LAFS

I actually like the Tahoe community a lot; enough for me to want to contribute to it. Tahoe 1.6 was my first sponsored package into Lucid earlier this month, and I'd really like to get together with other local Coloradoans and sprint on Tahoe.

If you just want to see what Tahoe provides, you'll want to check out the Test Grid. It gives you a nice web interface to explore Tahoe. You can create directories, upload files, etc.

The web interface wasn't enough for me. I installed Tahoe, and then made a few KVM virtual machines and set up my own test grid (albeit a crippled version, as I was lazy and only wanted to set up the parts I to play with). My installation had some hiccups, but was incredibly happy to find that my tahoe mv and tahoe cp were working.

Being a web developer by day, I think the web interface could use a little more love, and could be a bit sexier, but I suspect that the web interface isn't the main way to add things to a Tahoe installation.

I'd like to find some ways to contribute to Tahoe in the future.

Node.js

Node.js is a cute idea. It's a web server for server side javascript. It may be a little perverted to say, but I really love javascript. Sure, it has its bad parts, but the good parts are pretty good. Naturally, this meant that I was curious to write server-side javascript.

As I was working on it, I started writing a little web app. I was handling post data and stuff. I was returning html. Pretty soon I was writing a templating engine and basically re-implementing Django/Zope/Rails/Your-Web-Framework-of-Choice. This really put me off, because there are better existing tools to build web apps than building something from scratch with Javascript (no matter how much I like Javascript).

One thing I thought was funny was that my server side javascript was sending javascript through in the response for the client to work with. This is a phonomenon I'd never encountered before: using the same language on the server and the client. It made me wonder what having python in the browser would be like.

I think I might play with Node.js again when I don't have to create an entire framework around it.

Announcing the NCLUG/BLUG Blog Planet

2010-03-01T02:46:35+00:00

A few weeks back I pitched the idea of aggregating the blogs of members from the Northern Colorado Linux Users Group and the Boulder Linux Users Group to their accompanying mailing lists. Those members responded to me with interest, so I put together an aggregate site.

With some help from Sean, NCLUG, and Tummy.com (for hosting the NCLUG sites), I'm happy to announce the Northern Colorado Linux Blogs.

If you're in the NCLUG or BLUG area and want your blog aggregated, feel free to send an email to paul [at] eventuallyanyway [dot] com.

Twisted 10.0.0pre1 released

2010-02-22T13:59:52+00:00

It's not going to work, it's not the final release, but I'm very pleased to announce that Twisted 10.0.0pre1 is available for testing.

Download it, test it, play with it and help us make 10.0 the best release ever!

On a side note, this is hopefully the start of a much simpler, more automated release process and maybe even time-based releases.

Trivial bugs

2010-02-22T07:45:06+00:00

This is just a quick note really. One thing I've been trying to do more and more is to address simple bugs in a more timely manner.

I use the tag "trivial" to indicate to me that the bug is very simple to fix. By this I mean that I should be able to have the fix and the test all written in under an hour, and normally under 30 minutes.

Personally I'm (hopefully) fixing one trivial bug a day in addition to other work. This way the simple bugs get some attention, and I get the feeling of accomplishing something when other things are in the pipeline that take longer to get completed.

My scheduling of trivial bugs is somewhat arbitrary. Often the most recently commented on trivial bug will get my attention.

Yay Dell-with-Ubuntu down under

2010-02-15T06:12:02+00:00

Dell has been offering Ubuntu on selected models for a while. I had however nearly given up hope on being able to buy one, because they hadn’t started doing that in Australia. I am very glad to see this has changed though – check out their notebook page. Not all models yet, but a reasonable number have Ubuntu as an option.

Yay!