Planet Bazaar

March 11, 2010

Jonathan Lange

Have you tried lptools?

Have you tried lptools?

It's not at all officially associated with the Launchpad project, of course, but it's got a few nice things that you might want to look at, including:
  • a milestone manipulator
  • a code review notifier, and
  • a milestone to iCal exporter
One of these days, I'd like for there to be an official, awesome Launchpad command-line client. As it is, I'm happy with the world of people making extensions to meet their needs.

For other extensions, check out the lpx project or our list of clients.

by jml (noreply@blogger.com) at March 11, 2010 08:15 PM

launchpadlib powerup

In my last post, I introduced launchpadlib and demonstrated a very simple script that uses it. In this post, I'd like to build on that a bit and show you how to do something actually interesting.

In particular, I want to show you how to search for bugs, teach you a bit about Launchpad's internal data model and help you help yourself when it comes to figuring out Launchpad APIs.

The script at lp:~jml/+junk/bugstats is designed to tell you how good you are at filing bugs. It uses a very simple metric: out of the bugs that you've filed, how many actually have been fixed.
$ ./bugstats.py ubuntu jml
jml is 22.22% successful on bugs in Ubuntu
$ ./bugstats.py launchpad-code jml
jml is 47.63% successful on bugs in Launchpad Code
To do that, we need to:
  1. get the "project" and person referred to on the command line
  2. search for all fixed bugs filed by that person
  3. search for all bugs in total by that same person
  4. count them both
  5. divide them
  6. print them!
I say "project", but I really should say "pillar", which is the Launchpad technical term for a project (e.g. "bzr"), distribution (e.g. "ubuntu") or project group (e.g. "gnome"). A pillar is anything in first part of Launchpad URL that isn't a person.

We get the pillar and person like this:
   pillar = launchpad.projects[pillar_name]
reporter = launchpad.people[reporter_name]
Pretty easy, huh? Now, how do we search for bug tasks?

The first port of call is to go to the Launchpad API reference page. I'm going to look for the string 'reporter', since that's the one thing I definitely know I want to find.

Eventually, I found the searchTasks method (named operation) that's on pillars and takes a bug_reporter parameter and a status parameter. It returns a collection of bug_tasks, which are the objects that represent the rows in the table you see at the top of a bug page.

I can find the bugtasks for the bugs I've reported that have been fixed by doing:
   fixed_bugtasks = pillar.searchTasks(
bug_reporter=reporter, status=['Fix Released'])
It took me a while to figure out exactly how to spell "Fix Released". I ended up using trial and error.

Similarly, I can all the bugtasks for bugs I've filed by doing:
   total_bugtasks = pillar.searchTasks(
bug_reporter=reporter,
status=[
"New",
"Incomplete",
"Invalid",
"Won't Fix",
"Confirmed",
"Triaged",
"In Progress",
"Fix Committed",
'Fix Released'])
I cheated a bit for that one and looked at the launchpad code to get a list of all bug statusus. The default for searchTasks is to only return open bugs.

Once we've got the collections of bug tasks, we need to get their counts. In an ideal world, it would be len(total_bugtasks), but sadly bug 274074 means that len is really, really slow here.

Instead, I wrote this helper function:
def length(collection):
# XXX: Workaround bug 274074. Thanks wgrant.
return int(collection._wadl_resource.representation['total_size'])
With that, I can calculate & print my success rate at filing bugs:
   percentage = 100.0 * length(fixed_bugtasks) / length(total_bugtasks)
print "%s is %.2f%% successful on bugs in %s" % (
reporter.display_name, percentage, pillar.display_name)
Next up on the API, I'll talk about some of the gotchas and what you can do about them.

by jml (noreply@blogger.com) at March 11, 2010 05:59 AM

March 10, 2010

Jonathan Lange

Get started with launchpadlib

In my spare time, I sometimes talk to people about how they can get started with launchpadlib hacking.

launchpadlib is the Python client-side library that talks to Launchpad's own REST API. It turns out that customize scripted control of a bug-tracker-code-hosting-translation-distribution-building-cross-project-collaboration thing is actually quite handy.

If you want to get started hacking with launchpadlib, and you have Ubuntu, then install 'python-launchpadlib' now. I'm pretty sure you can also get it from PyPI.

You can check that it works by running:
$ python
>>> import launchpadlib
>>> launchpadlib.__version__
'1.5.1'

I'll be assuming you're running 1.5.1 or later.

I've written a very simple launchpadlib application that you can get with 'bzr branch lp:~jml/+junk/bugstats'. Each revision shows a meaningful launchpadlib script. You can get at the old revisions with 'bzr revert -r1' or 'bzr revert -r2' or '-r3'.

Here's what the simplest launchpadlib script that I could think of looks like:
import os
import sys
from launchpadlib.launchpad import Launchpad, STAGING_SERVICE_ROOT

APP_NAME = 'jml-bug-stats'
CACHE_DIR = os.path.expanduser('~/.launchpadlib/cache')
SERVICE_ROOT = STAGING_SERVICE_ROOT

launchpad = Launchpad.login_with(APP_NAME, SERVICE_ROOT, CACHE_DIR)
print launchpad.bugs[1].title
(Adapted from r2 of the above branch).

A few points.

  • We use STAGING_SERVICE_ROOT, which means that we're pointing
    at Launchpad's staging service,
    just in case we screw up any data.

  • We give the application a name, when you run the application, launchpadlib
    opens up a browser window letting you decide how far the application
    can act on your behalf.
  • We provide a cache directory. Credentials, among other things, get stored
    here.
  • We then login and get an object that represents a Launchpad instance
  • Once we've got it, we look at the collection of bugs, get Bug #1 and then
    print the title
Very simple. To learn how to write this application, I looked at the main Launchpad API help page, the
examples page and the reference documentation. You'll notice that I had to translate the reference documentation from REST-speak into Python-speak.

Already you have enough to go exploring with the Launchpad API and think of cool things to do. A bunch of people are already doing cool stuff and there are many projects that use launchpadlib.

Next up, I hope to show you some more complex things you can do with the API.

by jml (noreply@blogger.com) at March 10, 2010 09:43 PM

March 08, 2010

Jonathan Lange

Monitor & keyboard

I want to get a shiny new monitor and keyboard. The monitor is for coding, writing docs, answering emails and watching films. The keyboard is for same. They'll both plug into my Thinkpad X200 for now.

I'm tempted to get an Apple Cinema Display (but at such cost!). I don't know what keyboard I want. I think that the DAS keyboard I have isn't it.

Thoughts?

by jml (noreply@blogger.com) at March 08, 2010 11:27 PM

Paul Hummer

Curiosities for March 2010

I bought lemonade and a granola bar from a sidewalk lemonade stand yesterday. Yes, spring is here. This (hopefully) means less computery time and more outsidery time. Yes, I just made up the word "outsidery." Coining new words should be part of my curiosities this month...

As for actual things I want to explore:

  • Jetpack - Last month I switched back to Firefox from Chrome because Chrome just didn't hack it for me as a web developer. One thing that Chrome had that I really liked was the extension mechanism. Basically. it was just some javascript. Jetpack is an extension that provides similar support for Firefox. After diving into Firefox extensions a few times and needing a lifeguard to get out, I thought I'd explore Jetpack to create simpler Firefox extensions.
  • Android SDK - I offered to help a friend create an Android App. No pay was involved, but I have been provided a more recent Android phone to replace my Android Dev Phone, so I actually have something of an obligation to complete and app for Android as opposed to just poking around in it.

March 08, 2010 06:37 AM

March 04, 2010

Mark Shuttleworth

Light: the new look of Ubuntu

Jono Bacon, Alan Pope, and many others have written, yesterday we published a new visual story and style for Ubuntu. The core design work was lead by Marcus Haslam, Otto Greenslade and Dominic Edmunds, who are the three visual artists leading our efforts in the Canonical Design team. Once we had the base ideas in place we invited some anchor members of the Ubuntu Art community to a design sprint, to test that the concept had the legs to work with the full range of forums, websites, derivatives and other pieces of this huge and wonderful project. And apparently, it does!

Here are some additional thoughts.

Embracing both Ubuntu and Canonical

One of the real challenges for us has been to find a branding and design strategy which spans the spectrum of audiences, forums and dialogues that we cover.  With Ubuntu, it’s my specific dream to find a constructive blend of commercial and community interests, not only for Canonical but for other companies. That has made our design and branding work difficult – the distinctive look of Ubuntu lent itself well to pure community messaging, but it was hard to do a brochure for Canonical data center services for Ubuntu on servers. We have not only Ubuntu, but also Kubuntu and an important range of derivatives that all have a role in our ecosystem.

So we spent a lot of time trying to distill the requirements down into a set of three dimensions:

Dimensions for our visual language

We found a set of ideas which each represent those spectrums, and which work together.

For example, we identified a palette which includes both a fresh, lively Orange, and a rich, mature Aubergine, which work together. The use of Aubergine indicates Commercial involvement of one form or another, while Orange is a signal of community engagement. The Forums will use the Orange elements more strongly, and a formal product brochure, with descriptions of supporting services, would use more of the Aubergine.

On the consumer/enterprise spectrum, we took inspiration from the aerospace industry, and identified a texture of closely spaced dots. When you see more of that, it means we’re signalling that the story is more about the enterprise, less of that, and it’s more about the consumer. Of course, there are cross-overs, for example when we are talking about the corporate desktop, where we’ll use that closely space dot texture as a boundary area, or separator. We also identified shades of Aubergine that are more consumer, or more enterprise – the darker shades mapping to a stronger emphasis on enterprise work.

And on the end-user / engineer spectrum, we took inspiration from graph paper and engineering blue prints. When you see widely spaced patterns of dots, or outline images and figures, that’s signalling that the content is more engineering-oriented than end-user oriented.

And finally, we found a number of themes which enhanced and echoed those ideas. We use a warm gray supporting colour to give shape to pages and documents, and we built on the dots and circles to create a whole style for figures, illustrations and pictograms.

The beauty of this is that we can now publish content that spans the full range, and we generally know when we start the design process what sorts of visual cues we want to be signalling. Instead of having these different mental domains fight with one another, we can now convey quite subtle collaboration between community and corporate, or work which is aimed at engineers and developers from enterprises as opposed to developers working with consumers. Time will tell how it shapes up, but for now I’m celebrating the milestone and the efforts of the team that pulled it together. There’s something there for everyone who wants to participate in the great hubbub of Ubuntuness that is our shared experience of free software.

So, for example, here’s a conference banner. The strong use of Aubergine suggests that it’s more corporate messaging (Canonical is heavily involved). Orange is used here more as a highlight. The Aubergine is darker, and there’s quite a lot of the fine dot pattern. Below the image is a set of scales showing where on those spectra this work is pitched.

Cloud Banner

As another example, here’s a brochure with an emphasis on end-users who are thinking about adopting Ubuntu’s cloud infrastructure. Again, the fine dot patterns suggests a more enterprise focus, as does the use of the dark aubergine. You can see the circle metaphor used in the quote callout.

And here’s a similar brochure, but with a more developer or engineering oriented focus: note the use of the graph-paper theme with wide spaced dots, and outline shapes.

Finally, here’s an example of a brochure and CD cover for Ubuntu:

As you can see the idea is to signal a mix of both community and Canonical involvement in the message, addressing consumer audiences with a mix of developers and end-users.

A new Ubuntu font

We have commissioned a new font to be developed both for the logo’s of Ubuntu and Canonical, and for use in the interface. The font will be called Ubuntu, and will be a modern humanist font that is optimised for screen legibility. It will be published under an open font license, and considered part of the trade dress of Ubuntu, which will limit its relevance for software interfaces outside of Ubuntu but leave it free for use across the web and in printed documents.

It will take a few months for the font to be finalised, initial elements will be final in the next week which will be sufficient for the logo and other bits and pieces, but I expect to see that font widely used in 10.10. The work has been commissioned from world-renowned fontographers Dalton Maag, who have expressed excitement at the opportunity to publish an open font and also a font that they know will be used daily by millions of people.

Initial coverage will be Western, Arabic, Hebrew and Cyrillic character sets, but over time we may be able to extend that to being a full Unicode font, with great kerning and hinting for print and screen usage globally.  We are considering an internship program, to support aspiring fontographers from all corners of the world to visit London and work with Dalton Maag to extend the font to their own regional glyph set.

The critical test of the font is screen efficiency and legibility, and its character and personality are secondary to its fitness for that purpose. Nevertheless, our hope is that the font has a look that is elegant and expresses the full set of values for both Canonical and Ubuntu: adroitness, accountability, precision, reliability, freedom and collaboration. We’ll publish more as soon as we have it.

A good start

It’s been an exciting process, but I have the sense that we are just getting started. The language will get richer, we will find new things that we want to communicate, and new treatments and visual themes that resonate well with these starting points. We’ll find new ways to integrate this on the web, and on the desktop (look out for the two new themes, Radiance and Ambiance).  I hope we’ll see the language being used to good effect across everything we do, both commercial and community oriented. There’s a range of expression here that should be useful to artists across the spectrum. Let me know how it works for you.

by mark at March 04, 2010 07:26 PM

Paul Hummer

Free Software is a Pyramid Scheme

Last week, I was asked to meet with a family member for lunch. I went to venue only to find that the family member was joined by other people. Those other people wanted me to get involved in network marketing. People that aren't in network marketing often refer to network marketing as a "pyramid scheme." Yeah, one of those, you know, where you alienate your family and friends trying to get them to not only buy a product but to have them sell the product as well. I decided to decline the offer mostly because I'm a geek, not a salesman. I can't schmooze people; I'm direct and to the point.

On my bike ride home from that appointment though, I realized that I am a salesman, and I think I'm a pretty good one. The only difference is that I can only "sell" the things I truly believe in. One of those things is Free Software.

In 1999, I was a sophomore/junior in high school in a small podunk town. I don't remember a lot of my life then, but I do remember the computer lab high school. It had a T1 (lightning fast at the time) and hardly anyone at the school knew anything about computers. I guess you could say I was the man with one eye in the land of the blind.

With all that unused bandwidth, I had no choice but to start pirating software hand over fist. I stumbled over a piece of software called "Red Hat Linux" which sounded really geeky cool, and I had access to downloading all 8 cds (or whatever it was). I was such a pirate for downloading all those cds. Man, I was cool.

Installing it was a pain, eventually leading to me getting a book called "Linux for Dummies" before I really knew what to do. I played around a bit, but didn't really see the appeal of Linux over Windows. There was nothing like Visual Basic (a habit I'm ashamed to admit I developed during high school). I also wasn't a fan of the terminal.

2001 came and I found myself buying a copy of Mandrake 8.1 from Best Buy. It was sexier than Red Hat, and seemed to have everything I wanted. Unfortunately, I couldn't get my sound card to work, and every time I wanted to install something that wasn't an rpm from the cd, I had to go build all sorts of dependencies from source. It was hopeless, but I trudged on for the geek cred.

That same year, I discovered Debian because someone else I knew was using it. Debian was cool, because it had this apt-get thing that would help you install things magically. At this point, I took off. Installing things was so easy that I was always trying out new software. I was showing it to other people. I'd see people using a piece of Windows software and start talking about the Free version I got from apt, and how much better it was.

As time past, I started seeing the benefit of Free Software in everything. I started talking to people about how they needed to embrace this idea of Free Software. There was light in my eyes when I told people they didn't need to be slaves to software vendors; they had choice in how their computer worked.

Over the years, I've helped many people convert their lifestyles to use more Free Software in some capacity, whether it be through the use of OpenOffice, or the big jump to Linux. It's been rewarding and eductional for me, and it's even better when those that I've helped turn around to help others.

So, if I put this blog post to cheesy music on a DVD, sat you down and made you watch it, would you join the Free Software Pyramid Scheme? What if I promised you that if you worked as hard as I did, you would make as much money as I have convincing people to use Free Software?

March 04, 2010 01:29 AM

March 03, 2010

Jonathan Lange

Back from PyCon

I had a great time at PyCon 2010. The best part was definitely the Twisted sprint, which gave me a long overdue opportunity to do some actual coding.

Well, actually, I ended up spending a lot of the time organizing a release and writing process documents, so I didn't escape management as much as I would have liked.

Now I'm back, planning on doing the Twisted 10.0 release and working with didrocks to get better Launchpad integration with Quickly, as my contribution to Project Awesome Opportunity.

by jml (noreply@blogger.com) at March 03, 2010 08:59 PM

March 01, 2010

Bazaar Developers

Bazaar Explorer’s Killer Feature

Bazaar Explorer 1.0.0 hits the streets yesterday. It’s now well entrenched in my Top 5 applications along with Firefox, Thunderbird and gvim – I truly love using it. Some user docs are coming but in the meantime, here’s a quick look at my favorite feature … project-specific tools.

The first time I opened a repository called “bzr”, Explorer guessed that I was working on the core Bazaar project and asked if I wanted to use the bzr “Hat”. I replied Yes. Now, every time I open a branch in that repository, my toolbox magically gets a Bazaar Development section as shown below.

The Toolbox

The first 3 actions take me to the Open Questions, New Bugs and Active Code Reviews for bzr on Launchpad. That lets me reply to support questions, triage new issues and review merge proposals. Clicking on Servers pops up a menu of bzr-related web sites I visit from time to time, the PQM bot that merges proposed changes to our mainline (if and only if all tests pass) and our Hudson continuous integration server (that runs our regression test suite on lots of  different operating systems).

Servers for bzr

Likewise, Docs pops up a menu of documentation I commonly use …

Docs for bzr

BTW, you don’t need to be using a shared repository for Explorer to propose using a Hat for given location and its subdirectories. Opening a branch or checkout with the right name will work as well. If you don’t want to use a Hat for a location, that’s equally fine. Explorer will remember that fact and not ask you again.

This feature isn’t only for “bzr” development either. Explorer ships with around 20 hats for a variety of popular open source projects on Launchpad including MySQL, Inkscape, Gwibber, Mailman, GNOME Do and Launchpad itself. You can easily create your own as well by simply adding a hats/project-name/tools.xml file under your explorer configuration directory (e.g. ~/.bazaar/explorer). Here’s a sample tools.xml file:


<folder title="Tools">
<separator/>
<folder title="QBzr Development">
<toolset name="lp-devel" project="qbzr" />
</folder>
</folder>

It might just be me but I find project-specific tools mega cool: fast access to the places I need to visit, anytime I’m working on a particular project. Explorer is and will remain an easy-to-use GUI application suitable for casual users of version control. My grand vision though is for it to evolve into a collaboration-centric, BYO editor IDE, suitable for hard-code developers like myself to spend most of their day in. Keeping the easy-vs-powerful balance right won’t be easy but I’m confident we can do it. Give it a try and let us know how it works for you!


Ian ClatworthyThe ToolboxServers for bzrDocs for bzr

by Ian Clatworthy at March 01, 2010 06:33 AM

Paul Hummer

February 2010 Exploration Report

Earlier this month, I posted about some technologies that I was curious about. Here's my report:

Tahoe-LAFS

I actually like the Tahoe community a lot; enough for me to want to contribute to it. Tahoe 1.6 was my first sponsored package into Lucid earlier this month, and I'd really like to get together with other local Coloradoans and sprint on Tahoe.

If you just want to see what Tahoe provides, you'll want to check out the Test Grid. It gives you a nice web interface to explore Tahoe. You can create directories, upload files, etc.

The web interface wasn't enough for me. I installed Tahoe, and then made a few KVM virtual machines and set up my own test grid (albeit a crippled version, as I was lazy and only wanted to set up the parts I to play with). My installation had some hiccups, but was incredibly happy to find that my tahoe mv and tahoe cp were working.

Being a web developer by day, I think the web interface could use a little more love, and could be a bit sexier, but I suspect that the web interface isn't the main way to add things to a Tahoe installation.

I'd like to find some ways to contribute to Tahoe in the future.

Node.js

Node.js is a cute idea. It's a web server for server side javascript. It may be a little perverted to say, but I really love javascript. Sure, it has its bad parts, but the good parts are pretty good. Naturally, this meant that I was curious to write server-side javascript.

As I was working on it, I started writing a little web app. I was handling post data and stuff. I was returning html. Pretty soon I was writing a templating engine and basically re-implementing Django/Zope/Rails/Your-Web-Framework-of-Choice. This really put me off, because there are better existing tools to build web apps than building something from scratch with Javascript (no matter how much I like Javascript).

One thing I thought was funny was that my server side javascript was sending javascript through in the response for the client to work with. This is a phonomenon I'd never encountered before: using the same language on the server and the client. It made me wonder what having python in the browser would be like.

I think I might play with Node.js again when I don't have to create an entire framework around it.

March 01, 2010 03:48 AM

Announcing the NCLUG/BLUG Blog Planet

A few weeks back I pitched the idea of aggregating the blogs of members from the Northern Colorado Linux Users Group and the Boulder Linux Users Group to their accompanying mailing lists. Those members responded to me with interest, so I put together an aggregate site.

With some help from Sean, NCLUG, and Tummy.com (for hosting the NCLUG sites), I'm happy to announce the Northern Colorado Linux Blogs.

If you're in the NCLUG or BLUG area and want your blog aggregated, feel free to send an email to paul [at] eventuallyanyway [dot] com.

March 01, 2010 03:46 AM

February 22, 2010

Jonathan Lange

Twisted 10.0.0pre1 released

It's not going to work, it's not the final release, but I'm very pleased to announce that Twisted 10.0.0pre1 is available for testing.

Download it, test it, play with it and help us make 10.0 the best release ever!

On a side note, this is hopefully the start of a much simpler, more automated release process and maybe even time-based releases.

by jml (noreply@blogger.com) at February 22, 2010 01:59 PM

Tim Penhey

Trivial bugs

This is just a quick note really. One thing I've been trying to do more and more is to address simple bugs in a more timely manner.

I use the tag "trivial" to indicate to me that the bug is very simple to fix. By this I mean that I should be able to have the fix and the test all written in under an hour, and normally under 30 minutes.

Personally I'm (hopefully) fixing one trivial bug a day in addition to other work. This way the simple bugs get some attention, and I get the feeling of accomplishing something when other things are in the pipeline that take longer to get completed.

My scheduling of trivial bugs is somewhat arbitrary. Often the most recently commented on trivial bug will get my attention.

by Tim Penhey (thumper) (noreply@blogger.com) at February 22, 2010 07:45 AM

February 15, 2010

Robert Collins

Yay Dell-with-Ubuntu down under


Dell has been offering Ubuntu on selected models for a while. I had however nearly given up hope on being able to buy one, because they hadn’t started doing that in Australia. I am very glad to see this has changed though – check out their notebook page. Not all models yet, but a reasonable number have Ubuntu as an option.

Yay!

February 15, 2010 06:12 AM

Bazaar Developers

Bazaar adoption growing strongly

I’ve been tracking the popularity of the leading VCS tools on Ubuntu for the last 4-5 months using popcon. While popcon is far from perfect, I feel the results are a useful data point, given the popularity of Linux among software developers and Ubuntu among Linux distributions.

Here are the summary results.

Tool 19-Oct-2009 15-Feb-2010 Growth
Subversion 247760 282789 14.1%
Git 77791 94441 21.3%
Mercurial 28271 36086 27.5%
Bazaar 39391 51667 31.0%

As expected, all 3 major DVCS tools are growing faster than Subversion in percentage terms. What’s more interesting to me is that Bazaar and Mercurial are growing faster than Git, despite the buzz Git is currently enjoying. As a Bazaar developer, that’s truly awesome news.

Why do I say that? Looking back over technology trends, clean-and-simple products frequently lose the early battle against faster-but-more-complex competitors, e.g. Python vs Perl, GNOME vs KDE. Eventually though, the less complex tools become fast enough and powerful enough to satisfy most needs and their adoption takes off. That’s not to say tools like Perl and KDE are bad. I love them both but find myself using Python and GNOME more frequently these days. In the DVCS marketplace, I’ve always expected Bazaar (and Mercurial) to eventually grow faster than Git. I’m just ecstatic that it seems to be happening already.


Ian Clatworthy

by Ian Clatworthy at February 15, 2010 05:31 AM

February 12, 2010

Bazaar Developers

gwibber developers on Launchpad and Bazaar

Review of Launchpad and Bazaar on ArsTechnica by the lead developer of gwibber.

  • Likes the way the bzr client feeds into the web ui, by setting bug links etc
  • Easy automatic imports from cvs, svn, git and hg, either for a one-shot cutover or continuing tracking
  • More powerful bug tracking than github
  • Loggerhead feels slow and poorly integrated with Launchpad, but qbzr is brilliant
  • Merge proposals good for tracking contributions

mbp

by Martin Pool at February 12, 2010 07:40 PM

February 11, 2010

Robert Collins

Using UEC instead of EC2


So, we wanted to move a Hudson CI server at Canonical from using chroots to VM’s (for better isolation and security), and there is this great product Ubuntu Enterprise Cloud (UEC – basically Eucalyptus). To do this I needed to make some changes to the Hudson EC2 plugin – and thats where the fun starts. While I focus on getting Hudson up and running with UEC in this post, folk generally interested in the differences between UEC and EC2, or getting a single-machine UEC instance up for testing should also find this useful.

Firstly, getting a test UEC instance installed was a little tricky – I only had one machine to deploy it on, and this is an unusual configuration. Nicely though, it all worked, once a few initial bugs and misconfiguration items got fixed up. I wrote up the crux of the outcome on the Ubuntu community help wiki. See ‘1 Physical system’. The particular trap to watch out for seems to be that this configuration is not well tested, so the installation scripts have a hard time getting it right. I haven’t tried to make it play nice with Network Manager in the loop, but I’m pretty sure that that can be done via interface aliasing or something similar.

Secondly I needed to find out what was different between EC2 and UEC (Note that I was running on Karmic (Ubuntu 9.10) – so things could be different in Lucid). I couldn’t find a simple description of this, so this list may be incomplete:

  1. UEC runs an old version of the EC2 API. This is because it hasn’t implemented everything in the new API versions yet.
  2. UEC defaults to port 8773, not port 80 (for both the EC2 and S3 API’s)
  3. The EC2 and S3 API’s are rooted differently: at AWS they are at /, for UEC they are at /services/Eucalyptus and /services/Walrus
  4. UEC doesn’t supply a SSL API port as far as I can tell.
  5. DescribeImages has something wonky with it.

So the next step then is to modify the Hudson EC2 plugin to support these differences. Fortunately it is in Java, and the Java community has already updated the various libraries (jets3t and typica) to support UEC – I just needed to write a UI for the differences and pass the info down the various code paths. Kohsuke has let me land this now even though it has an average UI (in rev 27366), and I’m going to make the UI better now by consolidating all the little aspects into a couple of URL’s. Folk comfortable with building their own .hpi can get this now by svn updating and rebuilding the ec2 plugin. We’ve also filed another bug asking for a single API call to establish the endpoints, so that its even easier for users to set this up.

Finally, and this isn’t a UEC difference, I needed to modify the Hudson EC2 plugin to work with the ubuntu user rather than root, as Ubuntu AMI’s ship with root disabled (as all Ubuntu installs do). I chose to have Hudson reenable root, rather than making everything work without root, because the current code paths assume they can scp things as root, so this was less disruptive.

With all that done, its now possible to configure up a Hudson instance testing via UEC nodes. Here’s how:

  1. Install UEC and make sure you can run up instances using euca-run-instances, ssh into them and that networking works for you. Make sure you have installed at least one image (EMI aka AMI) to run tests on. I used the vanilla in-store UEC Karmic images.
  2. Install Hudson and the EC2 plugin (you’ll need to build your own until a new release (1.6) is made).
  3. Go to /configure and near the bottom click on ‘Add a new cloud’ and choose Amazon EC2.
  4. Look in ~/.euca/eucarc, or in the zip file that the UEC admin web page lets you download, to get at your credentials. Fill in the Access Key and Secret Access key fields accordingly. You can put in the private key (UEC holds onto the public half) that you want to use, or (once the connection is fully setup) use the “Generate Key’ button to have a dedicated Hudson key created. I like to use one that I can ssh into to look at a live node – YMMV. (Or you could add a user and as many keys as you want in the init script – more  on that in a second).
  5. Click on Advanced, this will give you a bunch of details like ‘EC2 Endpoint hostname’. Fill these out.
  6. Sensible values for a default UEC install are: 8773 for both ports, /services/Eucalyptus and /services/Walrus for the base URLs, and SSL turned off. (Note that the online help tells you this as well).
  7. Set an instance cap, unless you truely have unlimited machines. E.g. 5, to run 5 VMs at a time.
  8. Click on ‘Test Connection’ – it should pretty much instantly say ‘Success’.
  9. Thats the Cloud itself configured, now we configure VM’s that Hudson is willing to start. Click on ‘Add’ right above the ‘List of AMIs to be launched as slaves’ text.
  10. Fill out the AMI with your EMI – e.g. emi-E027107D is the Ubuntu 9.10 image I used.
  11. for remote FS root, just put /hudson or something, unless you have a preseeded area (e.g. with a shared bzr repo or something) inside your image.
  12. For description describe the intent of the image – e.g. ‘DB test environment’
  13. For the labels put one or more tags that you will use to tell test jobs they should run on this instance. They can be the same as labels on physical machines – it will act as an overflow buffer. If no physical machines exist, a VM will be spawned when needed. For testing I put ‘euca’
  14. For the init script, its a little more complex. You need to configure up java so that hudson itself can run:
    cat >> /etc/apt/sources.list << EOF
    deb http://archive.ubuntu.com/ubuntu/ karmic multiverse
    deb http://archive.ubuntu.com/ubuntu/ karmic-updates multiverse
    deb http://archive.ubuntu.com/ubuntu/ karmic-security multiverse
    EOF
    export http_proxy=http://192.168.1.1:8080/
    export DEBIAN_FRONTEND=noninteractive
    apt-get update
    echo "buildd shared/accepted-sun-dlj-v1-1 boolean true" | debconf-set-selections
    apt-get install -y -f sun-java6-jre
    

    Note that I have included my local HTTP proxy there – just remove that line if you don’t have one.

  15. Click on Advanced, to get at the less-common options.
  16. For remote user, put ‘ubuntu’, and for root command prefix put ’sudo’.
  17. For number of executors, you are essentially choosing the number of CPU’s that the instance will request. E.g. putting 20 will ask for an extra-large high-cpu model machine when it deploys. This will then show up as 20 workers on the same machine.
  18. Click save :)
  19. Now, when you add a job a new option in the job configuration will appear – ‘tie this job to a node’. Select one of the label(s) you put in for the AMI, and running the job will cause that instance to start up if its not already available.

Note that Hudson will try to use java from s3 if you don’t install it, but that won’t work right for a few reasons – I’ll be filing an issue in the Hudson tracker about it, as thats a bit of unusual structure in the existing code that I’m happier leaving well enough alone :) .

February 11, 2010 09:14 PM

Jelmer Vernooij

Nostalgia: 10 Years of Samba Hacking

While searching for something else I happened to come across one of my first posts to the ntdom list in November 2000.

My post is a simple question about a Samba crash that I myself no doubt had introduced. I'm sure I could have found a solution to it by using Google - excuse me, AltaVista - but I still received a friendly reply from Jerry explaining me to use GDB. I'm not too embarrassed, at least I used proper punctuation and wrote somewhat comprehensible English.

It's also strange to realize it's already been almost ten years since I started hacking on the Samba project.

by nospam@example.com (Jelmer Vernooij) at February 11, 2010 04:32 AM

Jonathan Lange

From the Strategist

If I don't dump a bunch of stuff that's on my mind, I'll never blog again. Here goes:
  • Launchpad users within Canonical seem to appreciate the Roadmap. Still receive some complaints about not communicating our plans enough, but not sure what to do.
  • Team lead meeting in London went well. Good to see the gang again.
  • The Wellington sprint was amazing. Rarely seen a sprint that's more productive.
  • My favourite result: a fixed, pre-planned annual schedule of sprints, including two annual whole-team sprints. I get to meet with my colleagues and have a life and Canonical saves money all at once.
  • The team leads like the Ready-to-Code and Launchpad Enhancement Proposal process things.
  • Daily builds work is going well. There's a lot of UI bits to consider, and we're trying to get the UI correct before building it. Grates my JFDI nerves, but maybe it's the right approach.
  • Challenged the Launchpad team leads to blog once every two weeks about anything, as long as it's vaguely related to Launchpad. The name of the challenge? Blog like it's 2006.
  • Community Help Rotation sucks and basically no-one really knows what to do about it.
  • I'm cautiously excited about using kanban to track our work-in-progress.
  • Launchpad needs graphs in the Launchpad application. I think all we need is someone to just add one. I don't have the time to do it as part of my day job, and I've got a backlog of other hacking tasks. Could I help you do it instead?
  • I'm going to PyCon.

by jml (noreply@blogger.com) at February 11, 2010 01:13 AM

February 10, 2010

Paul Hummer

My First Package

This blog post is solely to commemorate the day that I first contributed to Ubuntu packaging. Today, Daniel Holbach uploaded the fix to Bug #516744, which is a Tahoe 1.6 update for Lucid. I did the update, and there was only one real change that Daniel had to make.

So yes, this makes me happy. Carry on.

February 10, 2010 04:33 AM