BHUGA WOOGA!

I made a choice to go into system administration about a year ago, after a hiatus from web development. I’m getting back into coding again in a hurry, and rebuilding my toolchain is a pretty painful process. The biggest part of that toolchain has been Drupal 6, and it’s both a wonderful revelation and a hugely frustrating experience.

This will make more sense with some background. I stopped doing web development in ‘03, after using the Perl on Rails framework, which is exactly what we did not call the combination of Perl 5, Class::DBI, Template Toolkit, and CGI::Application those days, long before Rails was a blip on the radar. I got out of it because I saw I wasn’t capable of doing bigger projects alone, not with those tools, and I supported myself on making minor updates to other applications and the occasional small-scale bookkeeping app. When I came back to trying to find some full-time work, I went for system administration, which involves more interaction with people, or at least it used to—coding is much more social than it used to be.

I lucked into a job running one of the most complicated Drupal installations out there, with some 600k lines of code across 130 modules. I work with some terribly smart people, and we’re doing some very cool stuff. It took me quite some time to start getting into the Drupal stuff itself, but once I started to figure it all out, it clicked pretty well. It’s a welcome kind of uncomfortable not to be the smartest person in the room.

My previous toolchain consisted of some Perl’s copious libraries, the windows ports of MySQL and Apache, cygwin, and vi. In hindsight, this was childish, pathetic, comical, the kind of toolchain that wears clown shoes. It still let me do an awful lot, compared to the scripts I was writing in ‘99 that amounted to little more than CGI to SQL wrappers, but it was clear to me how limited they were.

What a difference a few years makes: now I’m on a Macbook Pro with a ton of handy Macports, a native set of system tools equivalent to or better than the Linux equivalents I’m used to, and tons of useful apps, all set at the low, low price of $20-$40 apiece, because the cult of Mac is a true religion, complete with tithes. I’m finally getting practice with source control in the kind of complicated environment where it matters, and the devs around me know a ton of nifty tricks and tools. All of that stuff is a vast improvement, but the single biggest change to the toolchain is Drupal.

Before I get into what I like and don’t like, there’s a question about how one should see Drupal: as part of a toolchain, or as an end-user piece of software. It’s designed, from the ground up, to let people start with nothing and end with a website, so perhaps considering it just a link in a chain is inappropriate, but I don’t think so. No website, no matter how little code is involved, should be considered to exist in a vacuum, that’s not how the web works today. Each site should be considered a node on a graph in addition to a site in and of itself, just like a library written for a project should be considered in the scope of reusability. In this respect, I’m weighing Drupal against web frameworks like Ruby on Rails or Django as opposed to other CMS’s, such as that recurring villain of the Drupal graphic novel, Wordpress.

At any rate, wow oh wow, what an improvement! I really used to write menu templates that had to check for what page was currently being loaded and make that option have ‘current page’ css? Really? Write SQL for anything that required a join and an order or limit at the same time? Really? Write out form HTML? Hand-write javascript for the most basic form validation? Write templates at all? Drupal lets me worry about coding and not display, and that makes me actually interested in writing software again. It’s an amazing tool, and as the browser becomes a platform, it’s great to have so many of the basic tools in the web designer’s toolkit be done better than most desktop client frameworks do desktop.

So the move to Drupal really is a groundbreaker for me, and now I’m busy with web development and writing silly blogs when I get home from busy days of the kinds of things sysadmins fill their days with. Unfortunately, I still see a lot of problems. The gains make it all very worth worthwhile, but some of the parts of Drupal are exceedingly frustrating when it’s seen as a strong link in a toolchain and not an end-all, be-all website.

The first place this toolchain could be improved is an automated module install system. There are about 40 competing solutions, but on the whole, it’s ridiculous that I can’t tell my website to install the google analytics module into itself. The Drupal modules site doesn’t really help a newbie find the kinds of modules that every site should have, things like pathauto and google analytics. It’s a library distribution system that’s technologically behind pear, ruby gems, easy install, and even behind CPAN, which is about a decade old. There’s a lot of competing solutions to this problem, and hopefully one emerges as a workable base soon.

I find that some parts of the system are configured in strange places. Blocks, for example, are edited in a special area for blocks. That’s appropriate. The location a block appears on, including page-by-page exceptions, is edited with the content of the block. Meanwhile, the node’s own page selects menus, URL aliases, and publishing options, just inline options if the viewer is an administrator. While the ability to make a block display based on some PHP code is perfect sometimes, other times, it’s more maintainable for a page to control what blocks it has, and not the other way around. Besides, it’s rarely appropriate to give a non-administrator PHP rights. Breaking up the configuration of a page like this makes it harder to reuse any special configuration or code that particular page might have. I think a few instances of this kind of change might go a long way towards maintainability.

Hand in hand with maintainability is resuability. I have a lot of experience with this as a Drupal sysadmin. All of those great CCK types and views and whatnot require significant overhead to export and import properly, and keeping them in version control is even more difficult. It’s a huge problem when you manage as many sites as we do, and lots of people are trying to solve it: there’s a lot of good work going on in the change management group, tools like CoCKtail solve the problem by letting CCK types be a simple bit of text, and the ephemeral autopilot threatened to put the entire databases under version control to deal with the problem.

That problem looks like it’s being solved: good. It’s a shame that so much work is being spent on the simple problem that Drupal litters the definition of a site across code, a database, and a filesystem, and that it does it in a more or less unrepeatable way. Point to a particular file in a site’s files directory: is it in use, or is it some poor lost inode, left adrift from 2 upgrades ago? Best just to leave it be.

None of these problems are terribly hard to solve for D7—it’s just a point of view change. And really, at the end of the day, that’s a long blog post for a bunch of problems that aren’t going to keep me from picking something else, so let’s not assume anyone’s about to abandon all of those useful contributed modules in order to do something drastic. Let’s just keep an open mind.

We finished the video for Dries’ keynote just under the wire, as pretty much all such events need to be. Arto, Miglius and I had stayed up until past sunup for the last few days to make it happen. First Dan left, and then Miglius left on Saturday morning so that he could get stuck in Frankfurt for 24 hours. Once he got in to Boston, he logged on quick like a bunny and went back at it. Arto and I worked another 30-odd hours during Saturday and Sunday. Sometime during Monday, which I largely slept through, some of the office folk sent out a message noting that that our pile of pizza remains, chicken bones and coffee stains was not particularly helpful to the kitchen’s ambiance. I don’t think they know who did it, and I’m kind of afraid to fess up. Sorry, ladies.

Unlike most demo work, a ton of what went into this will be useful later. If our organizations were not keen on using RDF, we’d not have worked on this so hard. Arto’s module stuff is anything but smoke and mirrors, and we figured out a lot of limitations to Exhibit and Potluck that will be important to understand later. These are now posted in our internal wiki and I will go and post them on the Simile project’s site if I ever get a chance. It’s worth a whole post in and of itself.

While Arto busied himself turning Drupal into the world’s easiest to use RDF endpoint, Miglius and I combed datasets that would make for a decent demo and messed with Exhibit views. There’s a lot of RDF data out there, but it doesn’t all lend itself to being shown on a map, and people can only read so much on a video screen during a presentation. At the end of the day, I’m the only one with Leopard (and thus Screenflow), so I ended up doing the actual screencast.

Screencasting is an interesting thing. It’s easier to script than a regular movie, but difficult to properly realize. There’s a fine line between too little and too much data, without having awkward pauses and without skipping over too much. You have to take into account that different viewers have different levels of experience with the material, different reading speeds, whatever. I made a detailed narration that was a bit too fast paced for the keynote; that wasn’t a problem, as Dries had already communicated that he’d prefer to do the narration himself.

On Monday, Arto and I woke up about a half hour before the talk and got on IM. As the talk began, we realized that we really needed to have this data up where people could get it. And we really wanted them to be able to get it—we’d worked ridiculous hours on this thing. So that’s when we decided the site needed to be public.

We started to make that happen. There was a fair bit of configuration to be done to make it useful; Arto got the video onto s3 while I messed about with some permissions and redirects. I typoed just about everything I did related to that—I don’t think I did a single thing once. Halfway through the whole thing I realized I had stage fright; I couldn’t type because my hands were shaking. The video I had worked so hard on was about to be placed up to awe or bore a sizable number of people, on whom much depends. And there was still a possibility that Dries would use my narration, in my mind, as we’d given him the final cut of the video with extremely little time to rehearse anything he wanted to say. So there I was, still in bed, with the door shut and the window blocking out what passes for sunshine in Stuttgart, and I was nervous as hell about being up in front of a crowd.

Stage frightened of nobody at all. What a cool world we live in, that such a feeling can now be transferred over the wire.

Anyways, we did a good job (well, mostly Arto did a good job) of getting the video out there for anyone who wanted it, and at least a couple of people did. Here’s another copy, if you’re curious: