Roger Searjeant's blog

Goodbye Blogger - hello WordPress.com

2011-04-02T22:20:00.002+01:00

Blogger just isn't an attractive platform any longer. Development appears to have all but ceased, the interface isn't great, and the integration with other Google apps is neither good enough nor important enough to me to keep me here.

One reason I blog less than I want to is that displaying source code in Blogger is much harder than it should be. There are instructions available for integrating the excellent Syntax Highlighter from Alex Gorbatchev, but frankly it's more fussy than it should be. Google of all people should be making this easy.

So this blog is moving to searjeant.wordpress.com from today. Barely anyone reads it, so it hardly matters, but for anyone who cares, now you know.

JVM Language Futures - Groovy++, Mirah, Gosu ...

2011-03-28T23:01:00.001+01:00

Interesting times, with so much happening in the JVM language space. It's amusing, too, to see static typing making a comeback via some of the very same projects which helped to champion the cause of dynamic languages and duck-typing.

The three I've noticed all look promising and merit a bit of investigation. In order of importance (IMO):

Mirah. From Charles Nutter, who created JRuby, so this one has probably the best pedigree. I don't much like Ruby syntax, but I absolutely agree with the guiding principles.
Groovy++. I really like Groovy (and Grails). The Groovy++ project goals overlap to a degree with Mirah, mostly in terms of static typing and achieving native Java runtime performance.
Gosu. This one is interesting because it was created by a company team to meet their particular needs and not primarily to invent a new language.

These three languages have quite different starting points but appear to share broadly similar goals. Essentially, they are trying to obtain as many of the benefits of a dynamic language as possible (leaner code, less ceremony) while at the same time carrying as few of the disadvantages as possible (no types = poor tool support, dynamic method dispatch = relatively slow). That's a difficult sweet-spot to hit.

Although I ranked these three in terms of importance, I actually think Groovy++ may have the best chance of getting into production code first because it's more mature than Mirah and has the great benefit of being an extension to something which already exists and is successful and established. I'd love to see Groovy++ adopted officially and absorbed into the main Groovy roadmap. Being able to add static typing just where you want (and leaving it out where you don't), getting traits into the language and getting better runtime performance are surely strong arguments in favour.

Xubuntu desktop

2011-01-01T22:06:00.000Z

I've lost count of the number of false-starts I have had with Linux in the last few years. I really do want to settle on a Linux-based desktop for a number of reasons: zero capital cost, no licensing restrictions or 'product activation' steps and everything appears to run faster than on Windows. Most of my development work is on the JVM platform (Scala, Groovy and Java) so I am not dependent on Windows. I should add that I have no time for the anti-Microsoft movement: I still think Windows (especially 7) is a very good desktop OS and it's likely to remain my default fallback for a while. I'm still not completely comfortable if I can't boot back into Windows if Linux dies (or more likely, I screw it up) and there are still a few things I use which don't yet have Linux builds or good alternatives.

With Xubuntu 10.10 I'm daring to believe that I might have a stable enough platform to be able to make a bigger commitment to Linux, certainly as a development desktop. I've been running it alongside Windows 7, using the Wubi installer. This is by far the best way to experiment with Linux: you get very close to the experience of a 'proper' installation (I don't think I have been able to detect the virtual disc performance hit) and you don't disturb Windows or the drive partitioning in any way.

I have two reasons for wanting to move on from a Wubi install:
1) Virtual drive size. I went with the default (17GB or so) and eventually this will probably be an issue.
2) You cannot use Remastersys on a Wubi installation. To use this handy utility, you need a real installation. I know this, because I trashed my first Wubi installation when I tried.

Apart from creating a complete backup of your installation, Remastersys can also do something else: generate a new master ISO image of your installed OS, containing all the configuration changes you have made, so you can walk up to another machine and just deploy it. That's worth having, because I don't want to have to go through the process of installing my stuff all over again (and waiting for the upgrades) and I will want to standardise the development desktop we use via a deployment image.

Happily, there is a way to move your Wubi Ubuntu to a separate partition, turning it into a full installation. This thread on the forum provides a really good step-by-step guide, complete with downloadable script. I plan to try this but only after I have made a drive image of my current Windows installation (because if I don't, then I know it will go badly wrong).

So what's left? What are the things I still cannot do on Linux? The final barriers to a complete switch? There aren't many:

Some digital devices, notably my Sony Walkman, just don't have drivers or companion software. With the Walkman, the problem is Sony's truly dreadful SonicStage software. Much as I hate it, it's the only way to convert and sync stuff to the device. When I last looked, there were a few poor-quality tools which claim to work with the Walkman but when I tried them they messed-up the device's track database and filesystem. Not going that way again.
My Vodafone USB wireless broadband device. There's a Windows driver for this, but no Linux software, as far as I know.
VISIO. I do need a decent 2D drawing / diagramming tool. I have tried Dia, but I just don't like it and development appears to be stalled. In the past I have almost ignored OpenOffice Draw (because I have Visio), but I think Draw may be just good enough, if I make a bit of effort with it.

There are a few other minor things, but these are the three I would pick as most irritating. To fix (1), some might suggest I buy a different media player! Maybe, but this is really just an example of a more general issue which only the passing of time will solve: manufacturers and software publishers supporting deployment to non-Microsoft environments. And there's WINE of course: this does solve some problems, but unsurprisingly not everything runs under WINE (SonicStage, for example) and even the things which do run, look pretty awful.

Xubuntu (and I think the plain Ubuntu distro) also appears to have removed another barrier to adoption which I just could not work around before: poor font rendering. When you spend so much time reading, editing and writing on-screen text, fonts (and rendering) really matter. In the past, Linux font rendering was truly dreadful: inconsistent across applications, and never very good in any of them. That seems to be over. Java applications still leave something to be desired, but that's down to Java, not Linux.

Cautious optimism, then.

Keith Douglas. Poet, Tank Commander.

2010-11-11T15:16:00.001Z

It’s Armistice Day here in the UK, an appropriate day for BBC Four to air a programme about Keith Douglas, a poet and a tank-commander during WW2.

I stumbled on a poem of his some years ago, quoted in a newspaper article. It made such an impression that I tore it out of the page and kept it, vowing to find out more about the author. The poem I read that day was called “How to Kill”. There is a Guardian article about Keith Douglas, worth reading in its own right, which includes that poem. I’ve reproduced it, below: I do hope that doesn’t get me into copyright trouble.

Keith Douglas also wrote a book of his experiences in the Western Desert called “Alamein to Zem Zem”, written while on leave in the UK prior to taking part in the D-Day landings in June 1944. I loved this book. It was out of print for a long time but is available, reprinted by the excellent Faber Finds imprint.

Now in my dial of glass appears

the soldier who is going to die.

He smiles, and moves about in ways

his mother knows, habits of his.

The wires touch his face: I cry

NOW. Death like a familiar, hears

and look, has made a man of dust

of a man of flesh. This sorcery

I do. Being damned, I am amused

to see the centre of love diffused

and the waves of love travel into vacancy.

How easy it is to make a ghost.

GMail Priority Inbox – at last

2010-09-05T22:16:00.001+01:00

Life without GMail (or Google Docs, or Reader or Calendar, for that matter) would be difficult to adjust to, but for a long time I’ve wanted the ability to prioritise the inbox contents, if only to ensure that unread items (however old they are) can be forced to the top of the pile.

Well, they’ve obviously read my mind (didn’t I un-tick the ‘Share my thoughts’ privacy option? Oh well …) because I’ve just been given the Priority Inbox Beta option. First impressions are very good. You can train it to recognize what’s important (and not) to you: we’ll see how well that works out, but just having unread stuff and starred items grouped together is a tremendous plus.

Can’t think why anyone would want to use any other email system.

NetBeans Platform 6.9 Developers Guide

2010-08-30T23:58:00.000+01:00

I've been asked to review the NetBeans Platform 6.9 Developers Guide by Packt Publishing. This is a book which I was considering buying in any case, so it's nice to be given a copy, even if it is the PDF rather than the paper.

I've bought quite a few PDF e-books in the past, mostly from Manning, usually because I just wanted the information right now, but also because having the PDF on the laptop weighs nothing. Commuting and heavy technical books don't mix. However, I do prefer paper: as soon as a PDF book becomes important to me, I buy the physical book.

NetBeans is easily my preferred Java / Groovy IDE, and the NetBeans platform has become even more compelling recently with OSGi support and Maven archetypes for NB modules. I have had a copy of the original NetBeans Rich Client Programming book published by Sun for more than a year, but to be honest I found the online resources, sample projects and Geertjan's tutorial screencasts rather more helpful than the book! As I went through the Sun book, I remember thinking that I would probably get a lot more out of it once I knew how to create a NB platform application. I think it's one of those books.

So I am very much looking forward to getting stuck into the Packt book. From a first look at the Preface and chapter summaries it appears to take an API-focused approach, dealing with 'top-10' in much the way Geertjan's screencasts do, which is a good sign because I found they worked very well. The book essentially builds a complete platform application as it goes along, so I'm hoping to see the authors tie together the various ideas using the application as the binding context.

I'll post my review here, but I'm on vacation so it will be a week or two yet.

Windows 7

2010-04-09T21:24:00.000+01:00

It's not often that I go out of my way to praise something from Microsoft, but I have been using Windows 7 Professional for a week or so now, and I must say this is easily the best desktop operating system I have ever used.

I bought the upgrade edition, but used the Custom Installation option, wiping everything (in my case, Vista and Ubuntu) from my hard drive. The installation was spectacularly uneventful - it just worked. I didn't time it, but it can't have been more than 30 minutes or so from start to finish: and no issues.

I haven't explored all the features yet (e.g. Libraries), but I'm truly impressed with just the basics - the whole taskbar implementation, the new tray/notification area, the clean desktop - and the fact that it feels much faster than Vista, though I've absolutely no hard evidence to back that yet.

Ophcrack NTLM Cracker

2010-03-15T21:55:00.000Z

Not sure how concerned we should be about this, but the demo is pretty impressive: I used PasswordSafe to generate a reasonably good password (12 characters, mixed upper/lower case, digits and symbols) and Ophcrack cracked it in a few seconds. The live CD should disturb anyone using standard XP password security.

HAPI 0.6 Released

2009-07-02T15:11:00.001+01:00

Anyone involved in HL7 processing should be aware of the HL7 API (‘HAPI’) project. HAPI is a set of Java libraries for parsing, unparsing and manipulating HL7 messages, plus some communication primitives for handling MLLP communication.

This was started some time ago by Bryan Tripp at the University Health Network in Canada, but recently the work of maintaining the project and pushing forward with development has been taken on by James Agnew.

Version 0.6 brings support for HL7 v2.5.1 and v2.6, and separate jars for the different HL7 versions. James’ announcement also hinted he’s looking at significant performance improvements in future releases. This is great news: thanks are due to James for his continued good work.

If you’re trying to find the bits, note that the download link isn’t in the HAPI site sidebar – go to the Using HAPI / Developing page, and you’ll find the link to the Sourceforge download page there.

Technorati Tags: HL7,Java

Groovy and Language Design

2009-06-13T12:18:00.001+01:00

I was a little surprised to read Elliotte Rusty Harold’s recent comments on Groovy, calling it the ‘Edsel of programming languages’. It’s an amusing comment, but pejorative and challenging enough to motivate a response.

The past few years have been interesting and fruitful from a programming language perspective: Ruby, Groovy, Scala, F#, and the evolution of C# are those which spring to mind.

Many of these developments have been about finding a way to push functional-programming idioms (lambda expressions, currying) or dynamic language features (type inference, closures) into well-established imperative languages like C# and Java. I think the results have been mixed: imperative languages are essentially assignment-based, not expression-based, so you end up with a hybrid which, from a language syntax point of view, is neither one thing nor another and arguably suffers as a result. But it’s difficult to argue that these efforts don’t deliver real positive utility.

An example of this is how C# has evolved support for lambda expressions, expression trees and implicit typing. First (in v1) we had simple delegates, then (in v2) we were able to drop the delegate syntax and declare anonymous methods inline, and finally (in v3) we can use a compact lambda syntax which looks a bit like ML. But because all this has been added to what remains fundamentally a statically-typed, compiled language, you still have to put up with all the nasty explicit declaration of types (and type parameters) on the l.h.s. of every assignment, just to keep the compiler happy. So the final step was to introduce the var keyword and bring implicit typing into the language. Not dynamic typing, just implicit typing: we get to trim away the ugly declarations but keep the compile-time type safety.

The confluence of these developments enabled things like LINQ: blending a query sublanguage into the primary language syntax. The best exposition of this I have read is by Ian Griffiths, in his excellent post on expression trees: see the example code just over half-way down, under the section ‘Expression Trees, LINQ, Deferred Queries, and Databases’. It’s all so superficially appealing – just look at the motivating use-case in Ian’s post. But still there’s a small, nagging voice in my head saying this isn’t quite right...

It feels as if we’re trying to pretend there’s no boundary (or intermediary) between the program text and the resource we’re querying. This is similar to the way we tried to pretend that remote procedure calls were really local calls in the old (and now largely discredited) distributed programming models such as CORBA and DCOM. Recall the fallacies of distributed computing and the tenets of SOA: these are essentially warnings about the dangers of implicit or flawed assumptions. Although the tenets have been challenged (and were threatened with retirement) most of what they said is timelessly valid. In the context of LINQ, I think my nagging voice is pointing to the boundary-crossing in a LINQ query and muttering “private implementation technique, not primitive construct”.

What’s this got to do with my starting point for this piece, which was Harold’s ‘Edsel’ jibe? I suppose it’s this: whatever we may feel about the virtue of purity in language design, in the end productivity and usability are the bigger drivers, and with skill it is possible to combine language features without creating a monster.

Technorati Tags: Java,Groovy,C#

Sun, Java and Innovation - reflections

2009-06-13T10:59:00.001+01:00

My previous post on the Sun acquisition was a rather naive and misty-eyed piece of sentimental warbling. The truth is that Jonathan Schwartz’s vision hasn’t delivered quickly enough for the market: the share-price has suffered and Oracle has picked up something of a bargain. Pity, because (perhaps looking through those rose-tinted spectacles again) I rather like Schwartz’s ideas for growing the business, essentially giving away the technology and then ‘monetizing adoption’. But it’s not happened nearly fast enough and one wonders whether it ever could.

An editorial in SD Times sums up Sun’s difficulties nicely as ‘too much vision, too little execution’. It’s not enough to give away tools and hope the next generation will adopt them. You need to innovate and to lead, because the brightest and best of the next generation want to climb the mountain, not travel for free in the foothills. To innovate, you need great scientists and engineers; to retain those folk, you need to pay them adequately and provide them with the facilities and resources they need to be brilliant. To do all of that, you need money. And that comes from sales. It’s as simple as that.

Just look at Microsoft Research: you just can’t foster this quantity and quality of innovation unless you have a lot of money to invest, and you only get that by doing a lot of very effective monetizing. Microsoft and Oracle: we don’t like them very much, do we? But they’re really good at monetizing.

As for Java, most seem to agree there is no threat to its future as an enterprise platform: it’s open-technology now and well embedded in the enterprise. What about the future of the Java language? Do we really need to worry about that? The language itself matters much less than the JVM, which surely has a very bright future given the newer languages targeting it, especially Scala and Groovy. Should we care whether (e.g.) closures make it into the Java language, if we can use newer languages like Scala and continue to leverage all the existing Java libraries, components and infrastructure?

Technorati Tags: Java,Sun,Scala,Groovy

As the Sun sets…

2009-04-28T21:54:00.001+01:00

Why does Oracle’s purchase of Sun make me feel slightly sad? Silly, sentimental reaction, isn’t it? I should know better. After all, today’s Sun isn’t the super-confident (some would say arrogant) innovator and market leader I grew up with: arguably, today’s Sun needs rescuing from itself, needs a sharper focus on what it does best, and to sell more of fewer things.

But I have a soft spot for Sun. My first proper programming job involved writing C/C++ (and using Cfront – remember that?) on a Sun-3 workstation, and various versions were part of my working life for some time. I loved the solid feel of these machines. Remember the optical mouse that only worked on those special, shiny metal mats?

Between then and now, my only links to Sun have been through OpenOffice and Java. I played with Java quite early on, abandoned it in favour of Microsoft .NET, but have recently (and happily) returned to it. For the last couple of years I’ve been an enthusiastic user of NetBeans – I do hope Oracle recognizes just how good NetBeans is. I have also tried OpenSolaris: ZFS is simply awe-inspiring, and very nearly enough on its own to make me run OpenSolaris, though truthfully I don’t think I need ZFS, and Windows remains simply more convenient and usable for everyday.

I’ve used OpenOffice for a long, long time. I know Writer pretty well, warts and all. You need to ignore some of the cosmetic shortcomings, persevere with it and appreciate its fundamental strengths; things which I think make Writer better than Word. Occasionally I use OOo Writer to help colleagues debug and rescue Word documents which have evolved uncontrollable formatting: it amuses me to be using a free tool to clean-up after a rather expensive one. I really hope Oracle will resource and manage the OpenOffice program properly. With the right additional effort, they have a potential Office-beater.

But the crown-jewels are Java itself, and the NetBeans IDE. Everyone is trying to second-guess what Oracle will do with Java: I don’t have anything to add. But I really want to add my voice to those hoping Oracle will recognize just how good NetBeans has become, not just as the best Java IDE out there, but also a first-class platform for building rich-clients.

Technorati Tags: Java,Sun,Oracle,NetBeans,OpenOffice

EIP Shapes for OpenOffice Draw

2009-03-28T11:06:00.001Z

If you’ve read the excellent Enterprise Integration Patterns (EIP) book by Gregor Hohpe and Bobby Woolf, you may also know that free VISIO stencils are available for the EIP shapes, from the download page.

This is great for those who have a copy of VISIO (or whose employer pays for a copy), but for those of us who don’t want to pay however many hundreds of dollars Microsoft wants for VISIO these days, an alternative (especially a free one) is always welcome. OpenOffice (OOo) Draw, though hardly as functional or attractive as VISIO, is both free and good enough for many diagrams.

So I decided to see if I could create a set of OOo Draw shapes from the VISIO stencil, and see whether Draw could be practical for integration diagramming. In short, I succeeded in converting the stencil, and I think Draw is just about good enough. For those not interested in the background, I’ve made the material freely available from this page on my website, so just go ahead and download/install. Otherwise, read on.

Exporting the VISIO Shapes

One way to get shape data into Draw is via SVG. So I started with the VISIO shapes, freely downloadable from the EIP site, and as I use VISIO at work I was able to load up the stencil and export it as SVG.

You need to install an OpenOffice extension for importing SVG. I’m using OOo 3.0: the extension installed without incident and the import was straightforward. I was surprised at how good the resulting shapes looked in Draw. Each shape is composed of drawing primitives, so you need to select and group them to create a shape you can place and move easily. I’ve done this for all the shapes in the download.

So now we have the shapes, where can we put them to make them easily reusable? In Draw, there’s no stencil feature as such: the only option is the Gallery:

The Gallery

The gallery feature in Draw isn’t very appealing, initially. When you first install OOo, the gallery contains some spectacularly awful materials: nasty backgrounds and bullets which I cannot imagine ever wanting to use, and it even contains some pointless sound-clips that surely only a 5-year old might use (and only then if they were desperate). The shapes and graphics are reminiscent of shareware software from the Windows 95 era. And what is the point of the built-in media player? Why can’t OpenOffice grow up? It’s so very close to being excellent in many respects: things like this just make it look silly and dated.

Anyway, back to the gallery feature. The one saving grace of this feature is that you can create new gallery themes and add your own shapes to them. To do this, create a new theme, give it a name and then drag your shapes into it. Note that when you drag/drop your shape you must hold down the mouse button and pause a second or two before dragging: this pause changes the drag-mode from a simple shape move to the required move/copy.

I’ve done this for all the EIP shapes, creating an EIP gallery theme:

Sharing the EIP Theme

Unfortunately, Draw doesn’t implement a theme export facility so to share this new theme with others requires some irritating file copying. The contents of the Gallery are stored in your profile (or home) folder. On Windows this is likely to be:

C:\Documents and Settings\<user>\Application Data\OpenOffice.org\3\user\gallery

On Unix/Linux, check your home folder. There are several file types in here and (unusually for OOo) they are binary, not XML. This makes it difficult to reverse-engineer their purpose and relationship but fortunately it doesn’t seem to be necessary to understand fully what’s going on. The key thing to note is that the numbers in the file names bind together related files. When you create a new theme, a new set of files appears to be created.

For the EIP theme I have manually renamed the files to use numbers which I hope won’t clash with any existing gallery files you may have. Check the zip contents before you extract the files into your OOo folder and if you have existing files with the same name, simply rename the files in my zip to be unique in your environment.

The Result

I think the result is pretty good. As a test, I have (partially) reproduced the diagram from the end-cover of the EIP book in a Draw file (.odg) which is included in the download zip:

I hope this may be some use to other folk. Download from this page.

Technorati Tags: OpenOffice,EIP,Enterprise Integration Patterns

OpenESB and BPEL

2009-03-27T19:21:00.001Z

Thanks to the folks who commented on yesterday’s post I have managed to overcome the assign problem and of course the answer was very simple: I had an inconsistent set of components installed in NetBeans and GlassFish. I was using NB 6.5 together with components downloaded from the OpenESB site.

The best solution is simply to download the complete NetBeans/GlassFish/OpenESB combination from the OpenESB downloads page and work with that. On the download page the top section contains the GA (stable) GlassFish ESB / NetBeans combination, and the nightly build of the latest (unstable) additional components (i.e. binding components, service engines etc.) which match this GA release.

The next major section on the page contains the GlassFish ESB v2.1 downloads (milestone 1 at the time of writing), which are not yet considered stable. I grabbed the 2.1 release, plus the HL7 BC (and NB design-time modules) from here. Installation was completely pain-free. I particularly liked the way the additional components are packaged as a jar-based installer – very nicely done.

Because the main package is a GlassFish ESB bundle, it installs by default to a non-standard path: C:\GlassFishESB. Everything is installed under this directory, including the configuration files for NetBeans, which would normally be placed in your home directory (under .netbeans). Presumably this is done to allow parallel installation with a stable NetBeans/GF without either interfering with the other. What this means, of course, is that all your preferences (editor settings etc.) are not transferred. This is easily fixed, manually, by copying over the stuff from: C:\Documents and Settings\<user>\.netbeans\6.5\config\Editors to C:\GlassFishESB\.netbeans\glassfishesb\config\Editors

Anyway, back to the BPEL issue. A clean rebuild of everything was sufficient: the CA deployed to GF, and it ran fine. I used Hermes JMS and watched the test messages arrive on the queue. Now I can push ahead and explore more of OpenESB and BPEL.

Technorati Tags: OpenESB,NetBeans,BPEL,Java

Problem with OpenESB BPEL assign

2009-03-26T22:41:00.001Z

After far too many cycles staring at code, building, deploying, debugging and undeploying a pretty simple composite application, the damn BPEL assign activity still causes selectionFailure at runtime.

The BPEL is trivial, and the BPEL copy elements for optional elements in the source schema are marked with the ignoreMissingFromData attribute. I have rebuilt everything from scratch. I know the redeployments are successful because the error reports points to the correct line following an edit and I can use the BPEL debugger to step through the assign, eventually failing on the optional element.

I was given the hint about the ignoreMissingFromData attribute by Michael Czapski, in a blog comment. Googling around for more help, I came across a few other postings, and discovered that the attribute could be placed at the process level or at the individual copy activity level. Jeff Sexton also has a nice post on the assign issue.

In Michael’s tutorial he does in fact place the attribute at the process level (see p.95 of the PDF document which accompanies his tutorial). Placing it at the process level is not very attractive because it will mask the absence of mandatory fields.

When placed at the copy element level, it seems to fail. Here’s the error I see, as output in the GlassFish console window:

BPCOR-6151:The process instance has been terminated because a fault was not handled; Fault Name is {http://docs.oasis-open.org/wsbpel/2.0/process/executable}selectionFailure; Fault Data is null
com.sun.jbi.engine.bpel.core.bpel.exception.StandardException: I18N: BPCOR-3023: Selection Failure occurred in BPEL({http://enterprise.netbeans.org/bpel/HL7Processor/bpHL7Processor}bpHL7Processor) at line 28!
BPCOR-6129:Line Number is 26
BPCOR-6130:Activity Name is Assign1

Note the line numbers in the above report: the first of these (line 28) is where the selection failure actually occurred, line 26 is the opening tag of the enclosing assign activity element. Here’s a shot of the source so you can see the corresponding line numbers. I placed all the optional elements together at the start of the assign:

So why doesn’t this work? Is this a bug? Perhaps I’ve run into something which only works in the JavaCAPS product and not in OpenESB? Shame.

Another unfortunate discovery is that the OpenESB UI in NetBeans 6.5 doesn’t seem to support adding the ignoreMissingFromData at the process level. In Michael’s PDF he selects the top-level BPEL process scope and uses the property panel to set this attribute. It doesn’t appear to exist in NB 6.5. In the shot below I’ve tried to capture the same UI as in Michael’s tutorial:

Note there’s nothing below the ‘Documentation’ entry: presumably this is additional UI support which you get in the commercial JCAPS product. I’m not that concerned with the missing UI, as long as I can add the attribute in the XML source. So I edited the BPEL XML, using Michael’s tutorial source as a guide. It’s easy to add the process-level attribute; only the namespace prefix had to be changed:

<?xml version="1.0" encoding="UTF-8"?>
<process
    name="bpHL7Processor"
    xmlns:ns3="http://www.sun.com/wsbpel/2.0/process/executable/SUNExtension"
    ns3:ignoreMissingFromData="yes"
    targetNamespace="http://enterprise.netbeans.org/bpel/HL7Processor/bpHL7Processor"
    xmlns="http://docs.oasis-open.org/wsbpel/2.0/process/executable"

So another clean/build/deploy and … it still didn’t work. So now I’m stuck, and I hate being stuck. It’s almost enough to put me off the product.

The impression I get is that BPEL is great for very simple maps and transforms, but anything complex quickly becomes awkward. I haven’t gone beyond ‘simple’ so far, so I’m getting a little concerned. The pretty visual tools and NetBeans integration are cool, but the complexity overhead seems high. It reminds me of Microsoft’s BizTalk product from a few years back – a lot of fiddly config, and not being able to ‘see the wood for the trees’.

But I want to persevere with OpenESB: something about this product suggests that you need to reach a certain level of enlightenment, after which everything becomes clear and the benefits outweigh the overheads. I’m just not there yet.

Technorati Tags: OpenESB,NetBeans,BPEL

DropBox again

2009-03-18T16:33:00.001Z

The previous post elicited a comment from Andy at CloudBerry Lab:

I am a developer of another online backup product powered by Amazon S3 http://cloudberrydrive.com/ that we are going to release to beta soon. I would appreciate if you take a look and may be post a review on your blog.

Well I did take a look, but the CloudBerry product isn’t really what I’m looking for. If all you want/need is a Windows-only client for accessing your S3 buckets directly and copying/moving content, then this product might well be just right. But it doesn’t really compete with DropBox or JungleDisk. Andy: if your product were closer to DropBox, and cross-platform, then I would have reviewed it. As it is, I can’t pass judgement.

If you have time to plough through a very long blog thread, check out Jeremy Zawodny’s excellent piece on Amazon S3 tools. The discussion thread contains links to most (maybe all) the tools out there.

I’ve also been reading the DropBox forums. It’s good to see plenty of other people have requested features I miss (e.g. ability to configure DB to ignore certain file patterns). There is an upcoming features wiki page which lists some things coming ‘soon’. Nothing on using your own S3 space though.

JungleDisk and DropBox

2009-03-16T16:21:00.000Z

There are quite a few network-drive products out there competing for our attention these days. I’ve tried quite a few (including Box.net and SkyDrive), but narrowed the choice down to the two which seem the best: JungleDisk and DropBox.

JungleDisk attracts me mostly because I can use my existing Amazon S3 storage behind the JungleDisk tool and pay only for the space I actually use. (Note, JD now also uses RackSpace CloudFiles, which actually looks even better than S3). DropBox is free for 2GB, then a hefty 99 USD annually, for 50GB. Round one to JungleDisk, in my view.

Then we come to the user-interface. Both tools integrate with Windows and Linux, JD using drive mapping to expose the storage and DropBox using a special folder inside My Documents, with icon overlays to indicate file status. Both support drag/drop access and run a small tray-resident UI application.

But DropBox is just so, so much nicer to use than JD in the everyday Windows context. It feels better integrated and the UI seems cleaner. Other folk have blogged about this difference and I must concur – DropBox has the edge.

Now to the subject which prompted this post in the first place. Neither of these products appears to handle proxy servers particularly well, especially when switching between proxy / no-proxy. If I restart Windows and forget to switch off the proxy in JD, here’s the mess I’m greeted with when Windows starts:

Yuk. Can’t it simply notice that the proxy isn’t responding, log the fact / decorate the tray icon, and leave it for me to sort out? It gets worse: if I click on the links (for more information) look what I get:

Is this really what they want the user to see? This is awful.

DropBox is slightly better, but still doesn’t work properly if I leave the proxy on, and restart. No nasty dialogs, but the network connection isn’t resolved, even if I set it to ‘auto-detect proxy settings’ which according to the DropBox site should use the IE settings. Why can’t these tools auto-detect proxies properly?

DropBox files are cached on the local machine which means if the network is down I can still work on all my files locally, and re-sync when I next connect. JungleDisk does cache your files, but in a pretty inaccessible way in your profile. The path will be something like C:\Documents and Settings\<user>\Application Data\JungleDisk\cache\e9998872111157539d8880eca4456345-default

Another good feature of DropBox which isn’t available in JungleDisk is sharing files and folders: in JD, everything is private.

DropBox gets so many things right. The one and only feature I want from JungleDisk is the S3 / CloudFiles backing store. Obviously, the DropBox business model is built around the 99 USD annual charge so I don’t know whether this can/will ever happen.

OpenESB and HL7

2009-03-11T23:24:00.001Z

Technorati Tags: Java,OpenESB,HL7,NetBeans

As well as looking at Apache Camel for healthcare integration, I have also been spending time with Sun’s OpenESB product. OpenESB is the open-source counterpart of the Java CAPS product, a mature and well-supported Enterprise Service Bus (ESB) built on the Java Business Integration (JBI) standard.

OpenESB is delivered along with the GlassFish application server (v2) when you install the SOA extensions to the NetBeans IDE.

Michael Czapski of Sun is an excellent source of information on healthcare-related application of the CAPS / OpenESB product line. He has created a really substantial demonstration project based on a healthcare integration scenario. His blog entry and the associated downloads (and screencast) are well worth getting hold of. The screencast and writeup are based on Java CAPS, the commercial product which provides a few nice UI improvements and I believe some additional tools which the OpenESB product does not have.

I decided to go through Michael’s complete tutorial, using NetBeans 6.5 and the bundled OpenESB / GlassFish combination, building the solution and making notes as I went along. I’ll put the source code for the NetBeans (6.5) and OpenESB project group up on a publically available Assembla Subversion space (URL to follow), in the hope they may be useful to others who want to use OpenESB to do a similar thing.

As already mentioned, the differences between Java CAPS and OpenESB are superficially small, so if you are already familiar with one or other product you probably only need Michael’s notes and material. However, if you’re relatively new to OpenESB (as I am), there are a few places where the absence of JavaCAPS tooling means you need to know what you’re doing to patch-up the OpenESB solution.

Getting Started

My first big lesson was getting a correct / compatible set of NB plugins and libraries installed. The zip archive provided by Michael appears to contain everything you need, including the XSDs, NB plugins and encoder libraries. As the article was published very recently, I assumed these were the latest versions so tried to configure my NB 6.5 instance using these.

I followed the instructions for installing the encoder library (first uninstalling the existing one), then installing the NB plugins. However, the plugin installation just wouldn’t go ahead:

I tried various combinations but in the end had to uninstall GlassFish v2 and run the NB installer again to reinstall GF plus the OpenESB bits. Fortunately, this is actually quite a quick process, and puts everything back as it was.

I won’t go through all my subsequent false-starts, but simply tell you to download a matched set of the latest versions from this location:

http://download.java.net/jbi/binaries/open-esb-full-install/nbm/latest/

The components here should all be compatible. It really is best that you download and install these yourself, so I won’t put them in the Assembla SVN space.

Doing without the Java CAPS Wizards

The most obvious differences when watching the screencast are the degree to which the commercial product contains wizard steps and conveniences for generating all the configuration items. Reasonable enough I think, and so far I have been able to complete every step using OpenESB and NetBeans 6.5.

When creating the concrete WSDL definitions with the HL7 bindings, the wizards in the Java CAPS product (seen in Michael’s screencast) do make the whole process slightly simpler. But if you take a little time to understand what these wizards are actually doing in the generated WSDL, it’s not too hard to complete the steps manually. That’s easy to say now, but I should ‘fess-up and admit that it took me a little while to spot what I hadn’t done, a couple of times.

For example, once I’d completed the HL7Consumer_CA_A01_A03Delim_HL7In WSDL (sorry Michael, I’m not a big fan of your naming convention!), I simply couldn’t add it to the Composite Application design surface: the tutorial document instructions didn’t appear to work. I could see the WSDL in the list of available WSDLs and select it – but could not add it to the canvas, and couldn’t see any error message to tell me why.

After a cup of tea and a biscuit and a bit of careful thought, I realised why. The HL7 protocol properties were completely missing from the port declaration in the Services section. It’s easy to fix this, using the context menu commands:

Then edit the properties in the property grids. Of course, in the JavaCAPS product the wizards prompt you for these properties and create the related WSDL bits behind the scenes. I don’t like depending on wizards. Fortunately the NetBeans tooling gives you just enough convenient UI to hide the underlying cruft, but also lets you work at the XML / source level and see what’s going on down there.

That’s probably enough for this post. There’s so much more to write but I need to knock it into shape before I publish.

Tool Support for DDD

2009-02-25T19:15:00.001Z

I like Eric Evan’s book Domain Driven Design very much. It’s such a nice distillation of a relatively small number of principles, all of which possess real, fundamental value. This is a protein-rich book: it may not feel as filling as the more prescriptive, methodology-driven ‘just-do-as-I-say-and-it-will-come-out-right’ books, but they’re just carbohydrate. DDD is good for you in the long-term.

I’ve just discovered that you can read some of the book online, via Google Books. You can also find out more from the DDD Site.

The real point of this post is to share my experience with a small, free tool called wikidPad, which I stumbled on last week when looking for a desktop wiki-based tool to use for capturing Ubiquitous Language (UL) definitions. Capturing and refining the UL is an essential part of applying DDD, and having a fast, convenient (and open) medium for doing this is essential. A wiki of some kind is almost ideal.

Most desktop or single-user wikis are either cut-down versions of the equivalent web apps (PHP or whatever), running on localhost ports, or they are single self-editing HTML pages e.g. TiddlyWiki. Many of these (including TiddlyWiki) are very good indeed, but somehow, none of them has felt quite right for the UL use-case, which is all about getting hyperlinked definitions into a reusable form as quickly as possible. wikidPad does this superbly well, in my opinion.

It’s also a small and fast desktop application, which I find a little more convenient than using a wiki app in a browser. Some examples to support that:

wikidPad automatically constructs a tree-view of pages on the left hand side of the application – this makes navigation very fast.
wikidPad does auto-completion of wiki-words (Ctrl-space), which makes writing new entries which link to existing items very fast.
Drag and drop support, and support for multiple tabs – again, all good for speed and usability.

When you’re ready to publish the UL to a wider audience, wikidPad will export the whole thing (or any selected part) as a set of HTML pages.

It’s my first day using this tool, so it may be that I’ll discover something to dent my enthusiasm. But so far, it’s all good.

Technorati Tags: Domain Driven Design,wikidPad,wiki

Assembla

2009-02-23T09:52:00.001Z

Technorati Tags: subversion,git,hosting

Although Project Locker has served me very well for a good many years, better alternatives have emerged recently. I didn’t set out to switch providers, only to see whether Project Locker would support Mercurial or git. At the time, neither were available – since then PL have added git support. It was while looking around at the many newer project hosting services that I came across Assembla.

The Assembla proposition is simple and very reasonable: public, open-source projects are hosted for free (up to 200MB). Private plan pricing is extremely keen:

$2.00 per user per space, per month
$3.00 per gigabyte of disk space per month
$8 per user per month maximum. After you pay for a team member in four spaces, additional spaces are free.

Private plans get unlimited disc space.

But for me, the economics of Assembla are only half the story: the other half is a combination of things: the clean, functional web interface, the range of additional tools (Trac, wiki, files, chat, project dashboard etc.) and a strong feeling that the people behind Assembla know what they are doing.

Assembla does offer git hosting, but I’m sticking with Subversion for now. SVN is good enough for my purposes and I can’t live without TortoiseSVN.

I’ve been using Assembla for a couple of months and my experience has been very good indeed.

Camel, HL7, HAPI and NAK Messages

2009-02-17T17:02:00.001Z

The previous couple of posts were related to using Apache Camel in a healthcare integration setting. The HL7 support in Camel is built on the HAPI library which provides Java classes for most of the HL7 v2 variants, plus MLLP support. So in integration terms, HL7 can be considered a combination of an endpoint type (the MLLP/TCP protocol) and a message format (the HL7 ER7 or XML representation).

The Camel architecture nicely isolates endpoint-types and message-structure types in Components and DataFormats, respectively. A Component is something which knows about a specific communication mechanism or protocol: in the case of HL7, the Camel HL7 Component knows how to exploit the underlying Apache Mina library to exchange delimited text messages using the HL7 protocol. The Component also acts as a factory for Camel Endpoint objects: it hands out individual Endpoint objects, each of which represents an instance of the endpoint-type implemented by the Component. Camel DataFormat objects marshal / unmarshal byte-streams or strings to / from Java objects of some type. This makes it much easier to build content-based routes in Camel.

HL7 ACK / NAK Messages

In the previous post, one of the issues I had to deal with in the example route was returning an ACK or a NAK to the message originator. The HL7 protocol uses a positive-acknowledgement scheme: receivers will ACKnowledge (ACK) messages they understand and can process, and will Negatively AcKnowledge (NAK) anything else. The HAPI library provides support for this in the makeACK method of the DefaultApplication class. Unfortunately, this is less helpful when we need to return a NAK.

Recall that the first part of my route was checking to see whether the right kind of message has been received. In the sample, I was testing just the message type (MSH-9.1) and the trigger event (MSH-9.2) fields. In reality, you would probably also want to check that the sender was using the same version of HL7 as you are expecting: you can do this by examining the value of field MSH-12. It will become clear why this is important, below.

Suppose the inbound message fails the MSH-9 test. The otherwise() clause in my little route will send the HL7 data (unmarshalled from the stream) to the badMessage() method on my handler bean. This method expects a single argument of type ca.uhn.hl7v2.model.Message, i.e. a generic HAPI HL7 message object. The argument can't be of a more specific type than this because the Camel route builder cannot know in advance what will be received and unmarshalled from the endpoint; it can only assume it will be a HAPI Message.

Constructing an ACK

In the badMessage method, I need to construct a HL7 NAK message to be returned to the caller. HL7 ACKs and NAKs share the same structure: we just need to change the value of one field (and optionally add a diagnostic message) to indicate NAK. The obvious solution is to call DefaultApplication.makeACK, mentioned earlier, and modify the returned structure as necessary. MakeACK's single argument is a single Segment, the intention being that the caller passes-in the MSH segment of the original inbound message:

Message makeACK(Segment inboundHeader)

The reason for this is that the makeACK method needs some values from the inbound MSH segment to populate fields in the Message Acknowledgement segment (MSA) of the ACK correctly. An HL7 ACK (or NAK) must be associated with the original message (i.e. the inbound message being ACK'd or NAK'd) so that the originator can distinguish which of its outbound messages is being acknowledged or rejected. This is done using the Message Control ID, which is field 10 of the MSH segment.

But remember where we are in the Camel route: we have an invalid message in our hands. We don't know the exact class-type of the inbound message object so we can't cast the Message to a specific type. And the Message interface itself doesn't provide a way to extract the MSH segment. So how are we going to call makeACK(Segment)?

We cannot simply new-up an instance of some specific message type (e.g. a version 2.3 ORM^O01) and use its MSH segment as the makeACK argument:

    // Assume ca.uhn.hl7v2.model.v23.*

    ORM_O01 orm = new ORM_O01();

    ACK ackMsg = (ACK)DefaultApplication.makeACK(orm.getMSH());        // <-- Bang!

    // Set MSA.1 to Application Reject (AR)
    ackMsg.getMSA().getAcknowledgementCode().setValue("AR");

    // etc.

This will compile, but won't work. The manufactured ORM_O01, despite being an instance of a specific version of HL7, is not properly constructed, in several ways. First, MSH-12.1 will not contain "2.3", which is surprising, given that HAPI should be able to populate MSH-12.1 for a new message instance. If we pass the ORM_O01 MSH to makeACK, it tries to get the value of MSH-12.1 but finds a null. This causes makeACK to fall-back to creating and returning an ACK for version 2.4 of HL7 (this is baked-in to the HAPI code - I'm not sure if this is an arbitrary HAPI design decision, or to do with the HL7 spec). And of course we get a ClassCastException at run-time as a result, because we are expecting a v2.3 object.

But even if this did work, isn't it ugly? We'd be baking-in a version of HL7 because we must choose a specific ORM_O01. So how do we create the ACK structure, and how can we make our code work for any version of HL7 supported by HAPI?

Terser to the rescue

Fortunately, HAPI provides a handy utility class for parsing and unparsing HL7 messages of arbitrary versions - the Terser. This will let us get hold of the MSH segment from the original message, so we could at least call makeACK with the original MSH:

    // This still depends on the ackMsg decl. being for the same specific version
    // as the original message.
    Terser t = new Terser(originalMsg);
    Segment msh = t.getSegment("MSH");
    ackMsg = (ACK)DefaultApplication.makeACK(msh);
    ackMsg.getMSA().getAcknowledgementCode().setValue("AR");

Now when we call makeACK we are passing-in a valid MSH so it will extract the HL7 version number and dynamically create the correct ACK class, using Class.forName. But as the comment says, this stilll only works if the compiled-in ACK declaration matches the returned ACK message version. We need to lose the compiled-in ACK declaration, but still find a way to populate the ACK message's MSA segment with the right values.

Fortunately, we can use the Terser to set the field values in a version-independent way. This is exactly what DefaultApplication's static methods makeACK and fillResponseHeader do. So I used a combination of the Terser and the guts of DefaultApplication.makeACK to create a couple of helper methods which do the right thing. I've put the source code on this page in my Google site, for anyone who wants to use it.

Compound Predicates in Camel

2009-02-16T16:15:00.001Z

Thanks to Claus Ibsen for such a helpful response to my previous post. As Claus points out, the answer is to use a Camel helper class, PredicateBuilder, which contains a bunch of useful static combinator methods. Claus shows in his post how to implement my trivial conjunction easily using the and() method. Note the generic signatures of all these helper methods, e.g.

<E extends Exchange> Predicate<E> and(final Predicate<E> left, final Predicate<E> right)

Because they accept Exchange (or any derived class), any custom Predicate you might need to write has access to the complete message exchange structure (in, out and fault), and all predicates could be applied to a custom extension of Exchange.

Having got over those two 'humps' (sorry - couldn't resist that), I'm feeling very good about Camel. It's a mighty impressive piece of work and looks like it has a lot of potential for the area I'm working in right now.

Camel Routes and HL7

2009-02-12T09:07:00.007Z

I've been spending some time with Apache Camel and ActiveMQ, with a view to using Camel in a healthcare integration project. Camel has support for HL7 both at the message structure level (wrapping access to the HAPI libraries) and at the protocol level (via the Apache Mina extensions for HL7 MLLP).

As anyone who's looked at Camel (or IONA's FUSE Mediation Router) will know, Camel implements a rather nice Java DSL for building routes. This lets you write things like:

from("hl7listener")

.unmarshal(hl7Format)

.choice()

.when( header("hl7.msh.messageType").isEqualTo("ORM") )

.beanRef("hl7handler", "handleORM")

.otherwise()

.beanRef("hl7handler", "badMessage")

// end choice block- marshal the ACK/NAK back to the TCP endpoint

.end()

.marshal(hl7Format);

Very readable, but as you might expect, it hides quite a bit of complexity. This is no bad thing, but you do need to understand what's going on behind the DSL.

Camel exploits Spring, and in the above fragment "hl7listener" and "hl7handler" are bean names. The unmarshal and marshal methods allow DataFormat objects to convert between byte-streams and more convenient objects, such as HAPI messages. The choice/when/otherwise operators allow predicates to control routing.

It's taking a while for me to grok Camel; here is a couple of things I'm not yet clear about:

Knowing how to get an intermediate stage in a route to return an ACK or NAK to the originator, without subsequent (or parallel) stages in the route from doing so.
Creating compound boolean expressions in predicates. For instance, if I want a conjunction inside a when(), I can't have it. There don't appear to be combinators for expressions.

In the fragment above for example, how can I express a conjunction/disjunction in the when( ... ) expression? It only seems to be able to handle a single condition: I haven't found a way to combine predicates e.g. I can't do this:

when ( header("hl7.msh.messageType").isEqualTo("ORM") &&

header("hl7.msh.triggerEvent").isEqualTo("O01") ). // etc.

I have tried alternatives, e.g. nesting the when clauses:

from("hl7listener")

.unmarshal(hl7Format)

.choice()

.when( header("hl7.msh.messageType").isEqualTo("ORM") )

.choice()

.when( header("hl7.msh.triggerEvent").isEqualTo("O01") )

.beanRef("hl7handler", "handleORM")

.otherwise()

.beanRef("hl7handler", "badMessage")

.otherwise()

.beanRef("hl7handler", "badMessage")

// end choice block- marshal the ACK/NAK back to the TCP endpoint

.end()

.marshal(hl7Format);

But this doesn't work either. I'm just starting out with Camel, so I expect I simply haven't read the right bit of the documentation (which even the creators admit is one of Camel's weak points).

Another problem is controlling which component in the route returns a response to the originating endpoint (the MLLP sender, in the HL7 case). I may have solved this one, but it's worth setting out the problem and my approach, in case it helps someone else.

My requirements are for the pipeline to log the inbound message (i.e. whatever is actually received on the HL7 socket), unmarshal the message to HL7 (HAPI), then validate the HL7 message type: at this stage in the pipeline I want to return a NAK to the sender (on the HL7 MLLP connection) if the message is not of the expected type, otherwise I want to write the message to a JMS queue for further processing and return an ACK.

The following route appears to work (note that I have omitted the message-logger stage):

// We need the specific HL7 DataFormat object for unmarshalling HL7 ER7 from the

// inbound MLLP link.

DataFormat hl7Format = new HL7DataFormat();

// Inbound HL7 comes from MLLP endpoint.

// Timestamp message, log receipt, unmarshal and perform initial

// handling, which just determines if its the right kind of HL7

// message.

// If so, return ACK to originator and push message to JMS queue.

// If not, return NAK to originator and end.

from("hl7listener")

.unmarshal(hl7Format)

.choice()

.when( header("hl7.msh.messageType").isEqualTo("ORM") )

.marshal().hl7().to("jms:queue:orderMessage.queue")

.otherwise()

.beanRef("hl7handler", "badMessage")

.end()

.marshal(hl7Format);

// The order message processor - pulls message from JMS queue and sends to

// the processor bean, which creates and returns an ACK to the hl7 channel

// if the message is of the expected type.

from("jms:queue:orderMessage.queue")

.unmarshal().hl7().beanRef("hl7handler", "handleORM");

All of this routing can be expressed in XML instead of using the Java DSL. Once I'm comfortable with Camel I may try writing a NetBeans plugin or RCP application to create visuals tools for Camel routes.

Matrix on Windows

2009-01-04T21:20:00.001Z

Very amusing short video, discovered via Philip Wadler's blog. Particularly liked the 'clippy-bending' scene.

Brunel, Robert Howlett and David White

2008-11-29T21:21:00.004Z

Just came across this wonderful short piece on the BBC website. Well worth watching. What a great idea: well done to David White and the BBC.

Modern-day civil-engineering rarely seems to aspire to match that mixture of functional beauty and sense of proportion which Brunel's works all seem to have. But I must put in a good word for the revived, extended and transformed St. Pancras International station. The main concourse is a marvellous place to be (apart from the silly, cheesy, oversized statue of the couple): a couple of times recently, when on our way back to King's Cross station, Dan and I have walked in just to take-in and admire what's been done.

Then we wander back to the dirty, grim, gloomy shed which is the current King's Cross. We've often talked about how wonderful it would be if something similar to St. Pancras were done for King's Cross. The good news is that Network Rail is working on it.

Logging in NetBeans using slf4j

2008-11-28T09:08:00.003Z

It's probably not necessary to start this article with a mini-lecture on why logging, like unit-testing, is a Good Thing, so all I will say is logging is important not only during development but also invaluable in diagnosing runtime problems, once a solution is delivered and running. From here on I'm going to assume you're convinced.

So, you know you should be logging, but how? And which of the many logging frameworks should you use? This article sets out to answer the first question, showing you how to get started logging in Java, using the NetBeans IDE, and offers an opinion on the answer to the second question.

There are several well-established ways to do logging in Java but the best approach I have come across so far is to use slf4j (the Simple Logging Facade for Java) for all logging statements in source code. The slf4j facade interfaces are completely independent of any specific logging implementation. The binding between the slf4j facade interfaces and a concrete logger implementation is done statically when the application jar is compiled: there is no runtime trickery involved (e.g. using class-loaders).

This approach allows you to replace one logging implementation with another without altering any of your logging code, just by binding to a different concrete logger (i.e. referencing a different logger implementation jar).

Splitting the logging job into two in this way means you need two references in your Java project, in order to use slf4j:

a reference to the slf4j API itself, which provides the interfaces you will use exclusively in your logging code.
a reference to a concrete logging implementation, which your logging code will call via the slf4j facade.

As the slf4j interfaces must be bound to a concrete logger implementation, you need to choose which logger to use. I have chosen Logback, which appears to be an improved version of log4j, one of the more efficient Java logging solutions.

This article describes what you need to do to start logging using this combination. I'm going to assume that you understand the value of logging, and know a little about log statements and logging levels: these are common to all modern logging frameworks, so I won't cover them in detail.
Logger Libraries

Download slf4j and logback. Extract the distribution archives to wherever you keep your libraries. Then, in NetBeans, create libraries (via Tools/Libraries) for both distributions.

For slf4j, include the slf4j-api.jar. The precise name of the jar file will depend on the version you download: mine is slf4j-api-1.5.0.jar
For logback, you need logback-core.jar and logback-classic.jar. Again, precise file name depends on version e.g. logback-core-0.9.9.jar

You could create a combined library as a convenience for simple projects, if you know that you will frequently want to use the same concrete logger with the facade. However, bear in mind that for projects which build libraries or components used in other projects, you should only reference the slf4j library: the logger implementation binding should be done by the application which uses your component.

The source code for this tutorial assumes two NetBeans libraries, one for slf4j and the other for logback.

Adding Logging To Your Code

In your NetBeans application project, add references to the slf4j and logback libraries. To start logging from your code, you need to do the following things:

Initialise a logger reference in every class you wish to log from.
Add Import statements for the required slf4j types
Add logging statements to your code.

Refer to the source code with this article. Look first at the Main class: the first declaration in the class is for the logger:

private static final Logger _log = LoggerFactory.getLogger(Main.class);

Notice that this logger is declared static: this is because Main houses the static main() application method, so the logger must be static for it to be referenced from the main method. Above, in the imports section, you will see the two imports necessary to use slf4j:

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Now open the slf4j-sample-lib project, a trivial sample library which the main application references. Look at the StringReverser class, and notice that the logger is declared slightly differently:

private final Logger _log = LoggerFactory.getLogger(this.getClass().getName());

This is a non-static declaration: only non-static methods will have access to this logger. If you look carefully at the declaration, you'll spot that this one is portable: it doesn't contain the class name explicitly. This declaration can therefore be pasted without modification into any class, but remember that this is non-static, so will not be usable from any static methods in the class.

As the static logger is usable from everywhere (non-static and static contexts), wouldn't it be convenient to have that declaration added automatically in every new class you create? Perfectly possible, in NetBeans. To include the logger declaration in your NetBeans source code template, go to Tools / Templates, open the Java branch and select Java class and click Open in Editor. You'll see the Java class template source, complete with template parameters for the class name and some other stuff I'll ignore here. All we need to do is add a single line to the class skeleton:

public class ${name}
{
private static final Logger _log = LoggerFactory.getLogger(${name}.class);

}

If you save the changes to your template and create a new class in NetBeans, you'll see the logger declaration with the correct class name parameter value has been added to the source. You could even add the necessary import statements to the template.

But the overhead of including the logger declaration is low (just one simple line), and NetBeans will take care of the imports for you (use Fix Imports with Ctrl-Shift-I), so you may decide it's not worth adding these lines to your templates.

Static versus Non-static Loggers

There's one more difference between static and non-static loggers which you should at least be aware of. If you declare the logger static, there will be exactly one instance of the logger retrieved for the class, shared across all instances. As the logger reference is shared, memory is allocated just once and the logger is initialised just once, rather than once per object. This is not usually a problem, but you should be aware that the same logger reference will be shared across every instance of the class.

When declared non-static, every instance of your class will incur the memory and initialisation cost for the logger reference variable. For the vast majority of classes this will not be a significant overhead, but again you should be aware of the tradeoff. Note that within the same application, retrieving a logger by a given name will always return the same logger object: using a non-static logger reference does not mean that every instance of a class gets a new logger object of its own.

For a complete account of this issue, you should consult the slf4j documentation which covers this. Read this section of the online documentation.

Logging Statements

As I said in the introduction, I'm assuming you are somewhat familiar with logging. If not, go back and read the material on the slf4 and logback sites, and follow-up the references there.

Use the logger method appropriate to the level you wish to log at, for example:

_log.info("'{}' reversed is '{}'", instance.getValue(), instance.reverseValue());

Possibly the most important piece of advise I can give here is to consider the cost of logging carefully, especially debug-level logging. The actual call to the logger object is not usually the dominant cost in debug logging: you are likely to be constructing moderately large strings to output, and the cost of preparing these may be where you incur the biggest cost. Consider a line such as this:

_log.debug("Debugging string: “ + ExpensiveGetter());

The runtime cost of constructing the string argument to this logging statement will always be incurred, whether the log-level is set to debug or to some other (less-detailed) level. Clearly that's an undesirable, and fortunately there is a very simple coding idiom we can use to avoid this cost:

if (_log.isDebugEnabled())
{
_log.debug("Debugging string: “ + ExpensiveGetter());
}

Simply guard the debug log statements with a test for the current (runtime) level of the logger. This does add a little source code; it is possible to reduce simple statements like the one above to a one-liner but I generally avoid this, valuing clarity over concision.

The cost of the debug statement will now be reduced to a simple boolean test, which is much more acceptable.

Configuring Logback

If you build and run the sample project accompanying this piece, you'll see the log statements are written to the console. This is the default behaviour for most logging solutions, logback included. In most production situations you will probably want to direct your log messages somewhere else, to a file for example.

In common with other logger implementations (such as log4j) the logback subsystem can be controlled through an XML configuration file. For a complete treatment of the options and syntax, you'll need to refer to the logback documentation and in this article I'll give a superficial account of a simple but usable configuration.

Here is the configuration file:

<?xml version="1.0" encoding="UTF-8"?>

<configuration>
<appender class="ch.qos.logback.core.rolling.RollingFileAppender" name="RootFileAppender">
<file>application-log.txt</file>
<append>true</append>
<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>debug</level>
</filter>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<fileNamePattern>application-log.%i</fileNamePattern>
<maxIndex>2</maxIndex>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<maxFileSize>100000</maxFileSize>
</triggeringPolicy>
<layout class="ch.qos.logback.classic.PatternLayout">
<pattern>%d{yyyy-MM-dd HH:mm:ss},%p,%c,%F,%L,%C{1},%M %m%n</pattern>
</layout>
</appender>
<root>
<level value="debug"/>
<appender-ref ref="RootFileAppender"/>
</root>
</configuration>

Although this is quite a simple configuration file, there's a lot of information in there and to understand it all fully you will need to study the logback documentation. I'll skim over the most important bits here and leave the rest to you.

Appenders are the part of the logging engine which actually write the log messages somewhere. You can choose from many different appenders – my favourite is the RollingFileAppender which writes to a text file and 'rolls-over' to a new file once the existing file passes a certain size, declared in the triggeringPolicy element. The file element declares the filename, and the rollingPolicy/fileNamePattern declares the filename pattern to use for old log files, as the file rolls-over.

Layouts define the format of the messages written to the log appender. I've gone with a standard pattern which appears in the logback documentation: please consult the logback documentation on format strings to discover all the options here. Essentially, every log statement will include the date and time, and the origin of the log message, including the line number.

So the net effect of this configuration is that logback will write all log activity out to the file application-log.txt in the working directory, rolling the file when it reaches the size limit I have set (roughly 100k).

Logback and the Classpath

As explained in the logback documentation, if you want to configure logback with an XML file then the file must be present on the classpath of the application, otherwise logback will not find it and will fall-back to console logging.

If you examine the source code accompanying this piece you'll see that I have placed the logback.xml file in the root of the source folder (src). In the NetBeans project, this is the 'default package': the name of anything placed here will have no namespace prefix. When NetBeans builds a Java application, it will compile any Java sources it finds in src and all its subdirectories, and will also by default include everything it finds in the root folder in the jar file. To prove this to yourself, go into the src folder, create an arbitrary file, say 'fred.txt', and rebuild the project. Now open the NetBeans Files view (Ctrl-2 will open this if it's not already visible), locate the 'dist' folder and open the tree until you locate the slf4j-sample.jar file. The NetBeans file view allows you to browse the jar file contents – just open the tree to the next level:

And there are the two files, logback.xml and fred.txt, packed into the root of the jar. We don't need fred, so delete the file and rebuild the jar.

So the source hierarchy in NetBeans contributes to the runtime classpath, and the 'dist' folder is where NetBeans writes out all artifacts resulting from a build. As the NetBeans documentation (and the generated readme) tells you, you can distribute your application by zipping-up and publishing the contents of the dist folder.

Let's run the application. Open a terminal window in the dist folder and type:

java -jar slf4j-sample.jar

The program should run, but nothing should be written out to the console. Instead, an application-log.txt file should appear in the dist directory. Open that file and you will see the result of the logging statements. Logback has obviously located the configuration file and applied the settings.

If you're thinking 'job done' at this point, not so fast! Just consider how logback has located and loaded the XML file. The logback XML file is packed into the (executable) jar file which is the main artifact generated by the build. When the Java runtime runs this jar it first examines the jar's manifest (the MANIFEST.MF file in the META-INF folder inside the jar), which looks like this (from the sample application jar):

Manifest-Version: 1.0
Ant-Version: Apache Ant 1.7.1
Created-By: 11.0-b15 (Sun Microsystems Inc.)
Main-Class: slf4jsample.Main
Class-Path: lib/logback-classic-0.9.9.jar lib/logback-core-0.9.9.jar l
ib/slf4j-api-1.5.0.jar lib/slf4j-sample-lib.jar
X-COMMENT: Main-Class will be added automatically by build

The runtime looks for the entry point of the program (the qualified name of the class containing the main method) and initialises the classpath from the Class-Path setting. Implicitly, '.' (the root of the jar structure) is also on the runtime classpath.

What this means is that when the logback system goes looking for logback.xml it will find the copy in the jar file. Consider the function of the logback.xml file: the users of the application will almost certainly want to configure the log behaviour themselves rather than accept the settings you have baked-in to the copy of logback.xml inside the jar file.

So, we want to distribute our default logback.xml file but not have it baked-in to the jar file: what to do? Having it in the source classpath is convenient because it appears in the NetBeans project view so we can view and edit it from inside the IDE. However, we don't want it packed into the jar, but we do want it on the runtime classpath. How can we satisfy all of these constraints?

We can achieve this with two simple steps:

To prevent NetBeans including the XML file from the jar file, we exclude logback.xml from the list of source artifacts that NetBeans considers as jar contents.
Add a step to the build process which copies logback.xml into the dist folder, placing it next to the jar file.

For step (1), open the NetBeans project properties dialog and navigate to the Packaging subsection, below Build. Add “logback.xml” to the list of Excluded files:

If you rebuild the application, then use a zip tool or archive manager to open the resulting jar file, you'll see that the logback.xml file is no longer included. There is more help on excluding content from your jar in this page of the NetBeans documentation.

For step (2) we need to arrange for the logback.xml file to be copied into the dist folder during the build process. To do this, we will add a very simple custom build-step to the NetBeans build process. As you probably know, NetBeans uses Apache Ant as the build engine, and the IDE generates the Ant script automatically. I'm not going to go into much detail concerning what you can do with custom build steps – I suggest you read the well-commented Ant scripts, and look at this section of the NetBeans documentation.

There are standard stages in the build process where we can add tasks ('targets' in Ant-speak) to the NetBeans build. The build.xml file (in the project root) contains a comment describing these. All we want NetBeans to do is copy the logback.xml file into the dist folder after it has finished building the jar file, so we will add a -post-jar target to copy the file:

<target name="-post-jar" >
<copy file="${src.dir}/logback.xml" todir="${dist.dir}"/>
</target>

Add this target at the end of build.xml, just before the close-tag for . Now rebuild the project, and take a look in the dist directory: you should see that logback.xml has been copied there. Now you're ready to zip the dist folder and distribute your solution.

That's the end of this little piece. You should know enough now to add straightforward, high-performance logging to your own projects. But there's a lot more to learn: to go further, study the slf4j and logback documentation, follow the examples, and experiment.

Leonard Cohen at the Albert Hall!

2008-11-20T15:31:00.001Z

On Tuesday my wife and I were finally able to use her (rather belated) birthday present from me - tickets for Leonard Cohen. At the Albert Hall! It was spectacularly good - great seats (stalls, so good line of sight over the arena floor), amazing sound quality, and of course the Man himself, a living legend, who delivered an astonishing 3 or more hours of the finest entertainment I can recall. The band were superb: every member individually talented but, as with all the best ensembles, the whole was even greater than the sum of the parts. I can't recall a better evening.

But it so nearly didn't happen, for us ... I had originally bought tickets on eBay, the only place I had been able to find really good seats for the London concerts so-far announced, which were at the O2 Arena. Not a venue we fancied much, but at the time I don't think the Albert Hall dates had been arranged. The O2 tickets were in the first couple of rows right in front of the stage, so we were pretty pleased with ourselves. I paid, and waited.

And waited ... and chased the seller, again and again. Who turned out not to have the tickets at all, but was "expecting to receive them from the promoter" shortly before the date. Lots of promises about keeping us informed, lots of reassurances they would arrive in time. Until eventually, two days before the gig, I receive an email from the seller saying that some tickets have arrived but are for the wrong event! By now we're way past the eBay 90-day limit for disputes. Moral: do not trust individuals selling concert tickets on eBay. If you do go ahead, demand the tickets or your money back well within eBay's limit.

Fortunately, I found an online ticket company (Double8tickets) with the AH seats, via a Google ad which appeared in GMail! Normally, I ignore these ads, but this one leapt out at me at a time when I was desperate to find tickets, so I wouldn't have to disappoint Chris. Just goes to show that these unobtrusive little ads can work very effectively, and are so much less irritating than the loud, flashy mess that flickers away above Microsoft's Live or Hotmail frame.

Community and Culture

2008-11-14T22:29:00.001Z

While looking around for Java dependency tools I came across KirkK's site, and his JarAnalyzer. As usual, I wanted to know a little more about the person behind the software, so looked up his blog and found an article from 2007 there which really resonated with me: .Net : Software & Technology @kirkk.com

Kirk and I have travelled in opposite directions: he crossed the tracks to work on a .NET project, whilst I've recently shifted my attention almost completely from .NET and the Microsoft environment (where I have spent the last 10 or so years) to the world of Java. What's interesting is that he so quickly formed the same view of the Microsoft development that motivated my move to Java.

Microsoft's greedy behaviour has done so much damage to its reputation and the level of goodwill amongst independent developers. Kirk cites the example of TestDriven.Net, but there are examples of other 'alt.net' type projects (NAnt and NDoc, for example) which Microsoft has effectively (though not directly) either killed or marginalised, not with licensing terms but by introducing proprietary (and arguably weaker) competing technologies. I'm sure that part of the reason for that, with the N-prefixed projects at least, was that they couldn't bear the prospect of absorbing something with what they would see as alien DNA into their product line.

This piece by Mike Hofer describes the NDoc demise and nicely summarises the twin underlying problems: the nature of the development community surrounding the Microsoft platform, and Microsoft's inability (or unpreparedness) to work with it. Mike's article, and the linked post by Charles Chen containing the email from NDoc's founder, make for quite depressing reading. Perhaps the emergence of a mean-spirited, mean-minded community is to be expected, when the centre of its universe is an avaricious commercial juggernaut?

I feel these things especially keenly, now that I'm looking over the wall from the Java community side. The contrast really couldn't be more stark, more impressive and more compelling. Even the large companies operating in this space, notably Sun Micrososystems, appear intelligent, enlightened and innovative; there's a very healthy culture here. (A quick and revealing experiment: take Microsoft and Sun - now try to find the corresponding CEO's blog. Top hit in Google for Steve Ballmer when I tried was a send-up site; top hit in Google for Jonathan Schwartz was Jonathan Schwartz's blog. And it's worth reading).

Coming back to the issue of building a healthy community around a technology, Sun's Java is surely the shining example of how to embrace the great work done by independent developers and build on it, rather than trying to crush it. Just look at NetBeans: this is an enterprise-quality IDE, easily the equal of Visual Studio, but not only is it open-source (and free to download) it also employs established open-source tools instead of imposing inferior alternatives: For unit testing, JUnit is completely integrated; NetBeans uses Apache Ant as its underlying build-system; it can work seamlessly with Maven through the excellent plugin. And you choose the version-control system you prefer (e.g. Mercurial, Subversion) and NetBeans will allow you to make full use of it, right inside the IDE.

Lastly, you are free to extend NetBeans by writing whatever plugins (modules) you please, without running the risk of getting into litigious exchanges like the TestDriven.Net debacle described here, which seems to me to plumb the very depths of time-wasting pointlessness.

I never intended to write all that: it was originally just a reaction to a (rather old) blog entry. But as I revisited the world of the Microsoft monoculture through the tale of those N-projects, it just tumbled out.

The Managerese Disease

2008-11-14T15:02:00.001Z

I recently stumbled upon a wonderful collection of that language we all love to hate: managerese. Kevin Boone's list is here:

The K-Zone: New developments in managerese

There are other lists like this sprinkled around the net, but this one is the best I've seen so far because it manages to combine a bit of thoughtful deconstruction with some satisfying disdain.

NetBeans - blank dialog box fix

2008-10-09T23:17:00.003+01:00

I've been using LinuxMint 5 for some weeks now, and a very fine distro it is, too. I've noticed that NetBeans 6.1 starts and runs a great deal faster on Mint than on Windows XP - the difference is quite significant. I've been wondering why that is, given that this is Java and therefore presumably only the low-level loaders and file-system interactions differ across platforms. That's just a passing observation and not the point of this post.

I found one annoying thing happening occasionally (but often enough to be irritating): NetBeans dialog boxes would appear completely empty. Take a look at the screenshot:

There didn't seem to be any pattern to this behaviour; it wasn't always the same dialog, and if you closed the dialog and opened it again it would appear normally. Not good.

Well, it turns out this is known problem when using the Compiz window manager on Linux - a quick search revealed a good post on the NetBeans forums about this subject. The answer is to use the latest JRE / JDK build as it addresses this compatibility issue. I decided to download and install JDK 6u10 RC and give it a try.

To get NetBeans to use the newer JDK, you can do one of two things:

Edit the NetBeans configuration file (in <installationFolder>/etc/netbeans.conf) and change the path pointed to by netbeans_jdkhome
Change your system default Java installation.

Not very keen on (1) - feels like I'm hiding a workaround in just one application's config, but the problem exists for any Java application I run on this system.

So I decided to do (2). It worked perfectly, so thought I'd share the steps I used to do it. First, where is Java actually installed on the filesystem? If you do 'which java' in a console, it will report /usr/bin/java. But if you look closely, this is just the first step in an indirection; the ls tool reveals:

$ ls -l /usr/bin/java
lrwxrwxrwx 1 root root 22 2008-10-01 06:55 /usr/bin/java -> /etc/alternatives/java

And /etc/alternatives/java is another step:

$ ls -l /etc/alternatives/java
lrwxrwxrwx 1 root root 36 2008-10-01 06:55 /etc/alternatives/java -> /usr/lib/jvm/java-6-sun/jre/bin/java

And the /usr/lib/jvm/java-6-sun file is actually a symbolic link to the actual Java installation folder, which is in the same folder. That's a lot of indirection, and I am quite sure there are good reasons for all of it, but I haven't time to learn all of them. There is a very detailed post by Anthony Richardson which patiently explains how to create a proper DEB package from the JDK download; this is almost certainly a good idea.

I chose to exploit the fact that the last step in the indirection chain is that symbolic link file. By unpacking the JDK directory into /usr/lib/jvm and getting the java-6-sun link to point to the new JDK folder, I decided I could replace 6u6 with 6u10, system-wide.

However, there is another file in this folder: the .java-6-sun.jinfo file (this is a '.' file - you'll only be able to see those if you use the a switch with ls (e.g. ls -al) or View / Show Hidden Files in Nautilus). Looking inside it, only the first line appears to contain version-specific stuff - everything below uses paths which use the symbolic link:

name=java-6-sun-1.6.0.06
alias=java-6-sun
priority=63
section=non-free

jre ControlPanel /usr/lib/jvm/java-6-sun/jre/bin/ControlPanel
jre java /usr/lib/jvm/java-6-sun/jre/bin/java
jre java_vm /usr/lib/jvm/java-6-sun/jre/bin/java_vm
jre javaws /usr/lib/jvm/java-6-sun/jre/bin/javaws
jre jcontrol /usr/lib/jvm/java-6-sun/jre/bin/jcontrol
<snip>

I decided to leave this as-is for now, and edit it later if required. So, in summary here is what I did:

Copied the JDK 6u10 contents into /usr/lib/jvm.
Went to the directory /usr/lib/jvm and opened a gnome-terminal as root (you need to be root or use sudo, to make changes here).
Completely unnecessarily, I backed-up the old symbolic link file, just in case. The easiest way is to use mv (by default mv doesn't follow sym-links), but if you'd prefer to make a copy of a symlink (rather than the object to which it points) you need to use the -P switch, e.g.
cp -P java-6-sun java-6-sun_OLD.
Made a copy of the jinfo file:
cp .java-6-sun.jinfo .java-6-sun.jinfo_ORIGINAL_1.6.0.06
Made java-6-sun symbolic link point to the new JDK directory:
ln -s jdk1.6.0_10/ java-6-sun

And it does seem to work. NetBeans reports that it's using 6u10, and I haven't seen any empty dialogs yet! Better yet, NetBeans seems to start and run faster, too. (I haven't timed it - this may be placebo effect...)

Hope this may help out other folk using LinuxMint or Ubuntu, facing the same problem. Of course, remember that you can easily disable Compiz (set Visual Effects to None in the Appearances Preferences), and you can get some simple effects back using Gnome Compositing, available via the Mint Desktop tool in Control Center.

Interlocking Fragility

2008-10-09T09:53:00.001+01:00

The October 1st edition of Edge includes the following quote from Nassim Taleb's book The Black Swan. Relevant, given what's happening right now:

From Edge 259 (about 3/4 of the way down the page):

NOTABLE QUOTE

"Globalization creates interlocking fragility, while reducing volatility and giving the appearance of stability. In other words it creates devastating Black Swans. We have never lived before under the threat of a global collapse. Financial Institutions have been merging into a smaller number of very large banks. Almost all banks are interrelated. So the financial ecology is swelling into gigantic, incestuous, bureaucratic banks – when one fails, they all fall. The increased concentration among banks seems to have the effect of making financial crisis less likely, but when they happen they are more global in scale and hit us very hard. We have moved from a diversified ecology of small banks, with varied lending policies, to a more homogeneous framework of firms that all resemble one another. True, we now have fewer failures, but when they occur ….I shiver at the thought."

— Nassim Taleb, The Black Swan (2006)

Disclaimer: I haven't read this book yet. Some comments I've heard or read suggest that it's superficial, rambling pseudo-science, more a collection of entertaining but relatively facile observations (perhaps including the above) than an exposition of something profound.

Still, this one seems to have come true.

Neil Bartlett's OSGi book

2008-09-23T09:20:00.001+01:00

Just found Neil Bartlett's OSGi book project:

"OSGi in Practice"

Looking good so far, but I've only skimmed the first couple of chapters. I plan to read and contribute to the comments and bug-tracker.

I don't think there is another good OSGi book out there yet, so this is potentially important. Great that Neil has decided to license the book under Creative Commons, too. Like most folk, if the book is good then I'll buy a paper copy.

OSGi, Java

Ubiquity, and Contrasting Cultures

2008-09-19T12:14:00.001+01:00

This is interesting: Mozilla Labs » Blog Archive » Introducing Ubiquity

Watch the video demonstration. What a great idea, and I'm impressed by how well this appears to work even at the prototype stage. I'm going to install and try it.

But even if it turns out not to be so great an experience for me (demos always work perfectly, don't they?), that's not the point: what matters is that some bright people are doing interesting and worthwhile things and freely sharing the outcome with us, while Microsoft apparently spends its time (and some of its money mountain) on stuff like this.

Links to these two articles were close to each other (can't recall where) and I was so struck by the contrast and what it reveals of the cultural differences that I felt compelled to write this.

Rick Wright

2008-09-15T20:23:00.001+01:00

BBC NEWS | Entertainment | Floyd founder Wright dies at 65

Such a shock to see this BBC news headline. Like many of my generation (40-something), I regard The Dark Side of the Moon as the key soundtrack of my early life. As a teenager I knew every word and every note of every song (probably still do) from an album I listened to endlessly back then, and have listened to on-and-off ever since.

The first copy I had was a copy of a school-mate's album, on a C90 cassette. I borrowed a typewriter to write out the lyrics. Somewhere, in the bottom of a box, I probably still have those sheets of paper. I wore out that tape. Eventually I'd saved enough for my own copy, but (would you believe it) I lent the album to someone and never got it back. But I'm sure I've got the posters, probably in the bottom of that same box, wherever it is. It's just impossible to explain how important that music was. It still affects me, even now. It will always be my favourite album. Bar none. Nothing can displace it, ever.

"I am not frightened of dying.
Anytime will do, I don't mind.
Why should I be frightened of dying?
There's no reason for it,
You've got to go sometime..."

(from The Great Gig in the Sky. Rick Wright / Clare Torrey)

Google Protocol Buffers

2008-09-15T10:28:00.001+01:00

The Protocol Buffers (PB) idea is obviously good from a performance point of view, but with so much messaging already defined in XSD, I want a tool which will generate the .proto file for a given XSD. Does such a thing exist yet? If not, it would make an interesting project. The obvious twin would be a library (linked to the generated .proto code) which accepts a DOM or a SAX stream and generates PB data - a kind of PB-adapter.

The value of protocol buffers is mostly in creating more compact representations on the wire, and speed in parsing/unparsing. But there's a lot of code out there already using XSD/XML internally which nobody wants to rewrite just to use PBs. Would the cost of conversion to/from an in-memory DOM cancel-out the gains?

Generating PDF from XML with XSL-FO

2008-09-12T16:10:00.007+01:00

One of the healthcare solutions I'm working on has to generate diagnostic report documents in a form we can distribute directly (e.g. secure email) to clinical staff. The diagnostic report text plus the usual patient demographics and such, arrives in a custom XML message from the integration engine. The solution uses HL7 v2 elsewhere, but the PAS messaging is exclusively via custom XML (not HL7 v3 XML).

The legacy document-generation subsystem used Word to create the document: a template was created to get the layout and formatting right, and simple placeholders used to mark the locations of the data we would substitute, taken from the XML data. This approach actually loads Word into memory (on the server!) to do the work - it's slow, memory-hungry and just generally clumsy and ugly. Plus you end up with a Word document, so the clinician needs Word (or the reader) to view it.

PDF is a more acceptable format, in my view. We can secure and digitally sign the document when we generate it to prevent subsequent changes. Recipients can view PDF on any platform with a free viewer. The problem for me was how to generate the PDF programmatically, from the XML data. There are probably several ways to do this, but I chose XSL-FO and the Apache FOP project mainly because I wanted to avoid using a proprietary PDF generator product (there are lots out there), but also since XSL-FO can do more than just generate PDF.

First problem: how do you create the FO which is the 'shape' of the generated document? Of course, you could simply read through all the manuals and write one from scratch. Well, I'm just plain lazy, you see, and I don't want to do all that. I want to take my nice OpenOffice document, or even Word document, and have a tool create an XSL-FO for that document. And I'd like the tool to be free (there are commercial tools of course, but I'm cheap). Does such a thing exist?

It does! Amazingly, precisely this facility exists in Abiword: not my favourite word-processor by a long, long way, but a good solution for this particular problem. OpenOffice should be really good at doing this, as it stores documents natively as XML and already uses FO internally for some style information. But, despite some promising hints, there is no mature support for this. This is a real shame: this is just the sort of thing OOo should be capable of, especially as it's apparently half way there already.

Here's what I managed to find on the OOo site:
Some OpenOffice transforms and things.
And this one talks about ooo2xslfo but that project looks as if it's just a SVN repository - not sure I have time to play with that.
The OOo XML site has a page on XML filters too.
Important to mention also that Microsoft does have an XSLT which you can apply to Word documents to generate XSL-FO. It's freely downloadable from this download page. I tried this, and it works, but the resulting FO is much messier than Abiword's.

Once you have the FO, the obvious step is to embed it in an XSL, add xsl:value-of elements in the appropriate places and use a transform to populate the template. This is the approach I took for the proof-of-concept and it worked well. The resulting PDF looks almost right - with a small amount of FO-tweaking, we should have something very usable.

But using XSL means loading up and running the (trivial) transform which I think may be very inefficient for such a simple case, plus it requires the FO to be edited. I've decided to use a simpler approach (using StringTemplate) which I hope will be more efficient, and requires less FO editing (just the addition of $fieldName$ placeholders). All we need is a list of (fieldname, XPath) pairs for our XML message, in order to drive this template. Of course, most other applications will need the power of XSL (e.g. to deal with tables of entries): I'm only avoiding it here because the data is so simple.

This is something we're bound to want to do again, in different contexts, so I'm using this project and the prototype to build a tool-chain and utilities for this, so we can use the same approach more easily next time.

Google Chrome, Part 2

2008-09-07T22:15:00.004+01:00
Mmmm. It all looked so good in the cartoon. The reality isn't quite there yet, but you can see where this is heading, and overall I'm optimistic. Here's a summary:
Windows only at the moment. Understandable, given the market share stats, but what a pity I can't run it on this LinuxMint laptop. Wondering what underlying runtime they're using: surely one of the key features of Chrome is that it will become a client-side platform. They need to be able to run multiple processes (presumably native processes) to get one process per tab.
On Windows XP, I found Chrome simply ate CPU cycles. The memory (working set) story wasn't as bad as I had expected (no worse than Firefox, anyway) but the CPU cost meant I was experiencing significant interruptions in other applications. I did experiment a bit, but so far I haven't been able to characterise the circumstances under which I see this.
Loved the ability to drag a tab out of the frame to create a standalone 'application', e.g. Google Documents or GMail. That works very well.
Rendering speed seemed higher than Firefox.
Plugins for sound and video worked for me (e.g. BBC news) but I really didn't set out to test this with different formats.
The lack of my Firefox add-ins took a little adjusting to! I'm sure this will eventually come.
The UI is quite bare, but you get used to it. The downloads bar (at the bottom of the screen) is actually rather good - better than FF.
The search/address bar (can't recall what they call it) is excellent - I found it worked very well for me.
The fact that Chrome imports history etc. from FF if you want it to (I did) means it does very nearly allow you to pick up from where you left off in FF.
On the whole, very impressive. But I'm not ready to replace FF just yet.

Google Chrome

2008-09-02T11:37:00.001+01:00

I must have missed the Chrome story, as I've only just seen a pointer to it from Darren Waters BBC blog piece. Having just read quickly through the Google Chrome storybook, this looks like a potentially important and exciting development. The storybook format is really excellent, too: what a contrast with the way other software companies (e.g. Microsoft) introduce a product!

Technically, this looks like a winner to me. Using a process instead of a thread per tab is a sound idea in principle, with lots of benefits (explained in the cartoon) which you pay for with a slightly higher initial resource footprint. I've little doubt that this will be a worthwhile price to pay, though: almost all of us have machines with enough CPU and memory to accept this.

It seems to me that the most important thing Google has done is to recognize that the nature of the browser has changed utterly, from what was simply a way to view HTML through to something which is trying to be a complete application platform. With Google Mail, Google Apps and Gears, plus a handful of add-ins, Firefox is currently central to the way I work, but because the underlying architecture is still anchored in the past, Firefox isn't going to be stable or performant (or secure) enough to cut-it, for much longer.

This seems to be Chrome's architectural starting point, and the team have sensibly decided to start with a clean sheet of paper, rather than simply build just another branded browser on top of old-world technology. In some ways, I wish the Mozilla engineers had gone down this route. Firefox has been a great success, but if Chrome is as good as the comic-strip suggests, I think Mozilla will need to raise their game significantly.

Climbing in Great Langdale

2008-09-01T09:51:00.012+01:00

On Friday, Becca and I had a wonderful day climbing and abseiling, in Langdale. We're both beginners with some experience of climbing walls but none so far on real rock. I managed to get a day's climbing instruction with Adam Marcinowicz of Adventure Peaks, via the Destination Cumbria agency. I can recommend both Destination Cumbria and Adventure Peaks: within hours of my initial enquiry, Lindsay Gibson had located three good options for us, and once we'd chosen AP it was all arranged quickly and efficiently. Great service.

We met Adam and colleagues at the Adventure Peaks store in Ambleside. AP is in the business of organising and running real expeditions to challenging places - the shop is full of serious (and expensive) equipment - so our little day out is really small-beer.

We spent the morning climbing on Lower Scout Crag, in Great Langdale. This is a beautiful place, but somewhat busy. It wasn't long after we got started that two other groups arrived and began setting up. We tackled two routes, Cub's Wall and Cub's Crack.

The first is relatively easy, but the crack is far harder: the initial moves required to reach good handholds completely defeated me! This was due to a combination of lack of experience / poor technique, and my hands and forearms being weak and out of condition. Becca finally made it up this route, after hanging on the rope and persevering.

We had a brief lunch stop on the pass between Great and Little Langdale, looking out over the valley to Gimmer Crag - beautiful scenery, but we were plagued by flying ants!

Then Adam took us to Cathedral Quarry, to do an abseil down the 30m quarry face. The walk from the car takes you through attractive country, and eventually into the quarry tunnel which leads into the cathedral cavern.

There are other tunnels leading back through the rock to the approach path - we took a wander through these, too. They are at a constant (low) temperature and pitch dark: climbing helmet or torch (or both) recommended, as I was glad of the helmet on one or two occasions!

The climb up to the abseil point is a scramble up some steep and wet paths - you need to take care here as you're more likely to get hurt on the way up here than on the way down the rope.

The abseil is around 30m, and sheer. About two thirds of the way down the rock face ends and you lower yourself down past the cavern mouth. At the top (quite unprotected and with no warning signs!) I was initially somewhat nervous. Like most folk, I don't like edges at height, so I prepared myself mentally for something I knew I was going to find challenging. I concentrated on two things: (1) treating it as exactly the same as coming back down from a climb up, and (2) trusting the equipment. It worked! By the time I was attaching myself to the rope, I was ready.

As I leaned back, put load onto the rope and stepped onto the face, I felt OK, though my heart-rate was a good bit above resting! It was really only that initial step over the edge which was hard - once I was comfortably into the abseil, I didn't really think about the height.

I ended up doing the descent twice, the second time with a different (and better) belay device which made it more comfortable. Becca managed three descents!

Someone has posted a YouTube video of their descent, which (if it's still there) gives you some idea of the location. I think their estimate of the height (140 feet) is a little on the high side, though: Adam thought 30 meters or so, which is nearer 100 feet.

We had a fantastic day, achieving more than I had expected, all thanks to Adam who was patient with us, and worked hard to give us the best possible experience.

Loweswater Lake

2008-08-26T08:14:00.003+01:00
It rained almost all of yesterday, but we still managed to get out for a decent walk, this time alongside Loweswater Lake which is one of the smaller lakes up here, but in a pretty setting, almost joined to the next lake, Crummock Water.

The walk we took takes you along a good path which lies about half way up the ridge overlooking Loweswater, eventually descending to a farm at the end of the lake, then back along the lakeside, past the National Trust bothy and the rowing boats for hire, which looked rather forlorn in this weather. We saw very few other people walking today, not surprisingly. The tops of the hills were hidden in cloud a lot of the time, but in true Lake District fashion they would suddenly clear, yielding glorious views.

Slightly annoyed to discover that my supposedly waterproof Berghaus jacket, isn't. You discover the meaning of 'waterproof' in this rain!

Lake District Break

2008-08-24T22:00:00.004+01:00
We're enjoying a very welcome week off, away from it all in the Lake District. We are staying in a really special place, Winder Hall in Low Lorton. Originally a Jacobean manor house, it looks like the Victorians had a hand in updating the interior, and much more recently the owners have created a magnificent place to relax. In the grounds there is even a hot-tub, and a summer-house containing a sauna! We've taken advantage of both - fantastic. The food is good, too - if you ever stay here, then be sure to have some of Clodagh's home-made seville-orange and lemon marmalade, it's epic!

There are some great pictures of Lorton Hall (before it became the Winder Hall hotel) on the Visit Cumbria website. Scroll down about half way to see the house and grounds, and the Pele Tower which forms part of the hall.

The surrounding area is simply outstanding. Today we walked up to the top of Barrow, close to Braithwaite, which is a short drive from Low Lorton. Barrow is not that high (around 1500 feet) but you get one of the best views in the Lake District from the top. The weather has been kind, so it was possible to see all the surrounding fells and a long way further out. This was a warmup for longer, higher walks later in the week - I'd like to get to the top of Great Gable, about twice the height of Barrow.

NetBeans 6.5 Unusable?

2008-08-21T09:17:00.003+01:00
I recently installed NetBeans 6.5 Beta on both OSs on my laptop (Acer 2GHz dual-core, 2GB, dual-booting Win XP and LinuxMint) and I almost immediately uninstalled it. What is going on? Some of the UI appears to be broken: controls not lining up correctly, some dialogs displaying completely blank contents, some the wrong size (e.g. far too big). Sometimes the last two could be fixed by closing/re-opening, but not every time. It also felt a lot slower than 6.1.

Seems other people have been having the same experience. A lot of what Casper wrote in his piece resonated with me, as I also come from the Visual Studio / C# background. I do hope the NetBeans team pays attention because this is an important product which is already good, and deserves to be better. I really don't want to use Eclipse, and I'd prefer not to pay for IntelliJ. Come on Sun!

97 Things Every Software Architect Should Know

2008-08-21T08:48:00.003+01:00
Ran across another one of those 'things every architect should know' sites, via Michael Nygard's blog. I haven't read every single one yet, but my unscientific selection yielded a few really good ones, and one or two I felt were less worthy (and not really architecture-specific anyway). I found myself using the titles to select which ones to read in detail, and this proved a pretty reliable guide to the value of the idea and the quality of the writing.

A side-effect of looking at this list was being asked to sign-up to Near-Time. I haven't yet figured out exactly what this is, but it appears to be a public, Web-2.0 collaborative workspace thingy. As usual, I'm probably way behind everyone else on this.

When Google Owns You

2008-08-12T11:45:00.000+01:00
I must admit I do sometimes wonder how I would cope if Google decided to pull the plug on my account. For me, I would find life without Google (especially GMail, Google Docs, Google Reader and Google Maps) very hard indeed. It's happened to Chris Brogan. Read his post When Google Owns You | chrisbrogan.com.

Finding a replacement which doesn't suck would be hard right now. I just don't really like Yahoo or Microsoft Live much (though I have accounts on both), and even with a replacement service I'd still not have access to the giant heap of old email and documents which Google does such a great job of storing and searching for me. And, despite other peoples' suspicions that Google no longer lives according to its founding principle ("Don't be evil"), I instinctively trust Google much, much more than Microsoft or Yahoo.

All this great Google stuff is completely free, so it's very difficult for any of us to complain if it does get withdrawn. I don't for one moment think that Google is likely to do that without good reason, but it puts all of us in a position of having little or no leverage. Personally, I'd be more than happy to pay a nominal amount for my Google services (by which I mean almost everything except search), to ensure continuity of access, security of my data and a contract which means I'm owed the services. How much? I don't know - what about 30 USD a year? That's unlikely to upset anyone who really wants or needs the service, but if enough people paid it could provide Google with a ton of additional money they could use to invest in making those services even better.

Will Google ever transition to a paid model? The Picasa photo service has a free and paid service already, probably because Flickr and others do this as well as to cover storage costs. But for the rest of Google, I'm not sure. You can imagine how administrative overhead might soar with paid accounts: not only would they be obliged to keep track of everyone's identity, account and payments, but armed with a contract, the kind of people who are intolerant of even occasional outages would feel entitled to complain loudly. I wonder if the folks at Google have 'done the math' and decided that any additional revenue would be eaten up by this sort of thing?

pastebin - collaborative debugging tool

2008-08-11T21:04:00.001+01:00
pastebin - collaborative debugging tool

Seen this? This is one of those truly great, simple ideas. Check it out.

Java and Ruby, C++ and VB

2008-08-08T13:44:00.001+01:00
Were you doing Windows development, about 10 years or so ago? If so, then I'm pretty sure you'll recognize the following.

The hard stuff, or more accurately, all the performance-critical stuff and the core components of your product would be written in C++. The functionality in the libraries you and your team created was then composed into COM components, and you designed interfaces, wrote IDL and created type libraries which exposed the component functionality to the outside world.

Lesser mortals could then use COM-aware languages and tools, like Visual Basic (VB), to create sophisticated applications to exploit the power of those components. Component-oriented software development was a reality. UI designers, the subject-matter experts and the VB programmers could get together and build something using your C++ classes, all without having to know about vtables, pointer arithmetic, or what a pure virtual function is.

This was COM (and OLE2) and it was great technology, because it allowed us to do mixed-language development, to compose systems from components without necessarily knowing how they were implemented internally. We could use the right language for the task in hand, which might be C++, Delphi, VB or even JavaScript, knowing the resulting component could be used by any other COM-aware environment.

It wasn't perfect. The COM abstraction was somewhat leaky because even when using 'scripting languages' like VB to combine COM components, you still needed to understand COM's reference counting semantics, and marshaling of data across the COM boundary wasn't always straightforward. But the benefits manifestly outweighed any of the technical difficulties.

With Java and especially .NET we moved away from the idea that we'd build lower-level functionality in one language and use a less formal or type-safe language to 'script' applications on top of it. Microsoft have always claimed that mixed-language development is a cornerstone of .NET and the CLR. It's certainly true that the CLI does explicitly set out to enable multiple languages to target the infrastructure, but this misses the real point, which is that relatively few projects appear to have exploited this. Most .NET solutions are developed in just one language, either C# or VB.Net, the choice usually made early, based more on team culture than on technical grounds.

I haven't heard anyone saying "Well, we used C# for the hairy low-level components, but we built the application itself in VB.Net because it's so much quicker to prototype". Does this happen now? I don't think so, yet back in the C++ / VB world, it really did.

I believe we are in a roughly similar position now, with respect to building complete solutions, as we were when VB and C++ were joined at the hip via COM. We want the big languages and frameworks for building powerful libraries, for high performance and for when we need (or get real benefit from) strong, static typing. But we all want to use these lovely new dynamically-types languages like Ruby, Python or Groovy because they're concise, powerful and fun. Look what I can do in 5 lines of Ruby! It would take 50 lines of Java to do that! And just look at the clever metaprogramming stuff! Wait! These languages are great! Why don't we write everything this way!

Now think back to VB ... who remembers this book ? This was one of those landmark books you recall long past the time you stop consulting it. Part of the charm of this book was the writing of Bruce McKinney, who managed to combine a determination to wring the very most out of VB with an iconoclastic view of Microsoft, the very people who both developed VB and published his book! I liked the book, and I loved the writing. Plus, it really was hardcore: if you paid attention to the book and practised what it preached, you became a better (or at least, more capable) VB developer.

Now, I'm not trying to suggest that Ruby (or Python, or ... etc.) is/are no better than VB6, because that's clearly nonsense. But I am suggesting there are strong parallels between the two situations. With the dynamic, interpreted languages like Ruby and Python we have flexibility, speed (of development) and agility, and we can get something running without all the troublesome attention to detail required by C# or Java. Just as we liked the VARIANT type, we like dynamic (and 'duck') typing, and we like the syntactic economy of not having to use type names everywhere.

So we have a similar situation now to that which existed when VB was at its hardcore height: the evangelists and enthusiasts want to build absolutely everything in Ruby (or Groovy, or Python, or etc...), whether it makes sense to or not. The problem is the 'evangelists'. Or fundamentalists - you decide which term you prefer. Their enthusiasm for their favourite language unfortunately causes them to stray into deeper academic debates concerning type systems. They are so anxious to promote their chosen path, they want to convince the rest of the world that you really don't need static typing, even that static typing cannot save you from anything, so why bother with it? Do everything in <someDynamicLanguage>. Hardcore! I'm not going to wade in to the silly static typing versus unit testing squabble. The essential point is that serious issues are often hijacked, trivialised and misrepresented to serve as ammunition in some language or platform flame-war. Here is an excellent, sane post, as an antidote, discovered as a link in this piece by the equally excellent Tim Bray.

The analogy I'm trying to use is a little strained, I will admit: VB was a fundamentally weak language (see Bruce McKinney's departing missive), whereas Ruby is not. The difference between then and now is that there is no need for a 'Hardcore Ruby' book because you don't need to strain at the boundaries of the language to do powerful things. The similarity rests on the fact that with technologies like JRuby (and IronPython) we have the ability to use a variety of languages and approaches to attack the different layers in our solutions.

LinuxMint 5 (Elyssa)

2008-07-20T21:50:00.003+01:00
I've lost count of the number of Linux distros I've tried and rejected because of:
1) Ugliness (especially screen font-rendering)
2) Device problems, mainly wireless networking
3) Video driver problems, e.g. machine locks up when Compiz or similar enabled.
4) No support (or broken support) for power management, hibernation etc.
5) All of the above.

Even recent 'big-name' distros like OpenSuse and Fedora have had one or more of these problems, at least for me, on my Acer laptop.

Finally I believe I have found a Linux distro I believe I really, honestly may be able to use as a Windows replacement. It's LinuxMint 5.

I am very sensitive to (or intolerant of) poor font rendering on laptop displays. This has been the stumbling block for all previous distros, even those with support for anti-aliasing and font-hinting. Finally, the LinuxMint distro team appears to have sorted this out: the fonts render beautifully.

All the other issues in my list are addressed: the system picked up my wireless network immediately and connected as soon as I entered the pass-phrase. As far as I can tell, all devices appear to work, and the power-management stuff works.

Finally, for me the aesthetics are spot-on. The default theme and the choice of colours really works for me. This matters, when you're spending significant tim in front of the system.

This is a truly excellent distro - far and away the best I have ever used. This one will stay on my laptop and I plan to make the effort to use it instead of Windows.

NetBeans 6.5 M1

2008-07-17T16:36:00.003+01:00
I've just installed M1 and I'm disappointed to note that it appears not to do text anti-aliasing in the editor. Not only that, but the advanced options panel just doesn't have any editor settings in it at all. Strange - it might be something to do with migrating my old settings (from 6.1) which the installer does automatically.

I do hope this is just a temporary glitch - without the anti-aliasing, it's not a pretty sight...

Turns out other folk have spotted this: I just Googled for posts on this and found that a commenter on Tor Norbye's blog mentions this. Tor's replt to this comment suggests it's just a Vista issue, but I'm seeing this on XP Professional too.

Moving to Blogger

2008-07-09T11:38:00.000+01:00
I'm not sure it's worth running a blog engine myself, any more. With something like Blogger, free of charge and hosted by Google, it's hard to see why I should keep up with WP maintenance and have to think about upgrades, backups, MySQL and so on.

I've had my own blogger account for a while, but done relatively little with it until this week. I haven't ever been what you'd call a prolific blogger, but if I'm going to increase the amount / frequency of writing, blogger is almost certainly more convenient.

So, please see my Blogger page from now on. This blog/feed will almost certainly disappear in the coming months.

Bletchley Park - we must save it.

2008-07-09T08:18:00.004+01:00
Unbelievable though it may be, Bletchley Park is under threat, simply because it doesn't have a reliable source of funding. Apparently, it has been deemed ineligible for funding by the National Lottery, and turned down by the Bill & Melinda Gates Foundation.

This is an outrageous, shocking situation. Can we really be guilty of placing such a low value on maintaining this important site? If you know the history behind Bletchley Park and the Enigma code, then you will understand why I feel so strongly about this. If you are not familiar with this amazing story, read around the subject a little and you will soon appreciate the magnitude of the achievement at Bletchley and its importance to this country.

Please do sign the e-petition. Go to this page and follow the instructions.

Schema Tools

2008-07-08T13:57:00.004+01:00
There are already lots of commercial tools out there for editing and managing XSDs but I wanted to create my own. I don't want to pay for a commercial tool, and I'll learn a lot in the process. The goals for the toolset are quite modest:

1) Given a schema and a root element, generate the element/attribute tree for that infoset.

When you are handed a new schema, it is usually helpful to be able to see what an instance document should look like. Of course, a single schema can validate potentially many different infosets, so the client must be capable of generating the infoset for any given root element.

2) Automatically generate test instance documents.

This is a natural extension of (1). In any application involving the processing of schema-constrained XML, you need to be able to test against valid and invalid instance documents, and to achieve full test coverage for any non-trivial schema you will need a lot of instance documents. This is possibly the most useful application for this project.

Schema types mostly correspond with domain types (i.e. ignoring metadata) and you ideally want to be able to use representative test-data. For instance, if the domain includes people's names, you would prefer to use a pool of semi-realistic family-name/given-name lists, instead of random sequences of characters. So the tool will also allow you to create lists of values and associate them with schema components, using data from the appropriate list when generating test instances.

3) Schema-driven UI generation and XForms

It follows from the previous two items that you will want to look at the XML instance data, not just when testing but also in the application. It's possible to look at the XML text, but it's much nicer to be able to create some sort of form or dialog which presents the data in a more natural way, e.g. as a set of label/text pairs. This part of the tool is intended to help with that; guided by the XSD and some input from the user, the tool will generate a UI for the instance document. I haven't decided which UI to target yet, but it's likely to be Swing/rich-client to start with. Eventually, I will want to generate web forms and XForms.

I've managed to implement (1) already. I'm not sure it covers every case yet, but it does work. The code is messy and the presentation is currently a tiny Swing application. My goal for the UI is to create a Netbeans module, and integrate it into the Netbeans IDE alongside the existing (excellent) schema support. If I can integrate my commands into the NB context menu, the ideal is to trigger my functionality from inside the standard NB schema views.

I have a few more ideas for exploiting this code, but those listed above are a good start I think. Once I have working code I'm prepared to release, I will post something online for people to play with (and/or tear apart).

Netbeans 6 - it's getting better all the time

2008-07-07T22:29:00.010+01:00
It's great to see the Netbeans QA process working so well. I reported two issues recently, and both have been fixed. One was a Subversion related issue; within 24 hours I was contacted by the developer assigned to the issue, and offered a patched jar to try! It fixed the problem, and the patch will be rolled-up in v6.5. Impressive.

Netbeans is getting better, faster, than anything else out there, as far as I can see. There are some things I still don't think are good enough yet (such as the UML support) and I wish the whole thing would start up much more quickly, but it's important to recognize just how good this tool already is. And it's a free, small download. The comparison with Visual Studio is almost irresistable; VS is a DVD's worth of code, costs a fortune and offers a much less capable code editor, less refactoring support and doesn't really support rich client development to the extent NB6 does.

Look the the Ruby support in NB6, too: although there is a lot of interesting work going on with dynamic languages at Microsoft, the impression you get is that these are somewhat 'second-class' projects with no real presence in the main-line Visual Studio product plans. JRuby, on the other hand, is almost front-and-centre in the Netbeans world. The integration of languages via the JVM, and the development of integrated tooling in Netbeans makes it possible to do serious work with Java and Ruby, right now.

Another thing: Sun isn't trying to shut-down or marginalise any of the community, open-source projects which populate the Java tools landscape. Instead, they've recognized and accepted the strongest members of the community and built tool support for them in Netbeans. Just take the most obvious examples: Ant, JUnit and Maven. Compare that with what Microsoft has done: ignored NDoc and produced Sandcastle, created MSBuild to replace NAnt, and they want you to use (therefore buy) their own version-control system rather than embrace Subversion or Mercurial. I'm sure Team System is probably fine, as long as you have deep pockets and you are prepared to submit totally to Microsoft's prescription for your development team processes.

PSL not DSL

2008-04-22T11:10:00.000+01:00
I've long felt that the DSL moniker is a little inappropriate for some (perhaps most) applications of the ideas behind it. The majority of 'DSLs' are really just little languages which help solve specific problems in the software development space, e.g. Rake, which provides a very nice language for expressing graphs of dependent tasks. Martin Fowler has written quite a lot on the subject of DSLs - this paper is a good example of a little language being described as a DSL.

So it was good to hear Jim Weirich (the creator of Rake) make this very point while talking about the DSL hype in this InfoQ interview. The relevant bit is something like 14 minutes in. I like the term Problem Specific Language (PSL), which Jim invents here.

I'm working on applying Ruby to modelling (and reasoning about) a real domain, and driving a tool-chain for forward-engineering artifacts from the Ruby description. Apart from Rake, I'm also looking at RSpec, which uses Ruby syntax to capture descriptions of system behaviour. I'm very keen to work on the 'real DSL' problem, looking for the sweet-spot where domain-driven design and programming overlap. This whole area seems very fruitful for Ruby, because the language allows us to create very rich and expressive scripts which not only read quite naturally, but can also control complex processes.

Wubi and Xubuntu

2008-04-17T22:05:00.000+01:00
After the last foray into the world of Linux I swore I wouldn't bother for another year or so. Somehow I came across a link to Wubi and decided it sounded too interesting to ignore.

Live CDs are not really practical for doing anything much more than a cursory look and installing to a partition is too much work. Wubi is a very clever 'third way', installing a distro as if it is a Windows application, yet actually allowing the OS to start from the Windows boot screen, at full speed. Clever stuff. It uses a virtual disk (like VMware), which the Wubi creators admit will make file I/O slightly slower.

Wubi is set up to work with Ubuntu Linux, which most people will experience via Gnome and the rather, er, brown theme. Ubuntu is also available with KDE but there's a third alternative, Xubuntu, which uses the xfce window manager. I like this even more because it's lighter/faster than either of the others, and offers just enough functionality, without getting in my way. I simply don't need (or want) a lot of silly 3D desktop effects, nor a hundred different ways of playing media files.

I simply ran the installer and selected Xubuntu. Some time later, it was ready. Reboot, select Xubuntu, and there it was. Amazing! Even more amazing was the fact that Xubuntu located all the laptop hardware including the sound and WLAN chipsets. As soon as I selected the network applet, it offered to connect me to my home WLAN. I was running and connected to the internet in minutes. No other installer/distro combination has got even close to being this good. Xubuntu is excellent - I have had no difficulty in installing the extras I need (Java JDK, OpenOffice and Netbeans, for example) via Synaptic.

There is a way to convert the Wubi install to something more permanent using LVPM. I will probably do this, but only when I've sorted out the partitioning of the drive - a chore I'm not looking forward to.

Tomorrow's code

2008-04-03T21:50:00.000+01:00
I've just finished reading this piece by Bill Thompson, on the BBC Technology site. After a potted history (taking in the dear-old BBC Micro, of course) he makes a good point about the way we no longer seem to be encouraging school students to learn to program: greater importance is attached to teaching them to use office suites and accessing the internet than to enquiring into how all this software got written, or why it isn't more reliable.

This is just one symptom of a larger problem facing science and engineering generally. Physics is under threat, engineering (so we hear) is less popular than ever, and it often feels as if greater value is placed on producing 'entrepreneurs' and managers than engineers. I think we're running the risk of forever losing our position as world-leading engineers and innovators; perhaps that position is already lost?

I want to see people excited and enthused by the truly great scientific challenges facing us. Our future national prosperity depends on us being smarter than the competition: we cannot grow bigger, we cannot be much more populous, we cannot rely on mineral or oil wealth, and surely we are beginning to appreciate just how precarious our position can become when we depend too much upon financial markets whose behaviours are globally linked, not wholly predictable and not under our control.

We seem to spend a lot of time looking back wistfully at past triumphs (like the invention of Radar, cracking Enigma and the birth of computation, discovering DNA), rather than looking forward and preparing to celebrate the next ones. Are we talking ourselves out of a great future, collectively mentally preparing ourselves to accept that our greatest achievements lie behind us? Do we really want to turn our country into a kind of museum whose dusty artifacts catalogue a brilliant past?

Hate Vista, love C#

2008-03-07T22:37:00.000Z
Came across this blog post, while following-up on a semantic-web related Google Code project:

Why I hate Windows Vista (and can’t wait to re-install XP) « The Wandering Glitch 2

I have often wondered: how can the company which brought us C#, the .NET Framework and the CLR possibly be the same company which excreted Vista?

The guy behind the above referenced post is clearly a fan of C# (as I am), but despises Vista (as I do, along with those who made all the colourful comments on his post). I have left Vista on my new laptop, but in a much reduced partition, and have given over the lion's share of the drive to my copy of XP Professional, which I still think is pretty damn good.

What about the competition? On the desktop, there isn't much that's credible. (I'm not considering MacOS because to get that you have to buy the hardware). So I spent (or should that read wasted?) a few hours with some of the latest crop of Linux distros last weekend: Fedora 8, openSUSE 10.3, the latest Mandriva, PCLinuxOS 2007 and probably another one but I can't be bothered to recall which. None of them impressed me in the least. Only ONE of them (PCLinuxOS) correctly detected all of the important devices on this laptop, including the wireless chipset. Plus they all looked terrible when compared to Windows XP, especially in terms of font rendering, so they couldn't even seduce me with glamorous graphics.

What really drives me mad is that none of these distros really stands out: they all look virtually the same (Gnome or KDE, plus or minus a colour-scheme and some desktop wallpaper), they all contain more or less the same rag-tag collection of packages, but (and here's the kicker) they're all different in irritatingly detailed respects, some of which are downright inconvenient such as package management or filesystem layout! Every year is heralded as 'the year of Linux on the desktop', and every year I dutifully have another look, then gratefully boot back into Windows XP and get on with business.

I wonder what's next for Windows? With Microsoft so chock full of truly bright and talented folk, I'm really hoping that we can look forward to the Windows that Vista should have been. And when they do finally release that, why not give away XP? Or, to avoid the inevitable complaints about anti-competitive pricing, make it 50 bucks? Then nobody except the bigots or fundamentalists will need to waste time with Linux. Go on Microsoft, do us all a favour.

Solaris Developer Express 01/08

2008-02-28T17:34:00.000Z
The latest release DVD dropped through the letterbox last week, and I eventually found time to try it out, hoping very much that it would fix the issues I mentioned in the previous post.

What a disappointment. First, although there is an option to upgrade the existing installation I decided not to take the small risk that this might not replace all drivers etc., so I went for a fresh install over the top of the old one. This went OK, right up to the point where I removed the DVD and rebooted...

After the initial startup, Solaris starts building a database of some sort. I think this is a one-off operation which it performs the first time a new installation is booted. I recalled seeing it before, so left it to run while I went to get a mug of tea. When I cam back, I found a dead laptop - no login prompt, no power light, nothing. It was then that I realised that I'd been running on battery power during the install and I hadn't flipped the power-supply wall switch to 'on'.

So what? Power switch to on, and start the laptop: it'll be fine. Wrong. The OS wouldn't boot. I can't recall all the rubbish that scrolled past, but I'm pretty sure that the power finally ran out while this configuration database (whatever it is) was building, and without it you're stuffed. I wasn't impressed.

But I was determined enough to start at the beginning (again), this time with AC power on! And of course this time it installed perfectly. However, after logging-in I was disappointed to discover that none of the networking devices were recognized, nor was the sound and I presume also the graphics chipset (as before). This time I didn't even see the nwam dialog, so I have no idea how to make Solaris usable on this laptop.

I'm afraid that's it, for me. I won't be wasting any more time on Solaris, until I'm sure it will at least connect to the network. Windows XP is still my favourite day-to-day desktop OS, because it just works. I'm so glad I didn't remove it.

But what a pity: I had high hopes that Solaris might offer what no Linux distro (so far) has been able to: a good, professional alternative to Windows, created and supported by a trusted company like Sun.

Solaris Developer Express on Acer Laptop

2008-02-19T14:10:00.000Z
I've been keen to try Sun's Solaris Developer Express (SXDE) for a while now. They'll ship you the DVD free-of-charge, so there's no reason not to give it a spin. I'm waiting to get 1/08 (this year's first drop), so I installed 9/07. You can read a short review of the 9/07 release here.

I made a partition for it on my Acer TravelMate 5720 (using Acronis Disk Director, which I can recommend), rebooted with the DVD in the drive, and followed my nose. It was very straightforward, and once into the main part of the install process I was able to leave it chuntering away while I got on with other things.

Eventually, a reboot (without the DVD in the drive of course) and the first pleasant surprise was that the Grub bootloader had correctly detected the other two OS on this machine (Vista and XP), and I was able to boot into all three without a problem. Full marks for that (and a sigh of relief). However, once into SXDE I discovered a few annoying issues: hardware support (predictably enough) plus strange behaviour from the wireless network software. Below is what I found, plus some pointers which I hope may help others in the same situation.

By default, SXDE uses something called Network AutoMagic (nwam) which supposedly detects wireless networks and gives you the opportunity to connect to them. Well, it certainly detected my wireless network, but stubbornly refused to connect me to it. It prompted (correctly) for the wireless network ID (not broadcast) and the WPA password, appeared to accept both, but didn't connect or give me an error message. Every few minutes, the nwam dialog popped up again, but repeated attempts made no difference. The documentation for nwam is quite poor and not helped by the fact that some URLs in Sun's online documentation seem to point to the wrong place. The OpenSolaris project pages were probably the most helpful. I tried stopping/starting the service and playing with the parameters, but this issue remains unresolved, and is very annoying.

Worse still, the NetLink BCM5787M Gigabit Ethernet PCI Express device (i.e. fixed-wire ethernet) also didn't appear to work, which meant no connectivity whatsoever! Not a great position to be in. But, there is hope - keep reading.

The other hardware issues were less irritating: the Texas Instruments 5-in-1 multimedia card reader has no Solaris driver (I don't really care), the audio controller (Intel 82801H) chipset is supposed to have a bundled driver but it doesn't work, and the Intel Mobile GM965/GL960 Integrated Graphics Controller is also supposed to have a bundled driver but isn't reported.

For anyone else doing this, I strongly suggest running Sun's excellent Device Detection Tool, a free (web-start) download which will give you a detailed report of what hardware you have, and whether there is a Solaris / SXDE driver for it. This gave me hints on where to look for the missing drivers, and told me exactly what hardware I actually have in this laptop - very useful.

This tool pointed me directly to the page on the Broadcom site where I can download the Ethernet drivers for my hardware. Pretty good, but when you get there, you discover the following message: "Note: Broadcom does not offer UnixWare, SCO and Solaris drivers
for NetLink Ethernet controllers." Not good. But it may be I can use the Linux driver. When I get time, I will return to this and post my experiences here.

Lastly, a note on aesthetics. Solaris uses Gnome by default, which is OK rather than outstanding in any way. But colours and fit/finish on the desktop are very good indeed: in my opinion, Solaris looks great in every way, except for font rendering. There is a font-smoothing facility (in preferences) but this just doesn't approach the quality of Microsoft's ClearType implementation in Windows. For me, font rendering is a big deal - I hate to look at ugly fonts or blurry, smeary characters. I suppose this may be because Solaris is using a down-level video driver, because it doesn't support the Intel Mobile chipset on this laptop. Get the font rendering right, and I'd be happy to sit in front of Solaris all day.

As soon as I have a network connection, I'll continue evaluating Solaris and post my findings here.

Java Futures - and Software Processeses (sic)

2008-02-17T11:59:00.000Z
If you're looking for a good summary of what's coming up in Java FX, the Java SE Update (the 'consumer JRE') or JDK 7, then this talk given by Chet Haase is a must-see. It's quite long, but it's worth taking the extra time because this is good, solid content. And it's so good to get all this material presented by a first-class technical presenter rather than having to sit through shallow Powerpoint 'fluff' from marketing.

After watching this I located Chet's blog, and found his hugely entertaining send-up of our industry preoccupation with methodology. I think we have a new candidate taxonomy of the software process landscape! Conference Driven Development reminded me of some Microsoft technologies I recall hearing about - people were building plans on top of some of this stuff before we'd even left the PDC...

I don't have time to play with Update N or to try Java FX Script just now. FX Script interests me mainly because of the nice binding environment it promises, but JRuby will probably get my attention before FX mainly because I can put it to use on real problems more quickly.

Oracle 10g XE, SQL Developer and Java

2008-02-09T20:27:00.000Z
I've been forced to install and use Oracle 10g XE recently. The experience hasn't been pleasant. I'm sure the database technology itself is perfectly good, but the tools and support are dreadful, when compared to Microsoft's SQL Server Express Edition or MySQL. I'm writing this while still cross, so it's not going to be very nice about Oracle or Java.

I need XE for several reasons, one of which is that I have a dump file containing a complete database which I need to work with. Oracle XE comes with a (rather lame) web-based admin module which doesn't even seem to have tools for importing database dump (.dmp) files. To do this you need the command line imp tool. This would have been OK if it had worked (it didn't), but even then I hate having to find the right command-line incantations for things I only occasionally have to do.

Why didn't imp work? I'm not sure yet because right now I'm so pissed-off I can't be bothered to scroll through the screen-fulls of error messages it generated. (By default it doesn't even write these to a log file either - you have to tell it to do that. Duh.) This is just flat wrong. If there are version issues, or permission issues or similar, I should get a simple message, early, to tell me this, and the import process should stop right away, not struggle hopelessly on, scrolling pages of crap at me.

The online documentation isn't very good either. I admit I haven't burrowed through every page but I shouldn't need to do that, just to import a dmp file and manage an additional database.

So, I looked for a management tool (like the tools you get with MySQL, SQL Server and PostgreSQL), and yes, there is one. The Oracle SQL Developer is a Java application which you can download with or without the JRE. Well I've got Java 1.5 and 1.6 on my machine (both JRE and JDK), so I figured I could take the smaller download. You get a zip which you unpack somewhere, and run the top-level exe.

So, I ran the EXE. And what's the first thing I see? This:

So, unlike many other Java applications, this one can't locate any of the (four) Java installations on my machine. Poor, but not disastrous I suppose, so I browsed to the Java 1.6 JDK and clicked OK. I expected the tool to start, but instead I got this:

What?! Perhaps it doesn't like 1.6 - perhaps I'll try 1.5. So I ran the EXE again, expecting to be prompted to browse to java.exe again. But no! Instead, I immediately got the second error dialog again! The act of browsing to some java.exe seems to write a setting somewhere, which is used in subsequent launches, even if it's wrong.

It's hard to believe that one of the largest software companies on the planet can offer this kind of low-rent, sub-shareware experience - especially on an entry-level product surely intended to attract new users of Oracle technology.

I'm just not going to waste any more of my time on this. It's more than enough to make me uninstall Oracle 10g XE and give up trying to use it. I've used MySQL and Microsoft SQL Server Express happily in the past, so I'll see if I can import or convert the dmp file and use one or other of those instead.

BPMN - XPDL - BPEL

2008-01-29T22:18:00.000Z
After writing the last piece it struck me that I had unintentionally suggested that BPEL and XPDL are somehow competing for the same role, which just isn't right. This post is an excellent summary of the true position - roughly, XPDL for process description, BPEL for programming.

So what I think I'm most interested in is using a combination of these two, treating the design of a workflow-driven solution as a layered artifact. The XPDL is relatively technology-neutral: the description is an abstraction which can be shared across an enterprise (or even between enterprises) because it doesn't commit to implementation details.

The XPDL layer would then be used to drive the BPEL layer, where processes and activities which appear as abstractions at the XPDL layer are translated into concrete actions (and transactions) against real systems or service endpoints.

When I get some time (!) I plan to experiment with this combination, to see how practical it is for modelling an existing healthcare workflow I've been involved in.

VS 2008 - Finally

2008-01-19T23:53:00.000Z
Installing .NET 3.5 from scratch was the answer. This took two attempts; the first failed because the installer told me that 3.5 was already installed! The only things I could find which suggested 3.x was the XPS Viewer, which appears under .NET Framework 3.0 in the 'Windows Features' part of the programs control panel applet. Anyway, with 3.5 installed, the VC# 2008 installer finally ran to completion and VC# appears to run fine.

Why did this have to take so long and give me so much grief? Why doesn't Vista display the .NET framework in the add/remove programs panel? Googling about this I discover that Microsoft wants us to consider the framework as part of the OS, and not something which can be installed or removed. This seems ridiculous, given that you can download and run the installers for the various frameworks and it can still be important to know which version(s) are installed.

This is another one of those stupid, stupid, unnecessary things which will drive some folk away from Windows and into the arms of Apple (or even the Linux community).

VS2008 Installer - part 2

2008-01-19T20:48:00.000Z
Well, the network install fared no better. This is very frustrating and also puzzling, because this is a new Vista install (albeit an OEM install on a new laptop) and as far as I can tell there are none of the VS2008 or even .NET Framework 3.5 components already installed: I've suffered in the past when trying out early-access releases, when stuff isn't properly uninstalled, but in this case there's nothing to uninstall.

So, next step is to start with the .NET framework alone, and work up from there.

Visual Studio Express 2008 Installer

2008-01-18T15:47:00.000Z
Bit disappointed with the latest VS Express 2008 installer - I've download the 'kitchen sink' ISO image (containing all the VS Express editions), but when I try to install VC#, I get these errors:

and this one:

Because this is a new laptop (therefore a fresh Vista install) I did wonder whether this was a Windows Installer service issue, but the service is there and appears to be the latest version. I'm puzzled, and more than slightly irritated. It must be something simple - I'll no doubt kick myself when I discover what it is. Meantime, I'll try the network install.

FEBE for Firefox

2008-01-16T21:31:00.000Z
I recently got a shiny new laptop, which is great, but usually this means installing and configuring a lot of stuff before it's really a comfortable place to be. I'm sure this is familiar to most folk. I expected one of the biggest challenges to be Firefox, as I have a number of plugins, a lot of passwords and the usual bookmarks and preferences.

Luckily, I found FEBE, a Firefox addin which effectively solves this problem. The addon can be found on the Mozilla site here, but the author's own site has the latest version - there seems to be quite a big version gap.

To take all of my settings to the new machine I used the 'Full Profile' option in FEBE which creates a single file containing absolutely everything: I did try experimenting with a subset but it wasn't successful. If you're not familiar with Firefox profiles I strongly suggest reading this section of the FAQ on the author's site and follow the instructions.

Ruby-on-Rants

2008-01-14T20:46:00.000Z
I expect plenty of other people will have enjoyed (endured?) this piece; I'm posting the link while I'm simultaneously laughing and giving thanks that I'm not him. This is serious, hard-core bitterness and resentment. This man needs help.

But the thing is, somewhere in that mountain of bile there are some grains of truth: in particular, some of his comments about consultancies ('The Hysteria of Consultancy' about half way down), and the kind of people encountered from time to time:

They then
build the team and dole out the positions: Software Architect to the shithead
who can’t code; Project Manager to the MBA asshole who talks like a car
salesman ... <snip>

Yep, I've met those two.

From C# to Java - syntax, libraries, generics...

2007-12-10T15:20:00.000Z
I was disappointed to discover that the standard Java regular expression library (java.util.regex) doesn't support named captures. I'm porting some C# code over to Java, which uses this feature. The best alternative seems to be JRegex, which forces me to depend on a non-standard library.

A porting exercise like this really highlights the differences between Java and C#. I must confess that I prefer C#'s syntax and the slightly cleaner code resulting from the (mostly) very well designed and implemented set of framework libraries. The C# designers of course had two great advantages: (1) being second, and (2) not having to worry over-much about portability.

Advantage (1) is about having a 'clean slate': they could take Microsoft's Java (J++), fix (or improve) awkwardnesses, and clean up the syntax without worrying about compatibility with existing code. Property syntax is a good example; when porting over my C# code, it was a pain having to convert every property to a getXXX() / setXXX(val) method pair. On the other hand, the Java bean convention has a modest advantage when using intellisense or similar in an editor: all the get and set methods appear in a nice list - you don't have to hunt for the properties.

Advantage (2) may be more debatable. Although it's true that the Java platform APIs are careful to avoid OS-dependent behaviour, it's sometimes necessary to be aware of potential differences in the platform beneath the VM: e.g. on a Unix-like system, a file may return false for both File::isDirectory() and File::isFile() (e.g. a block-device). To be fair to .NET, the core libraries provide good support wherever they touch a common native resource such as the filesystem.

Two simple examples where the .NET framework libraries provide an out-of-the-box solution, but Java still depends on third-party support:

1) Intelligently combine path fragments to yield a single, usable pathname, using the correct separator character:
// C#
string filePath = Path.Combine(pathFrag1, pathFrag2); // fragments are strings

// Java
import org.apache.commons.io.FilenameUtils; // Requires Apache Commons IO library
...
String filePath = FilenameUtils.concat(pathFrag1, pathFrag2);

2) Read the contents of a file into a string (and ensure the file is closed):
// C#
string fileContents = File.ReadAllText(filePath);

// Java
import org.apache.commons.io.FileUtils; // Requires Apache Commons IO library
...
String fileContents = FileUtils.readFileToString(new File(filePath));

Hardly a big deal in either case, but in C# you never have to hunt for the external library and ensure it's linked. So much of the .NET BCL becomes practically 'extended syntax': if only Visual Studio implemented the equivalent of 'fix imports' in Netbeans! I do get tired of having to go to the top of the file and add the 'using ...' lines.

Generics are another matter (and much too large a subject to deal with properly here). It is a simple truth that .NET does generics properly and Java does not. If you don't like that statement, read both this article and this one, before giving me any grief about it.

Here is the same combination of generic collection and property getter, in both languages:
// C#
//
List<Segment> m_segments = new List<Segment>();
...
public Segment[] Segments
{
get { return m_segments.ToArray(); }
}

// Java
//
ArrayList<Segment> m_segments = new ArrayList<Segment>();
...
public Segment[] getSegments()
{
return m_segments.toArray(new Segment[m_segments.size()]);
}

Again, not a huge difference, but I know which I prefer to look at. The Java toArray implementation requires us to provide the new Segment array as an argument because at runtime Java's ArrayList doesn't possess the generic type information to make the copy without the type-hint. The copying is done using System.arraycopy and casts, because ArrayList's type parameter information is erased and replaced with Object (and casts) at compile-time.

Despite all of that, I'm keen to continue with both languages. If only the Java community would be prepared to go for a 'breaking change', a major step release which fixed some of the fundamental problems (e.g. generics implementation) at the expense of JVM back-compatibility, and cleaned up the syntax even further.

EHR: Workflow First

2007-11-13T16:38:00.000Z
This short piece on the Health Data Management site resonated quite deeply with me: we discovered this truth the hard way, during the implementation of a large-scale radiology project here in the UK. A lot of time was spent on technology (RIS, PACS, PAS, HL7 messaging, DICOM integration, etc.) early on, and a lot of assumptions were made about workflow and the implementation of the operational side of the solution.

WS-*, REST and security

2007-11-13T14:57:00.000Z
Via Don Box's recent post and a comment in Sam Ruby's reply, I found this presentation: worth reading. The latest version (together with a lot of other stuff the same folk have presented) can be found here.

The main message surely is to make the first gate as secure as you can: SSL + certificates. After that, if you need/want additional security then I think I agree with Don that uniformity, at least at the authentication level, is desirable. I just don't see where vested interests or additional value accrue from doing otherwise.

Migrating NetBeans Settings

2007-10-24T11:28:00.000+01:00
I've been using both the Beta and nightly-build releases of NetBeans 6 for the last few weeks. Every time I move to the next version, I have to sort out settings, preferences and libraries because the IDE doesn't offer to do this for me. Other folk have a similar complaint, e.g.
Migrating Netbeans class libraries between versions - TEERA 2.0
NetBeans preferences are stored in your profile, which in Windows is a folder structure rooted at: C:\Documents and Settings\<yourUserName>\.netbeans\<versionNumber>

The problem with this folder structure is that it mixes IDE settings with things like the local library collection and worst of all, a local class-information cache and file history (under the var folder). The whole thing can grow to quite a size (mine's well over 80MB).

The key subfolder appears to be config. I managed to migrate all the most important (and time-consuming) settings by copying these folders from one version to the next:

Editors

Preferences

org-netbeans-api-project-libraries

This doesn't quite do everything – docking window positions and toolbar button sizes are obviously stored somewhere else, but I can live with this.

I've voted for this to be fixed: see http://www.netbeans.org/issues/show_bug.cgi?id=42157

Ronnie Hazlehurst

2007-10-09T17:51:00.000+01:00
So sad to hear that Ronnie Hazlehurst died on 1 October. There have been plenty of well-written obituaries (e.g. the BBC, the Guardian, and Telegraph), but I wanted to write a few lines myself because for me, like most people of my generation, his music formed an essential part of the backdrop of my early life. I still find the tunes he wrote for all those wonderful BBC sit-coms hugely evocative: the instant I hear the opening few notes of "The Fall and Rise of Reginald Perrin" I'm transported back...

I hadn't fully appreciated the sheer brilliance of the man until I read some of the obits: for example, he used the morse-code letters for "Some Mothers do 'Ave 'Em" to dictate the rhythm, and then scored the piece for two piccolos! Every piece seemed to fit perfectly the character of the programme for which it was written. Some might say that we will always make that judgement in retrospect, because there never was any other theme tune association. Well, perhaps, but just listen to "The Two Ronnies", "Fall and Rise", and especially "Yes, Minister": these are works of art - you just couldn't improve on them. Those tunes will be forever special to all of us who grew up with them.

Dare Obasanjo on the Release of the Source Code of the .NET Framework Libraries

2007-10-09T16:44:00.000+01:00
Link: Dare Obasanjo aka Carnage4Life - On the Release of the Source Code of the .NET Framework Libraries

As usual, someone else has already written it: Dare's piece on this announcement reflects my views exactly. Since I've been playing with Java lately (see previous post) I've become used to the idea that I can jump straight into the source for almost anything, certainly for the JDK libs.

Still, most of us I'm sure would agree that this is a Good Thing, both in purely practical terms for working programmers, and on another level, more evidence that Microsoft is beginning to 'get' some of the things the other side of the industry managed to 'get' long ago.

There's another way the developer will surely benefit: being able to read through the real code is the best way to appreciate and absorb the good design principles enshrined in (most of) the .NET Framework libraries. The principles which guided the team are described in the excellent Framework Design Guidelines book by Cwalina and Abrams, a book I'd recommend even to folk who don't use the Microsoft platform. True, you can use Reflector to reverse the libraries, but the original source will presumably retain comments, which may reveal subtleties around intent, choices and so on.

NetBeans 6.0

2007-10-09T11:51:00.000+01:00
After my last post on NB and Eclipse, you might think that I'd never go anywhere near NetBeans again. Well, predictably enough some things (chiefly HL7, Ruby and RDF) have dragged me back to NetBeans and Java, so I grabbed NB 6.0 Beta 1 and gave it another try.

Much, much better. Somehow, the startup time has been reduced quite a bit, everything felt faster and the whole tool is shaping up rather well. A complete contrast to my previous (and quite recent) experience. I played with the startup settings to improve performance even further (details somewhere below) and now I'm quite happy with it.

You do need to spend a bit of time with NB to appreciate just how good it actually is: the code editor features outshine Visual Studio quite easily - better refactoring support, better code navigation being the two I immediately appreciate. Simple example: want to go to a definition? Hold down CTRL and statement elements become hyperlinks. Adding libraries and references is as simple as in VS, and you can create project groups which are similar to VS solutions. I've barely scratched the surface.

The set of plugins in the default download of NB 6.0 provides a lot of functionality, not all of it really ready for daily use, in my opinion. The UML support appeared good until I tried to use it for a substantial reverse-engineering job: it took a long time and the resulting class diagrams were slow and awkward to render and navigate. Not really a priority for me, though.

Subversion support is also provided and this is definitely a priority for me. Sadly, this appears to be weak, too. First, NB appears unable to import new (unversioned) projects: the 'Import into Repository...' command seems to be permanently greyed-out. Oddly, the 'Commit...' command is available even though the project folder is completely unconnected to my SVN repository. If I invoke that, I get a partial list of new files in the grid, and the option to commit them; clicking the commit button apears to work, but after a while I see a popup dialog saying "Action canceled by user", even though I did nothing!

Ruby support is good, but I'm a novice Ruby developer and have yet to exercise all the Ruby features. The ever helpful Roman Trobl has provided some good Flash demos of Ruby support: I recommend watching the demo of NB's RoR support, where Roman builds a bare-bones blog application in a couple of minutes. I haven't found a better Ruby IDE yet.

I plan to put more information on using NetBeans on the Java section of my Wiki, especially for folk coming to Java and NetBeans from a Visual Studio background. So far I've only added a note on the configuration settings I've adopted which improve performance considerably - more soon.

How long can this last? Well, on the evidence of the last few days, I'm optimistic.

G.ho.st in the machine

2007-10-03T12:31:00.000+01:00
G.ho.st in the machine : Blogs : BCS

Peter Murray blogs about healthcare informatics on the BCS site, but the post that I've linked to here is all about g.ho.st, an amazing piece of work all done in Flash (as far as I can see), offering a kind of VM accessible through the browser. Thanks to Peter's post, I've signed up. I also managed to get my preferred user-name, 'roger', so I guess there aren't too many users yet.

The Flash applet does all the work of course, much like an X Window System display server does when you drive it over a network (I can remember actually doing this! It's a fundamental feature of X: how many people still exploit it?).

With g.ho.st you get 3GB(!) of space and they claim you can ftp from Windows Explorer stright to your online store. I tried this but it didn't work for me: I tried twice to upload a couple of PDFs - Windows thought it had transferred them but they didn't show up in g.ho.st.

A bit of a toy, as it stands, but very impressive and worth keeping an eye on.

Facebook

2007-10-02T08:16:00.000+01:00
Sorry, but I just don't like Facebook. Actually, you can extend that dislike to all social networking sites, though to my eternal shame I must admit to being in at least one (LinkedIn).

I think this clip very nicely sends-up the whole Facebook thing. Too much free time ...

IONA Artix: Video Tech Brief (John Davies)

2007-09-27T17:08:00.000+01:00
IONA Artix: Video Tech Brief

I've been looking at the Java and SOA landscape again (I expect to write more on this) and came across this interview with John Davies of Iona. I'm pointing to this video (link is above this text) not because it's a particularly colourful performance (forgive me Mr. Davies) but because it's worth listening to what he has to say.

This is all about high-volume bank transactions which handle complex data structures (he talks mainly about Swift), exchanged and processed in XML messages. The architectures they are using for this are all based on Java ESB/SEDA platforms, on top of which Iona adds its Artix Data Services to handle metadata management and transformation services.

First, it's notable that they are building on established open-source projects - Iona has a very strong investment in open-source solutions - using Apache ActiveMQ as the basis of their FUSE enterprise messaging product.

That's all very interesting, in an industry-direction sort of a way, but it was something else (technical!) that really caught my eye (or ear): John Davies talked about the database bottleneck for these high-volume transactional systems: the messages being persisted are XML (hierarchical data) and they simply cannot accept the overhead of an ORM layer and mapping these to tables, so they're using a completely different approach, saving them as immutable BLOBs, indexing appropriately. New versions of the same message are simply stored as new objects, the original is not touched. Combine this with a massively parallel service layer and distributed store (he talks about running Gigaspaces on Azul - some 700-odd cores!) and you have a very interesting proposition.

It's in a follow-up comment on the ServerSide page where he expands a little on the problem of efficiently storing immutable, hierarchical objects and points to Subversion as one good way to accomplish this and handle versioning, where performance isn't an issue. But he also mentions ZFS, a new filesystem being developed by OpenSolaris - this offers a transactional, pooled-storage abstraction which is exactly what this sort of architecture needs.

This is fascinating stuff. Reading around this subject it's very clear that a lot of intellectual effort and investment has been poured into solving the problems of building and operating truly scalable ESB and SOA-based solutions and novel, high-performance persistence, and most (all?) of this amazing work has been done with Java, and is open-source.

As I worked my way through web pages and PDFs, I didn't find references to Microsoft's technologies - what, if anything, are they doing in this area?

Trying out Blogger

2007-09-26T08:25:00.000+01:00
This is obligatory 'first post' to see that it works. I don't know whether I'll post regularly here or stick to using WordPress on my own site.

Amazon Marketplace

2007-07-30T12:39:00.000+01:00
I've used Amazon (.co.uk) quite a bit. I've bought quite a range of stuff, from books and CDs right up to my latest digital camera. So far, I've been a very happy customer: prices have been keen, stuff has arrived quickly, been well packed and always exactly what I ordered.

So I'm feeling a bit bruised today, after discovering that the delivery charge for the two Compact Flash cards I've just ordered will be around 40% of the item price! About 9 pounds postage, on a twenty-two pound bill! These are small, light objects and they're being sent from inside the UK. Outrageous.

The Amazon webpage for the item is here.

Notice the headline price, and the fact that there is no indication of the delivery charge. Of course, I should have carefully read the subsequent pages before clicking on the 'confirm order' button, but I just didn't expect that I would need to check an Amazon order (even involving an Amazon Marketplace seller) for this sort of thing.

The marketplace seller is called _memorymegastore_ -- I'll wait to see how fast a delivery I get for my 9 pound, then leave them some feedback. I've emailed Amazon too, but don't expect much...

Caveat emptor...

NetBeans and Eclipse (again)

2007-07-25T10:14:00.000+01:00
Java technology is so frustrating: I'd like to get deeper into it mainly because of projects like Mirth, but the whole developer experience feels so excruciatingly awful when compared to Visual Studio and C#. I decided to give NetBeans another go because there's a new developer release (6.0) and I found the 5.5 release reasonably good. After waiting over two and a half minutes for the damn thing to start, I remembered just how ugly it was. Even with a bit of fiddling (make the icons smaller, change the editor font, etc. etc), it still looks and feels clunky, ugly and slow. Then I did the usual smoke test - create a new standard project, build it and run it, just to be sure all the bits are in the right place. Clunk, ... grind ... whirr ... splat. What's taking so long? This is a trivial project. Eventually, it did build and it did run. And it looked awful. The UI editor (frankly, one of the better Java UI editors out there) is still nowhere near as good as Visual Studio. NetBeans is free, it has a lot of features, it can create a lot of different project types, it has UML and BPEL built-in and it has Sun behind it, but it just doesn't encourage me to persevere with it.

What about Eclipse (and Europa)? Well, the initial install is quite quick but then you have to download the Europa packages. After about 30 minutes or so, I've got the whole enchilada (easier just to get everything rather than fret about the dependencies) and I can start the thing. The startup time is less than NetBeans but still rather slow compared to VS. Now to create the 'hello world' Swing or SWT application (I don't really care which toolkit). So, File, New Project. My, but there are a lot of project types. Go for Java (there is no finer-grained optionality here). Now, how do I add a dialog class? Can't see dialog class in File/New, so I choose File / New / Other. Gulp. There are 37 groups of project item! (I counted them). Thirty seven GROUPS! This is not use-case driven; this is madness. So I choose the Java category, and only 'Class' seems appropriate.

My patience has run out (again). Both tools, for different reasons, turn me right off Java development.

HL7 Standards and Message Mapping

2007-06-30T10:28:00.000+01:00
For most of the last year I've been working on healthcare integration projects, involved in HL7 and related technologies. One of the biggest frustrations I've encountered is the HL7 standard itself, access to it (and related information) and the way the standard is published.

The HL7 (v2) standard was published as a body of Word documents, apparently without any machine-readable version. You might have expected that the standard would have originated as a database plus supporting commentary but it seems that the Word documents have always been the definitive standard. You could of course write software to crawl over the documents and extract the standard but it looks to me as if this would be pretty painful; I haven't checked every document, but I'm not sure they are all consistently formatted.

Almost from the start I decide to write tools to help me with message profiles, mappings and transformations, but to do that effectively you really need to have the standard in a machine-readable form.

I did come across a German site which offered an Access DB version of the standard but this has now become an HL7.org product (I believe it was originally unaffiliated but legal pressure was brought to bear...)

The fact that HL7.org is a quasi-commercial entity irritates me: I don't mind special-interest bodies charging for 'value-added' things like books, papers and conferences, but making the standard itself proprietary and to restrict access to it just feels wrong. Standards like this should be open to all.

What prompted this post was discovering this effort to capture HL7 segments, fields and tables in an Excel spreadsheet. Pity that the link to the file doesn't seem to work: Matthew, if you read this, please check the link in your post.

Thank goodness v3 is XML based. Presumably the specification will be driven from XSDs. Reading the HL7 'statement of principles' (SOP) for v3 they appear to regard the v2 principle of starting from documents and deriving the technical artifacts as 'more direct', as 'one simply edits ... the appropriate word processing document'! Direct, certainly, but desirable? The SOP indicates that a tool-chain will be used to derive documentation from 'computerized models' which sounds better.

I was able to get access to the latest v3 standard draft, and part of this site contains the XSD but they're all linked separately, and as HTML! Why on Earth not provide a zip download of the whole thing?

Google Gears, Silverlight and the RIA

2007-06-12T07:31:00.000+01:00
I use Google Docs and Google Reader all the time. With Google Gears, these are set to become even more useful because they'll work when I'm offline. The Reader already works this way (see this post for more), but not Docs + Spreadsheets yet. Rumour has it that this is coming. If they solve the sync problem properly, this will be seriously good: Docs is fine for note-taking and certainly good enough for blog posting. And of course they must extend Gears to GMail.

Web 2.0, Ajax + Gears (+ whatever server-side stuff you use to generate all this) may be good enough technology for building browser-applications which work offline, but underneath the covers it's still based on HTML, grinding out a ton of hard-to-debug JavaScript, and running inside the browser frame. For the poor developer, even though dedicated folk spend hours creating libraries like script.aculo.us (and Google's own GWT ) the experience of creating these applications remains pretty dire. And the result of all this extraordinary effort is still something which doesn't even approach the sophistication of the equivalent desktop application. As I said, I really like Google Docs, but it isn't Word, and won't ever be.

Then there's Adobe Flash and ActionScript, which has been around for a long time, is well established and has been used to create quite complex browser-hosted applications, such as Gliffy, which is (a bit) like Visio in a browser frame. Flash is cross-platform, as well as cross-browser, but still there's quite a gap between developing for Flash, and developing for the desktop. Now Adobe Labs is promoting AIR (the technology formerly known as Apollo). This looks like a warmed-over Flash, and appears to be based still on JS and HTML.

When I first heard about Silverlight, I wondered why Microsoft appeared to be tilting at the Flash/ActionScript windmill? Why would they expect developers of browser-hosted, graphical presentations to switch away from Flash? But that's not really what Silverlight is about: this is about creating a platform for Rich Internet Applications (RIAs) based on the .NET platform, which makes this potentially much more interesting, especially if development can be done in C# using Visual Studio, rather than mashing together markup and script. The runtime appears to be a a slightly reduced version of the CLR, supporting WPF/XAML as well as the traditional HTML/JS mixture. Quite remarkable, in a 1.1MB download. Rather than repeat more technical details here, I'll refer you to the Silverlight architecture summary paper.

Silverlight already runs on Mac as well as Windows, and in all the popular browsers. Could it be ported to Linux/Unix? The Mono project has proved the portability of the runtime, so it doesn't sound like such an outrageous suggestion.

Google Docs for Blog Posting

2007-06-06T13:59:00.000+01:00
I'm finding Google Docs more and more useful for note-taking online, and quickly bashing together an outline document. I've just discovered that it will also post to weblog engines: this post was created in Google docs, the published straight to WordPress, via the MovableType API. This appears to work quite well, except that the document title doesn't appear to make it across to the blog posting (Update: yes it does - keep reading). You can tag the document in Google, and these tags are supposed to carry over into the blog posting.

I'll use this posting to experiment a bit. First, if I insert a picture into the document, will that be uploaded and linked in the post? I'll insert the image directly below this line:

Well, that worked! The secret is to choose the metaWeblog API instead of the MovableType API, despite the fact that Google recommends the latter: WordPress supports both, but the metaWeblog API appears to be better supported. I tried using the MovableType API initially; the posting title wasn't carried over, and tags didn't work. Both work with metaWeblog.

This really is excellent, especially because of the image embedding. Now all I want is for Google to add support for Google Gears to Google Docs and Spreadsheets, and I'll be very happy indeed.

XMI from Assemblies

2007-06-06T10:51:00.000+01:00
I was about to start writing a tool to do this when it occurred to me that someone else must have done this already. Googling located two solutions:

DOTNET2UML from AgileFactor, see this page.

Xmi4DotNet, see this page.

Of these, (1) seems to be out of date and doesn't handle .NET 2.0, so it's no good to me at all. Option (2) appeals the most because it's an addin for Reflector. Trouble is, the current version of Reflector won't load it! I haven't time to work out why, but looking in the Google code issue list for this project, someone has reported this error already. Would be nice to see this fixed. I'd quite like to write code to do the reflection + generation myself, but it's just a distraction I can't afford right now.

WordPress v2.2

2007-06-06T07:37:00.000+01:00
Upgrading successfully from 2.0.2 to 2.2 was not as straightforward as it should have been.

I followed all the instructions, disabling all the plugins and setting the theme back to the original default, but when I ran the wp-admin/upgrade.php script I got a lot of database errors. The blog worked, but the categories were missing and the admin pages which refer to them showed errors.

These seem to be due to database permissions issues. Googling for a solution I found a reference or two (e.g. this one), so I restored the old DB (what a good thing I did the backup!) and used phpMyAdmin to change the db-user permissions to include ALTER. This appeared to fix the problem because the upgrade script worked perfectly.

My main gripe is that despite the (tedious, manual) DB fix, most of my posts appeared to lose their category tags. Eventually, these appeared again, but I'm not sure exactly why.

The upgrading process is irritating. WordPress is such a fine piece of work overall, it seems such a shame that upgrading should have been be so clumsy and buggy.

Tate Modern

2007-04-15T17:10:00.000+01:00
We had a great family day out yesterday at the wonderful Tate Modern gallery in London. It's the last week of Carsten Höller's slide installation - great fun, whether or not you think it's art! Take a look at the website because there are plenty of photos (even videos) of the slides, including a time-lapse of their construction. The tickets were free, but you had to queue for them. We managed to get quite a few tickets, so we were able to enjoy all of the slides. The highest (from level 5) looked pretty daunting from the turbine-hall, but was actually rather a gentle experience and you can't really see out of the slide properly so you're not aware of the starting height.

Tate Modern is quite simply superb. I don't like all the exhibits (for example, there's a big Gilbert and George thing happening there right now, but I'm not a fan) but I love the building, the spaces within, the facilities and the atmosphere. And it's free. Places like Tate Modern are so important: it matters that we spend public money this way. The whole time we were there, we were happy, and all the other people we saw seemed to be, too. I felt uplifted in some way, not just because I had enjoyed myself there, but because I had shared that enjoyment with so many other people.

Jim Gray

2007-02-22T13:54:00.000Z
I can't believe that I missed the news that Jim Gray had been reported as lost-at-sea. This is just terrible. Jim Gray is one of the greatest minds around. Whenever I've heard Jim speak, or read one of his papers, I've been struck by the depth of his insight and the power of his quiet, engaging style.

The last time I saw and heard him in person was at PDC2003. He talked about high-performance computing (and distributed copies of this paper), discussed the building of very large servers (e.g. for the SkyServer project), and said something which stuck in my mind ever since. He was talking about the problems of processing very large volumes of scientific data and building petabyte-scale computing facilities (later expressed in this paper). He said (this is not a verbatim quote), "We've got to start thinking in terms of moving the program to the data, instead of the other way round, which is what we've been doing for years".

This is a profound idea, which is workable now that we have almost ubiquitous networking, grid-computing, and fast, cheap hardware (CPU and storage). Network bandwidth, though improving all the time, is not increasing fast enough to keep track with the increase in data we need to analyse.

Maybe Jim is still out there on the Farallon Islands, getting away from it all. I just wish he'd let us know he's OK.

Java Futures

2007-02-13T16:00:00.000Z
Have just come across two useful posts on the future of Java. Bruce Eckel's piece is mainly focussed on web applications. I agree wholeheartedly with his opinion that the web is "a mess", and that the technologies which have sprung up to help make the mess manageable (such as Google's Web Toolkit) are merely a convenient veneer to hide the ugliness and limitations of the underlying platform (i.e. JavaScript, HTML and the browser), which are always revealed in the end. The failure of Java applets, and in fact in any of the 'active content' technologies to solve the Rich Internet Application (RIA) problem, reflects the difficulties in solving distribution, installation and security issues.

I also enjoyed his comments about how Java was 'rushed out' to fill the gap, then subject to extensive refactoring. The analogy with agile methods is just about admissible, especially in connection with AWT and Applets, but Eckel also notes the vitality of the Java world, and the strong, positive effect of competition on the Java / C# landscape.

Much of the rest of his post is an enthusiastic account of how Flash effectively solves the RIA problem, albeit in a vendor-dependent way. We may not like de facto standards emerging from a single commercial source, but when the technology is good enough we will find ways to compromise our principles.

The other piece is an IBM DeveloperWorks article by Elliotte Harold. This is much more about the Java platform itself, and what we can expect to see happening to it in 2007. With the decision to open-source the JDK, Sun has created the possibility of forks in the Java roadmap, allowing experimenters to introduce language or platform features. This is a mixed blessing: the Java world is already a complicated place to be, and this will make it even more so. On the other hand, this will allow a lot of talented people outside Sun to work on things like language primitives for structured data types, such as tables (for SQL integration) and trees (infosets and XML). There is a perception that Java lags behind C# in this respect (e.g. LINQ in C#).

Open-sourcing Java is good news, up to a point, but I just wish Sun would work harder to pull together the language, platform and libraries story into a more consistent whole. There are simply too many ways to do something: for example, how many web-service frameworks are there? I can think of three or four without trying too hard. All different. (Axis 1 and 2, XFire, JAX-WS, XINS, ...)

There's no doubt that C# and .NET is a simpler, cleaner and more consistent place to be than the Java world (especially the 'enterprise' Java world), but then it damn well ought to be, because it only has to live on top of one OS, and isn't subject to any sort of community process to decide its future: only Redmond (and some Microsoft Research sites around the world, notably here in Cambridge) determine the trajectory of .NET and the C# language. (Don't buy the line that the whole shebang has been ported to lots of other platforms: take a look at any of them and show me where the enterprise features are. Yes, you can compile and run a C# program on Linux, but can you deploy and run a server-side C# component which depends on Enterprise Services, e.g. distributed transactions? No, you can't ).

Arguably, C# and .NET owe their very existence to Sun's Java initiative. This remains a largely unacknowledged technical debt, often ignored by Microsoft's supporters. For many of the features we take for granted, Java got there first. It really is a cross-platform proposition including on the server, albeit with some residual, ugly cruftiness which is being weeded out gradually.

Personally, I really like C# and the .NET platform, due to the internal consistency of the runtime and C#'s cleaner syntax. However,I really want to see Java develop and evolve, partly because I like it, but largely because we benefit in all the obvious ways from a competitive environment: if one or other platform put the other out of business, innovation and improvement would be subordinated to establishing and consolidating a monopoly position.

Formatting XML using Python

2007-01-26T09:49:00.000Z
Update: This article contains formatting errors, resulting from the WordPress editor I think. In future, I'll write articles in the wiki and simply refer to them from in here. This article appears on this page in the wiki, with better code markup.

The current work project involves a fair bit of work with XML Schema and instance documents which are validated against these schemas. The instance documents are generated by one application, destined for consumption by another; the resulting XML is, unsurprisingly, not formatted for reading by humans. As usual with XML, you end up having to look at it occasionally in Notepad (or EditPlus, my favourite Windows workhorse editor), and then of course you want to see the document structure nicely indented (which, by the way, means that the newest EditPlus release will automatically create folds for you - very nice).

So, you locate or create a nice little script to pretty-print the XML. Perhaps the most obvious way to do this is using the identity transform, in XSLT. But as I had been writing little Python scripts to generate and transform XML on this project, I decided to write a tiny Python function to 'tidy' XML files. I've attached a complete script to this post, but the lines which actually do the work are:

dom = minidom.parse(open(inFile))
dom.writexml(open(outFile, "w"), addindent=" ", newl='\n')

The writexml method from the minidom package is where the 'pretty printing' is actually happening. If you run this script against an XML file, it will appear to work - the resulting XML is indented and formatted. However, there is a 'gotcha', which is the real point of this post.

If your XML is validated against a schema, and the schema contains an enumerated type, the nicely formatted instances of the enumerated type in the XML document are not schema-valid! Here's an example schema fragment:

<xs:simpleType name="MessageType">
<xs:restriction base="xs:string">
<xs:enumeration value="REF_INC"/>
<xs:enumeration value="REF_TRI"/>
<xs:enumeration value="REF_REJ"/>
<xs:enumeration value="REF_ACC"/>
</xs:restriction>
</xs:simpleType>

And here's a fragment of XML from a document instance:

<Header><MessageID>20070125152405435</MessageID><MessageType>REF_ACC</MessageType>
<MessageTypeVersion>0.5</MessageTypeVersion><Destinations>
<Destination>PRC</Destination></Destinations>
... etc.

Here's what writexml produced:

<Header>
<MessageID>
20070125152405435
</MessageID>
<MessageType>
REF_ACC
</MessageType>
<MessageTypeVersion>
0.5
</MessageTypeVersion>

Indented, and with newlines between elements. Unfortunately, newlines are also inserted into the text values. In this example, MessageType is no longer schema-valid: the whitespace is included in the value of this element. This is, of course, because the text values are sub-nodes of the MessageType element node. The documentation doesn't appear to offer much help, and experimenting with writexml arguments didn't result in anything better.

There's also a toprettyxml method:

prettydoc = dom.toprettyxml(indent = " ", newl = "\n")
fp = open(outFile, "w")
fp.write(prettydoc)

This is no better. In fact, I seem to end up with multiple, redundant newlines in the output. This is getting silly! All I want is nicely formatted XML, for goodness' sake! Have I missed something here? This is just not worth this much effort - the XSLT works fine.

Turns out I'm in good company - while Googling for enlightenment on the dom methods, I came across this post by Bruce Eckel. He went a good deal further than me and wrote what looks like a proper solution (though I admit I haven't checked).

There are two lessons here: (1) although I have a fondness for Python, and I do use it for scripting tasks, many corners of the library are frustratingly badly implemented and/or documented (especially the latter), and (2) you really need to understand the XML model when you're working with XML Schema. I recommend reading Eliot Rust Harold's book, Effective XML: see Item 10 (White Space Matters).

Virtualization - QEMU and VMWare

2006-10-06T13:07:00.000+01:00
Discovered QEMU the other day, and I've just given it a spin. Virtualization is all the go, these days: the VMWare product is now free for personal use (if I understand the license), and remains my preferred technology on Windows. But this QEMU thing is new to me, and appears to be open source, too.

To use QEMU on Windows, you need to go to this site and download the qemu-0.8.2-windows.zip file first. Create a folder for it, and unzip the contents. You can try it out right away, by running the qemu-win.bat script. What you will see (if, like me, it runs straight out of the box) is a new console window containing a very small command-line Linux. So far so good, but (of course) I really want to run a more functional distro.

I have the ISO image for the Live CD of the excellent PCLinuxOS distro, and you can edit the batch script to load from this image:

    qemu.exe -L . -m 256 -boot d -hda harddisk.img -cdrom pclinuxos-p93a-minime.iso

The "-boot d" ensures we boot from the CDROM device and the -cdrom argument is the name of the ISO file (which obviously needs to be in the same folder).   The hda argument specifies the file to be used as the virtual hard-drive: I created a new (larger) one with the qemu-img.exe tool, called harddisk.img, and that's what appears in the line above.

Well, I can report that this works! It's a bit slow, but it does work. To make it perform better, you are supposed to use the QEMU Accelerator as long as you're on an x86 machine. Unzip (or untar/gz) the archive, locate the kqemu.inf file, right-click and choose Install. Then, open up a cmd window and type "net start kqemu" (go here for documentation). Once the service is running, you add -kernel-kqemu to the argument list in the batch file.

For me, this didn't work at all: I saw the boot screen and the PCLinuxOS progress bar, but it didn't get further than that. I suspect this is something to do with the fact I'm booting a live CD image and not a regular boot from the virtual drive. I'm going to persevere a bit, in spare time: more later if I get it to work.

Good Agile, Bad Agile

2006-10-03T21:21:00.000+01:00
Just been sent a link to Stevey's Blog Rants: Good Agile, Bad Agile: This is simply priceless polemic, well worth taking the time to read right through. The early paragraphs take apart Agile and XP, aka 'bad agile'. It's funny, but I'm convinced it's also grounded in truth. I especially like the Scientology analogy. Then he gets into describing 'good agile' which is, apparently, what life at Google is like. The prose turns from scathingly cynical (but entertaining) to slightly gushing and evangelical, as we're introduced to the over-the-top, out-of-this-worldness that is Google on the inside. It really does sound scarily ideal...

There's so much here which chimes with my own instincts (and prejudices). There are serious points being made, too, but I'm not even going to try to summarise them: you can't boil this stuff down into sound-bites or bullet-points. In a way, that's part of the message; our industry has wasted a lot of time and effort trying to do just that, and look where we've ended up.

Anyway, this guy really can write. Subscribed.

Google Reader Update

2006-10-03T08:25:00.000+01:00
The Google technologies just keep on improving: I've just started to use the new version of Google's feed reader, and I really think this is the browser-based reader which makes fat-client reader applications redundant (well, for me at least). I have been using RSS Owl, an Eclipse RCP application which I think is very good (and cross-platform, and free), but I really want to store my RSS feed subscriptions on the web rather than copy a file around, and my requirements for a reader are pretty basic.

How much further will I want to go with this? I already use GMail and the Google Calendar, which effectively makes Outlook redundant, and the Google Notebook for capturing material and scribbles quickly, and storing those on the web too. I'm not sure about the Writely word processor and the spreadsheet (whatever that's called) - when I last used Writely (months ago) it wasn't very impressive. But there is little doubt in my mind that for web-oriented applications, and for basic calendar/diary stuff, keeping things on the web and accessing them through a browser is almost ideal, provided I can rely on being 'always connected'. Broadband in the UK is becoming more and more like running water: only the remotest places don't have it.

The bigger question is whether I trust Google with all my data. How comfortable am I that Google can (in principle, if not in fact) read my emails, see my calendar, know what sites I monitor via RSS/Atom, and read my notes and documents? Currently, none of this bothers me in the least, for a number of reasons. First, there are other large companies which already know far more about me (and my lifestyle) than Google does: my bank and the supermarkets I use, for example. Second, I can choose what I store in any of the Google applications, and I could easily and cheaply encrypt anything I didn't want stored en clair. Lastly, I'm pretty sure that anyone looking at my Google data would have a hard time finding anything worth exploiting anyway: I'm not stupid enough to store anything truly valuable there. Every day, we all leave behind us trails of transactions, CCTV images, cellphone location updates, building entry/exit events, etc. These things are (for most of us) completely out of our control: is it paranoia, stupidity or vanity that makes people talk-up the 'danger' of online services?

Netron, RIP?

2006-10-02T11:16:00.000+01:00
The Netron project (Sourceforge) has disappeared. It looks like the author has sold out to (or been employed by) Northwoods Software. Good luck to him, I guess. I used Netron for a while, but found it frustratingly patchy and the author seemed more preoccupied with adding features and presenting a lot of demo apps rather than with quality. What's the alternative, in the .NET world?

R.W. Emerson Quote

2006-08-10T13:55:00.000+01:00
People are always putting quotes in blog posts, web pages or (heaven forbid) email signature lines. Normally, I ignore these but this one really stopped me in my tracks.

Catching up

2006-08-04T21:35:00.000+01:00
Another long gap between posts. So what's happening? Well, we went back to The Hague the other weekend to visit friends we made while living there.   It was great to have the chance to eat, drink and chat with them again: the children all went to school with each other for five years or so, and despite being apart for over a year they all got on as if they'd never been apart. On the Saturday eveing we got together at the excellent, friendly Lof der Zotheid, in Breitnerlaan. It's evening like this (and food/drink like this) that I do miss. The next morning, our hosts (Simon and Maureen) laid on breakfast for four families, so the party extended well into Sunday! Eventually, we had to make our way back to the Hoek, for the ferry. The journey's so easy, we ought to do this more.

Daniel had his 13th birthday in July! We managed to get him what he really wanted: a good, second-hand soprano Saxophone (a Yanagisawa!) from the ever-helpful folk at Wood, Wind and Reed. Daniel loves it. Good thing too, because it wasn't cheap! But it's worth every penny, as he will do it justice - he's already working on Grade 7 (on the Alto Sax) - and he's playing with great assurance. As a birthday party, we took half a dozen of his friends paint-balling at Go Ballistic in Thetford Forest. I must this is a very professionally run place: superb support for the whole day, everyone had fun and no-one got hurt.

Becca has just returned from a week's camping with her Guides patrol. They were somewhere near Coltishall, and it did rain for a couple of days but this doesn't seem to have dampened her enthusiasm; she's starting to plan a family camping holiday already! This week, she's been abseiling, climbing, kayaking, hiking, and on some enormous zip-wire, so coming home is a little bit of an anti-climax for her.

Work-wise, it's been a busy time, working on a healthcare-related project (more about that another time), leaving little time to get to grips with the personal development projects.   It also means I have plenty to write about my adventures in the world of HL7, DICOM, CDA and the rest of the healthcare informatics alphabet soup. So much to do, so little time...

I've been meaning to use a Wiki to keep notes on projects and as a cheap, fast way to publish stuff to the web, but I don't write often enough to make it worth investing too much time there. At work we use the Confluence publishing system which is pretty good: it's commercial, but there's a free individual license. I tried it, but the (memory) overhead of running Tomcat on my virtual server is a little more than I can accept for just a wiki.

DokuWiki is superb, and with the Monobook theme it looks and feels very much like Mediawiki. It's a PHP wiki engine, a much lower-overhead proposition and almost as functional. My hobby project now is to add a simple XMLRPC interface, which I intend to use as the basis for a page publishing and editing facility, via OpenOffice Writer (and possibly Word, too).   I'm going to use Python on the server-side; it's a superb language and very good for web development. I'll probably use Webware for Python to run the server-side code - it's a fast, simple and effective container.

Java and .NET - choices.

2006-06-01T13:24:00.000+01:00
I'm not usually drawn to online 'Java versus .NET' discussions. They tend to attract evangelists and bigots, and rarely lead to any insight. This article on TheServerSide is a little better than most; there is still a hefty proportion of noise, but enough signal gets through to make it worth reading a selection of the follow-up comments (I have only managed to get about half way through, skim-reading). Quite a few of the comments reinforce my own views, hence this post.

Software engineering is (or should be) about getting things done - creating and delivering something. You use the tools and techniques which work well for you, in your environment. In my view, Java and .NET are roughly equivalent in terms of their ability to enable delivery. You cannot guarantee success by choosing one technology over another, but you can improve your chances: you select the one which most closely fits your circumstances. This piece of common-sense at least emerged from some of the comments. If your business (therefore your customer's) is tightly bound to the Microsoft platform, then .NET probably makes good sense for you, regardless of engineering-purity arguments in favour of an alternative.

The breadth-of-choice issue, in relation to Java frameworks, libraries and technologies (and even IDEs), is an understandable point, but I see this is as one of Java's great strengths. The Microsoft monoculture means you don't have to think for yourself, and for many a jobbing programmer, this is a good thing. What's needed is strong technical leadership where decisions concerning the language, tools and technologies to be used by default are made by senior engineering, following some process of study and selection. With the burden of technology-selection drastically reduced (even removed), development teams can get on with applying the selected technology, gaining experience with it and feeding-back into an iterative improvement process managed by the technical leaders.

Some will complain that this smacks of a 'two-tier' technical hierarchy, with certain individuals occupying elevated positions and having control of process and technology. Well, yes, that is exactly what I am suggesting. I don't see a workable alternative that doesn't result in, at best, less effective teams constantly trying different things, and at worst, complete chaos. It doesn't mean the leaders are always right, and (as the hint at an iterative process indicates) it doesn't mean things can't change.

This article is timely because I have only recently started trying to use Java and Eclipse, after many years of C# and .NET. I can sympathise with the .NET enthusiasts because I am at the stage where I know how to do it using C#/CLR but I'm not quite sure of the best way to do it using Java. As it happens, I don't think the range of alternatives is daunting at all: it looks to me as if I can go a long way with Eclipse (with additional libraries), Spring and Hibernate. What continues to amaze (and delight) me is the range and quality of supporting libraries in the Eclipse space: there is no equivalent in the Microsoft world.

Eclipse, EMF/GEF/GMF

2006-05-30T11:15:00.000+01:00
The support for modelling and model-visualization in Eclipse is very impressive, but a little difficult to get to grips with quickly. I had already started working with the free IBM red-book on EMF and GEF when I discovered that there is now a project called Graphical Modeling Framework which appears to subsume EMF and GEF.

Moreover, I'm trying to use the Callisto build of Eclipse (which attempts to combine harmonized plugin and project versions), and the Omondo UML plugin no longer seems to work correctly when used to create EMF models graphically. I cannot seem to add an attribute to a model class in Omondo - the menu command simply doesn't exist. To be fair to Omondo, I think they recognize there are some issues and a forum comment has been posted to the effect that a new version will be released soon for Callisto.

What's so interesting to me is that the Eclipse/Java community appears to be ahead of Microsoft in the area of Domain Specific Languages (DSL) and model-driven development. Microsoft makes a lot of noise about 'Software Factories' and building support for creating and using DSLs in Visual Studio, but this has only recently become available and you must purchase the Team System edition to get it.   Team System requires an enormous investment, not just money but also commitment to Microsoft's view of how your team should work and collaborate. Eclipse is free, completely open, and cross-platform. I'm going to be looking in more detail at the libraries and tools for doing model-driven development, and building graphical editors.

Airkix!

2006-05-23T08:22:00.001+01:00
Rebecca (my daughter) and I finally got to use our Airkix vouchers last weekend! Chris bought me a voucher for my birthday, and Becca liked the idea so much that we got her a voucher for her birthday, so we all went to the Xscape Centre at Milton Keynes on Saturday to use them!

The Airkix website gives you a pretty good idea of what it's all about, but you really have to experience it; the sensation is predictable enough but I hadn't expected it to be so tricky to maintain a precise position. You need to find a stable body-shape, and then to make all adjustments slowly. And as you have only two 1-minute flights, you have to learn all this quite quickly.

Becca is very athletic and supple, so she found the right shape almost immediately. It took me (a much older, creakier body!) a while longer, but I got there. And we have a DVD of our session to prove it! Must try to get some of that on to this site.

The tunnel equipment is awesome - I have no idea how powerful the fans are, but air is drawn from a very wide base area up through a funnell-walled section into the chamber (which isn't very big), which increases the velocity of the airstream. I didn't watch the instruments but I believe the windspeed is in excess of 100mph. There's a fine degree of control available too: they reduce the windspeed for smaller, lighter flyers, or when someone accidentally ends up too high.

This was huge fun! I've always wanted to experience free-fall, but I'm a still a bit scared of parachuting. This is the only other way...

Airkix!

2006-05-23T08:22:00.000+01:00
Rebecca (my daughter) and I finally got to use our Airkix vouchers last weekend! Chris bought me a voucher for my birthday, and Becca liked the idea so much that we got her a voucher for her birthday, so we all went to the Xscape Centre at Milton Keynes on Saturday to use them!

The Airkix website gives you a pretty good idea of what it's all about, but you really have to experience it; the sensation is predictable enough but I hadn't expected it to be so tricky to maintain a precise position. You need to find a stable body-shape, and then to make all adjustments slowly. And as you have only two 1-minute flights, you have to learn all this quite quickly.

Becca is very athletic and supple, so she found the right shape almost immediately. It took me (a much older, creakier body!) a while longer, but I got there. And we have a DVD of our session to prove it! Must try to get some of that on to this site.

The tunnel equipment is awesome - I have no idea how powerful the fans are, but air is drawn from a very wide base area up through a funnell-walled section into the chamber (which isn't very big), which increases the velocity of the airstream. I didn't watch the instruments but I believe the windspeed is in excess of 100mph. There's a fine degree of control available too: they reduce the windspeed for smaller, lighter flyers, or when someone accidentally ends up too high.

This was huge fun! I've always wanted to experience free-fall, but I'm a still a bit scared of parachuting. This is the only other way...

Back again, at last

2006-05-17T12:25:00.001+01:00
Two weeks ago, following some interruptions of service, my hosting provider simply went out of business. This was a bit of a shock, as RazorLogix (RL) had looked like an almost perfect shared host - great service (until the outages), and a very good shared-host plan which included SSH access (non-root), Java, Python (including mod_python), RubyOnRails, Subversion support and more.

When the end came, it meant backing up everything and moving to a new host. Having done this only a month or two previously (in order to move to RL!), I wasn't very pleased. The first problem was, who to host with? I looked around for something close to the RL offering. There are other shared-host companies which offer mod_python, but usually these are a bit constraining and very few offer SSH access.

In the end, I decided to take a slightly bolder step and go for a Linux virtual server. I hadn't considered running my own server before because it had meant either running the hardware myself 24x7 (not really something I want to do), or purchasing access to a rack-mounted server (very expensive). Now that virtualization has arrived in the hosting market, running your own server is very cheap and gives you complete flexibility. The downside of course is that you have to do everything else: administer the server, install software and so on.

There appear to be two main approaches to the virtual server market: User Mode Linux (UML) or using a virtualization layer such as Xen. The result, as far as the user is concerned, is broadly the same: you have a complete Linux OS environment, an allocation of main memory, and some disk space (a few GB). Of all the providers I looked at, the two I concentrated on were Bytemark, which uses UML and Rimuhosting, which uses Xen.

I chose Rimuhosting. So far, so good. They are based in New Zealand but most of the hardware seems to be in US datacentres. Requests for help have been answered promptly and they do seem to know what they're doing. The standard VPS install has almost everything I need, right out of the box, including Java and Python. For most of the basic setup, there is a web-based control panel.

As for me, the experience has been a bit like going back in time, to the years I spent developing in C and C++ on Unix (12 years or more ago). It's surprising just how much has stayed with me, even csh/bash stuff. The biggest challenge has been understanding the components of the standard (RedHat Enterprise-based) distribution that Rimuhosting uses, e.g. Postfix and Dovecot for email. Having SSH access (and SCP access, so you can use something like WinSCP) means complete freedom: now I can do some of the server-side stuff I always wanted to do!

Back again, at last

2006-05-17T12:25:00.000+01:00
Two weeks ago, following some interruptions of service, my hosting provider simply went out of business. This was a bit of a shock, as RazorLogix (RL) had looked like an almost perfect shared host - great service (until the outages), and a very good shared-host plan which included SSH access (non-root), Java, Python (including mod_python), RubyOnRails, Subversion support and more.

When the end came, it meant backing up everything and moving to a new host. Having done this only a month or two previously (in order to move to RL!), I wasn't very pleased. The first problem was, who to host with? I looked around for something close to the RL offering. There are other shared-host companies which offer mod_python, but usually these are a bit constraining and very few offer SSH access.

In the end, I decided to take a slightly bolder step and go for a Linux virtual server. I hadn't considered running my own server before because it had meant either running the hardware myself 24x7 (not really something I want to do), or purchasing access to a rack-mounted server (very expensive). Now that virtualization has arrived in the hosting market, running your own server is very cheap and gives you complete flexibility. The downside of course is that you have to do everything else: administer the server, install software and so on.

There appear to be two main approaches to the virtual server market: User Mode Linux (UML) or using a virtualization layer such as Xen. The result, as far as the user is concerned, is broadly the same: you have a complete Linux OS environment, an allocation of main memory, and some disk space (a few GB). Of all the providers I looked at, the two I concentrated on were Bytemark, which uses UML and Rimuhosting, which uses Xen.

I chose Rimuhosting. So far, so good. They are based in New Zealand but most of the hardware seems to be in US datacentres. Requests for help have been answered promptly and they do seem to know what they're doing. The standard VPS install has almost everything I need, right out of the box, including Java and Python. For most of the basic setup, there is a web-based control panel.

As for me, the experience has been a bit like going back in time, to the years I spent developing in C and C++ on Unix (12 years or more ago). It's surprising just how much has stayed with me, even csh/bash stuff. The biggest challenge has been understanding the components of the standard (RedHat Enterprise-based) distribution that Rimuhosting uses, e.g. Postfix and Dovecot for email. Having SSH access (and SCP access, so you can use something like WinSCP) means complete freedom: now I can do some of the server-side stuff I always wanted to do!