Thursday, 2 July 2009

HAPI 0.6 Released

Anyone involved in HL7 processing should be aware of the HL7 API (‘HAPI’) project.  HAPI is a set of Java libraries for parsing, unparsing and manipulating HL7 messages, plus some communication primitives for handling MLLP communication. 

This was started some time ago by Bryan Tripp at the University Health Network in Canada, but recently the work of maintaining the project and pushing forward with development has been taken on by James Agnew.

Version 0.6 brings support for HL7 v2.5.1 and v2.6, and separate jars for the different HL7 versions. James’ announcement also hinted he’s looking at significant performance improvements in future releases. This is great news: thanks are due to James for his continued good work.

If you’re trying to find the bits, note that the download link isn’t in the HAPI site sidebar – go to the Using HAPI / Developing page, and you’ll find the link to the Sourceforge download page there.

Technorati Tags: ,

Saturday, 13 June 2009

Groovy and Language Design

I was a little surprised to read Elliotte Rusty Harold’s recent comments on Groovy, calling it the ‘Edsel of programming languages’.  It’s an amusing comment, but pejorative and challenging enough to motivate a response. 

The past few years have been interesting and fruitful from a programming language perspective: Ruby, Groovy, Scala, F#, and the evolution of C# are those which spring to mind.

Many of these developments have been about finding a way to push functional-programming idioms (lambda expressions, currying) or dynamic language features (type inference, closures) into well-established imperative languages like C# and Java. I think the results have been mixed: imperative languages are essentially assignment-based, not expression-based, so you end up with a hybrid which, from a language syntax point of view, is neither one thing nor another and arguably suffers as a result.  But it’s difficult to argue that these efforts don’t deliver real positive utility.

An example of this is how C# has evolved support for lambda expressions, expression trees and implicit typing. First (in v1) we had simple delegates, then (in v2) we were able to drop the delegate syntax and declare anonymous methods inline, and finally (in v3) we can use a compact lambda syntax which looks a bit like ML. But because all this has been added to what remains fundamentally a statically-typed, compiled language, you still have to put up with all the nasty explicit declaration of types (and type parameters) on the l.h.s. of every assignment, just to keep the compiler happy.  So the final step was to introduce the var keyword and bring implicit typing into the language. Not dynamic typing, just implicit typing: we get to trim away the ugly declarations but keep the compile-time type safety. 

The confluence of these developments enabled things like LINQ: blending a query sublanguage into the primary language syntax. The best exposition of this I have read is by Ian Griffiths, in his excellent post on expression trees: see the example code just over half-way down, under the section ‘Expression Trees, LINQ, Deferred Queries, and Databases’. It’s all so superficially appealing – just look at the motivating use-case in Ian’s post. But still there’s a small, nagging voice in my head saying this isn’t quite right...

It feels as if we’re trying to pretend there’s no boundary (or intermediary) between the program text and the resource we’re querying. This is similar to the way we tried to pretend that remote procedure calls were really local calls in the old (and now largely discredited) distributed programming models such as CORBA and DCOM. Recall the fallacies of distributed computing and the tenets of SOA: these are essentially warnings about the dangers of implicit or flawed assumptions.  Although the tenets have been challenged (and were threatened with retirement) most of what they said is timelessly valid.  In the context of LINQ, I think my nagging voice is pointing to the boundary-crossing in a LINQ query and muttering “private implementation technique, not primitive construct”.

What’s this got to do with my starting point for this piece, which was Harold’s ‘Edsel’ jibe?  I suppose it’s this: whatever we may feel about the virtue of purity in language design, in the end productivity and usability are the bigger drivers, and with skill it is possible to combine language features without creating a monster.

Technorati Tags: ,,

Sun, Java and Innovation - reflections

My previous post on the Sun acquisition was a rather naive and misty-eyed piece of sentimental warbling. The truth is that Jonathan Schwartz’s vision hasn’t delivered quickly enough for the market: the share-price has suffered and Oracle has picked up something of a bargain. Pity, because (perhaps looking through those rose-tinted spectacles again) I rather like Schwartz’s ideas for growing the business, essentially giving away the technology and then ‘monetizing adoption’.  But it’s not happened nearly fast enough and one wonders whether it ever could. 

An editorial in SD Times sums up Sun’s difficulties nicely as ‘too much vision, too little execution’.  It’s not enough to give away tools and hope the next generation will adopt them. You need to innovate and to lead, because the brightest and best of the next generation want to climb the mountain, not travel for free in the foothills. To innovate, you need great scientists and engineers; to retain those folk, you need to pay them adequately and provide them with the facilities and resources they need to be brilliant. To do all of that, you need money.  And that comes from sales.  It’s as simple as that.

Just look at Microsoft Research: you just can’t foster this quantity and quality of innovation unless you have a lot of money to invest, and you only get that by doing a lot of very effective monetizing.  Microsoft and Oracle: we don’t like them very much, do we? But they’re really good at monetizing.

As for Java, most seem to agree there is no threat to its future as an enterprise platform: it’s open-technology now and well embedded in the enterprise.  What about the future of the Java language? Do we really need to worry about that? The language itself matters much less than the JVM, which surely has a very bright future given the newer languages targeting it, especially Scala and Groovy.  Should we care whether (e.g.) closures make it into the Java language, if we can use newer languages like Scala and continue to leverage all the existing Java libraries, components and infrastructure?

Technorati Tags: ,,,

Tuesday, 28 April 2009

As the Sun sets…

Why does Oracle’s purchase of Sun make me feel slightly sad?  Silly, sentimental reaction, isn’t it?  I should know better. After all, today’s Sun isn’t the super-confident (some would say arrogant) innovator and market leader I grew up with: arguably, today’s Sun needs rescuing from itself, needs a sharper focus on what it does best, and to sell more of fewer things.

But I have a soft spot for Sun. My first proper programming job involved writing C/C++ (and using Cfront – remember that?) on a Sun-3 workstation, and various versions were part of my working life for some time. I loved the solid feel of these machines. Remember the optical mouse that only worked on those special, shiny metal mats?

Between then and now, my only links to Sun have been through OpenOffice and Java. I played with Java quite early on, abandoned it in favour of Microsoft .NET, but have recently (and happily) returned to it. For the last couple of years I’ve been an enthusiastic user of NetBeans – I do hope Oracle recognizes just how good NetBeans is.  I have also tried OpenSolaris: ZFS is simply awe-inspiring, and very nearly enough on its own to make me run OpenSolaris, though truthfully I don’t think I need ZFS, and Windows remains simply more convenient and usable for everyday.

I’ve used OpenOffice for a long, long time.  I know Writer pretty well, warts and all. You need to ignore some of the cosmetic shortcomings, persevere with it and appreciate its fundamental strengths; things which I think make Writer better than Word. Occasionally I use OOo Writer to help colleagues debug and rescue Word documents which have evolved uncontrollable formatting: it amuses me to be using a free tool to clean-up after a rather expensive one.  I really hope Oracle will resource and manage the OpenOffice program properly. With the right additional effort, they have a potential Office-beater.

But the crown-jewels are Java itself, and the NetBeans IDE.  Everyone is trying to second-guess what Oracle will do with Java: I don’t have anything to add.  But I really want to add my voice to those hoping Oracle will recognize just how good NetBeans has become, not just as the best Java IDE out there, but also a first-class platform for building rich-clients.

Technorati Tags: ,,,,

Saturday, 28 March 2009

EIP Shapes for OpenOffice Draw

If you’ve read the excellent Enterprise Integration Patterns (EIP) book by Gregor Hohpe and Bobby Woolf, you may also know that free VISIO stencils are available for the EIP shapes, from the download page.

This is great for those who have a copy of VISIO (or whose employer pays for a copy), but for those of us who don’t want to pay however many hundreds of dollars Microsoft wants for VISIO these days, an alternative (especially a free one) is always welcome.  OpenOffice (OOo) Draw, though hardly as functional or attractive as VISIO, is both free and good enough for many diagrams.

So I decided to see if I could create a set of OOo Draw shapes from the VISIO stencil, and see whether Draw could be practical for integration diagramming.  In short, I succeeded in converting the stencil, and I think Draw is just about good enough. For those not interested in the background, I’ve made the material freely available from this page on my website, so just go ahead and download/install.  Otherwise, read on.

Exporting the VISIO Shapes

One way to get shape data into Draw is via SVG. So I started with the VISIO shapes, freely downloadable from the EIP site, and as I use VISIO at work I was able to load up the stencil and export it as SVG. 

You need to install an OpenOffice extension for importing SVG. I’m using OOo 3.0: the extension installed without incident and the import was straightforward. I was surprised at how good the resulting shapes looked in Draw.  Each shape is composed of drawing primitives, so you need to select and group them to create a shape you can place and move easily. I’ve done this for all the shapes in the download.

So now we have the shapes, where can we put them to make them easily reusable?  In Draw, there’s no stencil feature as such: the only option is the Gallery:


The Gallery

The gallery feature in Draw isn’t very appealing, initially. When you first install OOo, the gallery contains some spectacularly awful materials: nasty backgrounds and bullets which I cannot imagine ever wanting to use, and it even contains some pointless sound-clips that surely only a 5-year old might use (and only then if they were desperate). The shapes and graphics are reminiscent of shareware software from the Windows 95 era.  And what is the point of the built-in media player?  Why can’t OpenOffice grow up?  It’s so very close to being excellent in many respects: things like this just make it look silly and dated.

Anyway, back to the gallery feature. The one saving grace of this feature is that you can create new gallery themes and add your own shapes to them. To do this, create a new theme, give it a name and then drag your shapes into it.  Note that when you drag/drop your shape you must hold down the mouse button and pause a second or two before dragging: this pause changes the drag-mode from a simple shape move to the required move/copy.

I’ve done this for all the EIP shapes, creating an EIP gallery theme:


Sharing the EIP Theme

Unfortunately, Draw doesn’t implement a theme export facility so to share this new theme with others requires some irritating file copying.  The contents of the Gallery are stored in your profile (or home) folder. On Windows this is likely to be:

C:\Documents and Settings\<user>\Application Data\\3\user\gallery

On Unix/Linux, check your home folder. There are several file types in here and (unusually for OOo) they are binary, not XML. This makes it difficult to reverse-engineer their purpose and relationship but fortunately it doesn’t seem to be necessary to understand fully what’s going on. The key thing to note is that the numbers in the file names bind together related files.  When you create a new theme, a new set of files appears to be created.

For the EIP theme I have manually renamed the files to use numbers which I hope won’t clash with any existing gallery files you may have.  Check the zip contents before you extract the files into your OOo folder and if you have existing files with the same name, simply rename the files in my zip to be unique in your environment. 

The Result

I think the result is pretty good.  As a test, I have (partially) reproduced the diagram from the end-cover of the EIP book in a Draw file (.odg) which is included in the download zip:


I hope this may be some use to other folk.  Download from this page.

Friday, 27 March 2009

OpenESB and BPEL

Thanks to the folks who commented on yesterday’s post I have managed to overcome the assign problem and of course the answer was very simple: I had an inconsistent set of components installed in NetBeans and GlassFish.  I was using NB 6.5 together with components downloaded from the OpenESB site. 

The best solution is simply to download the complete NetBeans/GlassFish/OpenESB combination from the OpenESB downloads page and work with that. On the download page the top section contains the GA (stable) GlassFish ESB / NetBeans combination, and the nightly build of the latest (unstable) additional components (i.e. binding components, service engines etc.) which match this GA release.

The next major section on the page contains the GlassFish ESB v2.1 downloads (milestone 1 at the time of writing), which are not yet considered stable. I grabbed the 2.1 release, plus the HL7 BC (and NB design-time modules) from here.  Installation was completely pain-free.  I particularly liked the way the additional components are packaged as a jar-based installer – very nicely done. 

Because the main package is a GlassFish ESB bundle, it installs by default to a non-standard path: C:\GlassFishESB. Everything is installed under this directory, including the configuration files for NetBeans, which would normally be placed in your home directory (under .netbeans). Presumably this is done to allow parallel installation with a stable NetBeans/GF without either interfering with the other. What this means, of course, is that all your preferences (editor settings etc.) are not transferred. This is easily fixed, manually, by copying over the stuff from: C:\Documents and Settings\<user>\.netbeans\6.5\config\Editors to C:\GlassFishESB\.netbeans\glassfishesb\config\Editors

Anyway, back to the BPEL issue.  A clean rebuild of everything was sufficient: the CA deployed to GF, and it ran fine.  I used Hermes JMS and watched the test messages arrive on the queue. Now I can push ahead and explore more of OpenESB and BPEL.

Technorati Tags: ,,,

Thursday, 26 March 2009

Problem with OpenESB BPEL assign

After far too many cycles staring at code, building, deploying, debugging and undeploying a pretty simple composite application, the damn BPEL assign activity still causes selectionFailure at runtime.

The BPEL is trivial, and the BPEL copy elements for optional elements in the source schema are marked with the ignoreMissingFromData attribute.  I have rebuilt everything from scratch.  I know the redeployments are successful because the error reports points to the correct line following an edit and I can use the BPEL debugger to step through the assign, eventually failing on the optional element.

I was given the hint about the ignoreMissingFromData attribute by Michael Czapski, in a blog comment.  Googling around for more help, I came across a few other postings, and discovered that the attribute could be placed at the process level or at the individual copy activity level. Jeff Sexton also has a nice post on the assign issue.

In Michael’s tutorial he does in fact place the attribute at the process level (see p.95 of the PDF document which accompanies his tutorial). Placing it at the process level is not very attractive because it will mask the absence of mandatory fields.

When placed at the copy element level, it seems to fail. Here’s the error I see, as output in the GlassFish console window:

BPCOR-6151:The process instance has been terminated because a fault was not handled; Fault Name is {}selectionFailure; Fault Data is null
com.sun.jbi.engine.bpel.core.bpel.exception.StandardException: I18N: BPCOR-3023: Selection Failure occurred in BPEL({}bpHL7Processor) at line 28!
BPCOR-6129:Line Number is 26
BPCOR-6130:Activity Name is Assign1

Note the line numbers in the above report: the first of these (line 28) is where the selection failure actually occurred, line 26 is the opening tag of the enclosing assign activity element.  Here’s a shot of the source so you can see the corresponding line numbers. I placed all the optional elements together at the start of the assign:


So why doesn’t this work? Is this a bug? Perhaps I’ve run into something which only works in the JavaCAPS product and not in OpenESB?  Shame.

Another unfortunate discovery is that the OpenESB UI in NetBeans 6.5 doesn’t seem to support adding the ignoreMissingFromData at the process level.  In Michael’s PDF he selects the top-level BPEL process scope and uses the property panel to set this attribute.  It doesn’t appear to exist in NB 6.5. In the shot below I’ve tried to capture the same UI as in Michael’s tutorial:


Note there’s nothing below the ‘Documentation’ entry: presumably this is additional UI support which you get in the commercial JCAPS product. I’m not that concerned with the missing UI, as long as I can add the attribute in the XML source.  So I edited the BPEL XML, using Michael’s tutorial source as a guide. It’s easy to add the process-level attribute; only the namespace prefix had to be changed:

<?xml version="1.0" encoding="UTF-8"?>

So another clean/build/deploy and … it still didn’t work. So now I’m stuck, and I hate being stuck. It’s almost enough to put me off the product.

The impression I get is that BPEL is great for very simple maps and transforms, but anything complex quickly becomes awkward. I haven’t gone beyond ‘simple’ so far, so I’m getting a little concerned. The pretty visual tools and NetBeans integration are cool, but the complexity overhead seems high. It reminds me of Microsoft’s BizTalk product from a few years back – a lot of fiddly config, and not being able to ‘see the wood for the trees’. 

But I want to persevere with OpenESB: something about this product suggests that you need to reach a certain level of enlightenment, after which everything becomes clear and the benefits outweigh the overheads. I’m just not there yet.

Technorati Tags: ,,

Wednesday, 18 March 2009

DropBox again

The previous post elicited a comment from Andy at CloudBerry Lab:

I am a developer of another online backup product powered by Amazon S3 that we are going to release to beta soon. I would appreciate if you take a look and may be post a review on your blog.

Well I did take a look, but the CloudBerry product isn’t really what I’m looking for.  If all you want/need is a Windows-only client for accessing your S3 buckets directly and copying/moving content, then this product might well be just right.  But it doesn’t really compete with DropBox or JungleDisk.  Andy: if your product were closer to DropBox, and cross-platform, then I would have reviewed it.  As it is, I can’t pass judgement.

If you have time to plough through a very long blog thread, check out Jeremy Zawodny’s excellent piece on Amazon S3 tools. The discussion thread contains links to most (maybe all) the tools out there.

I’ve also been reading the DropBox forums. It’s good to see plenty of other people have requested features I miss (e.g. ability to configure DB to ignore certain file patterns).  There is an upcoming features wiki page which lists some things coming ‘soon’.  Nothing on using your own S3 space though.

Monday, 16 March 2009

JungleDisk and DropBox

There are quite a few network-drive products out there competing for our attention these days.  I’ve tried quite a few (including and SkyDrive), but narrowed the choice down to the two which seem the best: JungleDisk and DropBox.

JungleDisk attracts me mostly because I can use my existing Amazon S3 storage behind the JungleDisk tool and pay only for the space I actually use. (Note, JD now also uses RackSpace CloudFiles, which actually looks even better than S3).  DropBox is free for 2GB, then a hefty 99 USD annually, for 50GB.  Round one to JungleDisk, in my view.

Then we come to the user-interface. Both tools integrate with Windows and Linux, JD using drive mapping to expose the storage and DropBox using a special folder inside My Documents, with icon overlays to indicate file status.  Both support drag/drop access and run a small tray-resident UI application.

But DropBox is just so, so much nicer to use than JD in the everyday Windows context.  It feels better integrated and the UI seems cleaner.  Other folk have blogged about this difference and I must concur – DropBox has the edge.

Now to the subject which prompted this post in the first place. Neither of these products appears to handle proxy servers particularly well, especially when switching between proxy / no-proxy.  If I restart Windows and forget to switch off the proxy in JD, here’s the mess I’m greeted with when Windows starts:


Yuk. Can’t it simply notice that the proxy isn’t responding, log the fact / decorate the tray icon, and leave it for me to sort out?  It gets worse: if I click on the links (for more information) look what I get:


Is this really what they want the user to see?  This is awful.

DropBox is slightly better, but still doesn’t work properly if I leave the proxy on, and restart. No nasty dialogs, but the network connection isn’t resolved, even if I set it to ‘auto-detect proxy settings’ which according to the DropBox site should use the IE settings. Why can’t these tools auto-detect proxies properly?

DropBox files are cached on the local machine which means if the network is down I can still work on all my files locally, and re-sync when I next connect.  JungleDisk does cache your files, but in a pretty inaccessible way in your profile.  The path will be something like C:\Documents and Settings\<user>\Application Data\JungleDisk\cache\e9998872111157539d8880eca4456345-default

Another good feature of DropBox which isn’t available in JungleDisk is sharing files and folders: in JD, everything is private.

DropBox gets so many things right. The one and only feature I want from JungleDisk is the S3 / CloudFiles backing store. Obviously, the DropBox business model is built around the 99 USD annual charge so I don’t know whether this can/will ever happen.

Wednesday, 11 March 2009

OpenESB and HL7

Technorati Tags: ,,,

As well as looking at Apache Camel for healthcare integration, I have also been spending time with Sun’s OpenESB product.  OpenESB is the open-source counterpart of the Java CAPS product, a mature and well-supported Enterprise Service Bus (ESB) built on the Java Business Integration (JBI) standard.

OpenESB is delivered along with the GlassFish application server (v2) when you install the SOA extensions to the NetBeans IDE.

Michael Czapski of Sun is an excellent source of information on healthcare-related application of the CAPS / OpenESB product line. He has created a really substantial demonstration project based on a healthcare integration scenario. His blog entry and the associated downloads (and screencast) are well worth getting hold of.  The screencast and writeup are based on Java CAPS, the commercial product which provides a few nice UI improvements and I believe some additional tools which the OpenESB product does not have.

I decided to go through Michael’s complete tutorial, using NetBeans 6.5 and the bundled OpenESB / GlassFish combination, building the solution and making notes as I went along. I’ll put the source code for the NetBeans (6.5) and OpenESB project group up on a publically available Assembla Subversion space (URL to follow), in the hope they may be useful to others who want to use OpenESB to do a similar thing.

As already mentioned, the differences between Java CAPS and OpenESB are superficially small, so if you are already familiar with one or other product you probably only need Michael’s notes and material. However, if you’re relatively new to OpenESB (as I am), there are a few places where the absence of JavaCAPS tooling means you need to know what you’re doing to patch-up the OpenESB solution.

Getting Started

My first big lesson was getting a correct / compatible set of NB plugins and libraries installed. The zip archive provided by Michael appears to contain everything you need, including the XSDs, NB plugins and encoder libraries. As the article was published very recently, I assumed these were the latest versions so tried to configure my NB 6.5 instance using these.

I followed the instructions for installing the encoder library (first uninstalling the existing one), then installing the NB plugins. However, the plugin installation just wouldn’t go ahead:


I tried various combinations but in the end had to uninstall GlassFish v2 and run the NB installer again to reinstall GF plus the OpenESB bits. Fortunately, this is actually quite a quick process, and puts everything back as it was.

I won’t go through all my subsequent false-starts, but simply tell you to download a matched set of the latest versions from this location:

The components here should all be compatible. It really is best that you download and install these yourself, so I won’t put them in the Assembla SVN space.

Doing without the Java CAPS Wizards

The most obvious differences when watching the screencast are the degree to which the commercial product contains wizard steps and conveniences for generating all the configuration items. Reasonable enough I think, and so far I have been able to complete every step using OpenESB and NetBeans 6.5.

When creating the concrete WSDL definitions with the HL7 bindings, the wizards in the Java CAPS product (seen in Michael’s screencast) do make the whole process slightly simpler. But if you take a little time to understand what these wizards are actually doing in the generated WSDL, it’s not too hard to complete the steps manually.  That’s easy to say now, but I should ‘fess-up and admit that it took me a little while to spot what I hadn’t done, a couple of times.

For example, once I’d completed the HL7Consumer_CA_A01_A03Delim_HL7In WSDL (sorry Michael, I’m not a big fan of your naming convention!), I simply couldn’t add it to the Composite Application design surface: the tutorial document instructions didn’t appear to work. I could see the WSDL in the list of available WSDLs and select it – but could not add it to the canvas, and couldn’t see any error message to tell me why.

After a cup of tea and a biscuit and a bit of careful thought, I realised why. The HL7 protocol properties were completely missing from the port declaration in the Services section.  It’s easy to fix this, using the context menu commands:


Then edit the properties in the property grids. Of course, in the JavaCAPS product the wizards prompt you for these properties and create the related WSDL bits behind the scenes.  I don’t like depending on wizards. Fortunately the NetBeans tooling gives you just enough convenient UI to hide the underlying cruft, but also lets you work at the XML / source level and see what’s going on down there.

That’s probably enough for this post.  There’s so much more to write but I need to knock it into shape before I publish.

Wednesday, 25 February 2009

Tool Support for DDD

I like Eric Evan’s book Domain Driven Design very much.  It’s such a nice distillation of a relatively small number of principles, all of which possess real, fundamental value. This is a protein-rich book: it may not feel as filling as the more prescriptive, methodology-driven ‘just-do-as-I-say-and-it-will-come-out-right’ books, but they’re just carbohydrate.  DDD is good for you in the long-term.

I’ve just discovered that you can read some of the book online, via Google Books.  You can also find out more from the DDD Site.

The real point of this post is to share my experience with a small, free tool called wikidPad, which I stumbled on last week when looking for a desktop wiki-based tool to use for capturing Ubiquitous Language (UL) definitions.  Capturing and refining the UL is an essential part of applying DDD, and having a fast, convenient (and open) medium for doing this is essential.  A wiki of some kind is almost ideal.

Most desktop or single-user wikis are either cut-down versions of the equivalent web apps (PHP or whatever), running on localhost ports, or they are single self-editing HTML pages e.g. TiddlyWiki.  Many of these (including TiddlyWiki) are very good indeed, but somehow, none of them has felt quite right for the UL use-case, which is all about getting hyperlinked definitions into a reusable form as quickly as possible.  wikidPad does this superbly well, in my opinion. 

It’s also a small and fast desktop application, which I find a little more convenient than using a wiki app in a browser. Some examples to support that:

  1. wikidPad automatically constructs a tree-view of pages on the left hand side of the application – this makes navigation very fast.
  2. wikidPad does auto-completion of wiki-words (Ctrl-space), which makes writing new entries which link to existing items very fast.
  3. Drag and drop support, and support for multiple tabs – again, all good for speed and usability.

When you’re ready to publish the UL to a wider audience, wikidPad will export the whole thing (or any selected part) as a set of HTML pages.

It’s my first day using this tool, so it may be that I’ll discover something to dent my enthusiasm. But so far, it’s all good.

Monday, 23 February 2009


Technorati Tags: ,,

Although Project Locker has served me very well for a good many years, better alternatives have emerged recently. I didn’t set out to switch providers, only to see whether Project Locker would support Mercurial or git. At the time, neither were available – since then PL have added git support. It was while looking around at the many newer project hosting services that I came across Assembla.

The Assembla proposition is simple and very reasonable: public, open-source projects are hosted for free (up to 200MB). Private plan pricing is extremely keen:

  • $2.00 per user per space, per month
  • $3.00 per gigabyte of disk space per month
  • $8 per user per month maximum. After you pay for a team member in four spaces, additional spaces are free.

Private plans get unlimited disc space.

But for me, the economics of Assembla are only half the story: the other half is a combination of things: the clean, functional web interface, the range of additional tools (Trac, wiki, files, chat, project dashboard etc.) and a strong feeling that the people behind Assembla know what they are doing.

Assembla does offer git hosting, but I’m sticking with Subversion for now.  SVN is good enough for my purposes and I can’t live without TortoiseSVN.

I’ve been using Assembla for a couple of months and my experience has been very good indeed. 

Tuesday, 17 February 2009

Camel, HL7, HAPI and NAK Messages

The previous couple of posts were related to using Apache Camel in a healthcare integration setting.  The HL7 support in Camel is built on the HAPI library which provides Java classes for most of the HL7 v2 variants, plus MLLP support.  So in integration terms, HL7 can be considered a combination of an endpoint type (the MLLP/TCP protocol) and a message format (the HL7 ER7 or XML representation).

The Camel architecture nicely isolates endpoint-types and message-structure types in Components and DataFormats, respectively.  A Component is something which knows about a specific communication mechanism or protocol: in the case of HL7, the Camel HL7 Component knows how to exploit the underlying Apache Mina library to exchange delimited text messages using the HL7 protocol.  The Component also acts as a factory for Camel Endpoint objects: it hands out individual Endpoint objects, each of which represents an instance of the endpoint-type implemented by the Component. Camel DataFormat objects marshal / unmarshal byte-streams or strings to / from Java objects of some type.  This makes it much easier to build content-based routes in Camel.

HL7 ACK / NAK Messages

In the previous post, one of the issues I had to deal with in the example route was returning an ACK or a NAK to the message originator. The HL7 protocol uses a positive-acknowledgement scheme: receivers will ACKnowledge (ACK) messages they understand and can process, and will Negatively AcKnowledge (NAK) anything else.  The HAPI library provides support for this in the makeACK method of the DefaultApplication class.  Unfortunately, this is less helpful when we need to return a NAK.

Recall that the first part of my route was checking to see whether the right kind of message has been received. In the sample, I was testing just the message type (MSH-9.1) and the trigger event (MSH-9.2) fields.  In reality, you would probably also want to check that the sender was using the same version of HL7 as you are expecting: you can do this by examining the value of field MSH-12. It will become clear why this is important, below.

Suppose the inbound message fails the MSH-9 test. The otherwise() clause in my little route will send the HL7 data (unmarshalled from the stream) to the badMessage() method on my handler bean.  This method expects a single argument of type ca.uhn.hl7v2.model.Message, i.e. a generic HAPI HL7 message object.  The argument can't be of a more specific type than this because the Camel route builder cannot know in advance what will be received and unmarshalled from the endpoint; it can only assume it will be a HAPI Message.

Constructing an ACK

In the badMessage method, I need to construct a HL7 NAK message to be returned to the caller. HL7 ACKs and NAKs share the same structure: we just need to change the value of one field (and optionally add a diagnostic message) to indicate NAK. The obvious solution is to call DefaultApplication.makeACK, mentioned earlier, and modify the returned structure as necessary.  MakeACK's single argument is a single Segment, the intention being that the caller passes-in the MSH segment of the original inbound message:
Message makeACK(Segment inboundHeader)
The reason for this is that the makeACK method needs some values from the inbound MSH segment to populate fields in the Message Acknowledgement segment (MSA) of the ACK correctly. An HL7 ACK (or NAK) must be associated with the original message (i.e. the inbound message being ACK'd or NAK'd) so that the originator can distinguish which of its outbound messages is being acknowledged or rejected. This is done using the Message Control ID, which is field 10 of the MSH segment.

But remember where we are in the Camel route: we have an invalid message in our hands. We don't know the exact class-type of the inbound message object so we can't cast the Message to a specific type. And the Message interface itself doesn't provide a way to extract the MSH segment. So how are we going to call makeACK(Segment)?

We cannot simply new-up an instance of some specific message type (e.g. a version 2.3 ORM^O01) and use its MSH segment as the makeACK argument:

    // Assume ca.uhn.hl7v2.model.v23.*
    ORM_O01 orm = new ORM_O01();

    ACK ackMsg = (ACK)DefaultApplication.makeACK(orm.getMSH());        // <-- Bang!

    // Set MSA.1 to Application Reject (AR)

    // etc.

This will compile, but won't work. The manufactured ORM_O01, despite being an instance of a specific version of HL7, is not properly constructed, in several ways. First, MSH-12.1 will not contain "2.3", which is surprising, given that HAPI should be able to populate MSH-12.1 for a new message instance. If we pass the ORM_O01 MSH to makeACK, it tries to get the value of MSH-12.1 but finds a null. This causes makeACK to fall-back to creating and returning an ACK for version 2.4 of HL7 (this is baked-in to the HAPI code - I'm not sure if this is an arbitrary HAPI design decision, or to do with the HL7 spec). And of course we get a ClassCastException at run-time as a result, because we are expecting a v2.3 object.

But even if this did work, isn't it ugly? We'd be baking-in a version of HL7 because we must choose a specific ORM_O01. So how do we create the ACK structure, and how can we make our code work for any version of HL7 supported by HAPI?

Terser to the rescue

Fortunately, HAPI provides a handy utility class for parsing and unparsing HL7 messages of arbitrary versions - the Terser. This will let us get hold of the MSH segment from the original message, so we could at least call makeACK with the original MSH:

    // This still depends on the ackMsg decl. being for the same specific version
    // as the original message.
    Terser t = new Terser(originalMsg);
    Segment msh = t.getSegment("MSH");
    ackMsg = (ACK)DefaultApplication.makeACK(msh);

Now when we call makeACK we are passing-in a valid MSH so it will extract the HL7 version number and dynamically create the correct ACK class, using Class.forName.  But as the comment says, this stilll only works if the compiled-in ACK declaration matches the returned ACK message version. We need to lose the compiled-in ACK declaration, but still find a way to populate the ACK message's MSA segment with the right values.

Fortunately, we can use the Terser to set the field values in a version-independent way. This is exactly what DefaultApplication's static methods makeACK and fillResponseHeader do.  So I used a combination of the Terser and the guts of DefaultApplication.makeACK to create a couple of helper methods which do the right thing.  I've put the source code on this page in my Google site, for anyone who wants to use it.

Monday, 16 February 2009

Compound Predicates in Camel

Thanks to Claus Ibsen for such a helpful response to my previous post.  As Claus points out, the answer is to use a Camel helper class, PredicateBuilder, which contains a bunch of useful static combinator methods.  Claus shows in his post how to implement my trivial conjunction easily using the and() method.  Note the generic signatures of all these helper methods, e.g.

<E extends Exchange> Predicate<E> and(final Predicate<E> left, final Predicate<E> right)

Because they accept Exchange (or any derived class), any custom Predicate you might need to write has access to the complete message exchange structure (in, out and fault), and all predicates could be applied to a custom extension of Exchange.

Having got over those two 'humps' (sorry - couldn't resist that), I'm feeling very good about Camel.  It's a mighty impressive piece of work and looks like it has a lot of potential for the area I'm working in right now.

Thursday, 12 February 2009

Camel Routes and HL7

I've been spending some time with Apache Camel and ActiveMQ, with a view to using Camel in a healthcare integration project.  Camel has support for HL7 both at the message structure level (wrapping access to the HAPI libraries) and at the protocol level (via the Apache Mina extensions for HL7 MLLP).

As anyone who's looked at Camel (or IONA's FUSE Mediation Router) will know, Camel implements a rather nice Java DSL for building routes.  This lets you write things like:

              .when( header("hl7.msh.messageType").isEqualTo("ORM") )
                    .beanRef("hl7handler", "handleORM")
                    .beanRef("hl7handler", "badMessage")
              // end choice block- marshal the ACK/NAK back to the TCP endpoint

Very readable, but as you might expect, it hides quite a bit of complexity.  This is no bad thing, but you do need to understand what's going on behind the DSL. 

Camel exploits Spring, and in the above fragment "hl7listener" and "hl7handler" are bean names. The unmarshal and marshal methods allow DataFormat objects to convert between byte-streams and more convenient objects, such as HAPI messages.  The choice/when/otherwise operators allow predicates to control routing.

It's taking a while for me to grok Camel; here is a couple of things I'm not yet clear about:
  • Knowing how to get an intermediate stage in a route to return an ACK or NAK to the originator, without subsequent (or parallel) stages in the route from doing so. 
  • Creating compound boolean expressions in predicates. For instance, if I want a conjunction inside a when(), I can't have it. There don't appear to be combinators for expressions.  
In the fragment above for example, how can I express a conjunction/disjunction in the when( ... ) expression?  It only seems to be able to handle a single condition: I haven't found a way to combine predicates e.g. I can't do this:

when ( header("hl7.msh.messageType").isEqualTo("ORM") &&
             header("hl7.msh.triggerEvent").isEqualTo("O01") ).    // etc.

I have tried alternatives, e.g. nesting the when clauses:

              .when( header("hl7.msh.messageType").isEqualTo("ORM") )
                    .when( header("hl7.msh.triggerEvent").isEqualTo("O01") )
                        .beanRef("hl7handler", "handleORM")
                        .beanRef("hl7handler", "badMessage")
                  .beanRef("hl7handler", "badMessage")
            // end choice block- marshal the ACK/NAK back to the TCP endpoint

But this doesn't work either.  I'm just starting out with Camel, so I expect I simply haven't read the right bit of the documentation (which even the creators admit is one of Camel's weak points).

Another problem is controlling which component in the route returns a response to the originating endpoint (the MLLP sender, in the HL7 case).  I may have solved this one, but it's worth setting out the problem and my approach, in case it helps someone else.

My requirements are for the pipeline to log the inbound message (i.e. whatever is actually received on the HL7 socket), unmarshal the message to HL7 (HAPI), then validate the HL7 message type: at this stage in the pipeline I want to return a NAK to the sender (on the HL7 MLLP connection) if the message is not of the expected type, otherwise I want to write the message to a JMS queue for further processing and return an ACK.

The following route appears to work (note that I have omitted the message-logger stage):

        // We need the specific HL7 DataFormat object for unmarshalling HL7 ER7 from the
        // inbound MLLP link.
        DataFormat hl7Format = new HL7DataFormat();

        // Inbound HL7 comes from MLLP endpoint. 
        // Timestamp message, log receipt, unmarshal and perform initial
        // handling, which just determines if its the right kind of HL7
        // message.
        // If so, return ACK to originator and push message to JMS queue.
        // If not, return NAK to originator and end.
              .when( header("hl7.msh.messageType").isEqualTo("ORM") )
                    .beanRef("hl7handler", "badMessage")
        // The order message processor - pulls message from JMS queue and sends to
        // the processor bean, which creates and returns an ACK to the hl7 channel
        // if the message is of the expected type.
            .unmarshal().hl7().beanRef("hl7handler", "handleORM"); 

All of this routing can be expressed in XML instead of using the Java DSL. Once I'm comfortable with Camel I may try writing a NetBeans plugin or RCP application to create visuals tools for Camel routes.

Sunday, 4 January 2009