If you are easily offended, look away now …<\/p>\n
Reliable message queues (ActiveMQ<\/a> in particular) are pretty handy things. They make it a lot easier to build reliable systems which are able to network problems<\/a>, hardware trouble<\/a> and temporary weirdness<\/a>. However, they always feel pretty heavyweight; suitable for “enterprise systems” but not quick shell scripts.<\/p>\n Well, let’s fix that. My aim is publish and receive messages to an ActiveMQ broker from the unix shell with a minimum of overhead. I want to have a ‘consume’ script which reads messages from a queue and pipes them to a handler. If the handler script succeeds, the message is acknowledged and we win. If the handler script fails, the message is returned back to the queue, and can be re-tried later (possibly by a different host).<\/p>\n STOMP<\/a> is what makes this easy. It’s a ‘simple text-oriented message protocol’ which is supported directly by ActiveMQ. So we won’t need to mess around with weighty client libraries. A good start.<\/p>\n But we still need to write a ‘consume’ program which will speak STOMP and invoke the message handler script. There are existing STOMP bindings for perl and ruby, but I’m pitching for a pure unix solution.<\/p>\n In STOMP, messages are NUL separated which made me wonder if it’d be possible to use awk<\/a>, by setting its ‘record separator’ to NUL. The short answer is: yes, awk can do reliable messaging – win!<\/p>\n We’ll need some network glue. Recent versions of awk have builtin network support, but I’m going to use netcat because it’s more common than bleeding-edge awks.<\/p>\n I also want to keep ‘consume’ to be a single file, but I don’t want to pull my hair out trying to escape everything properly. So, I’ll use a bash here document<\/a> to write the awk script out to a temporary file before invoking awk. (is there a nicer way to do this?)<\/p>\n There’s not much more to say except here’s the scripts: consume<\/a> and produce<\/a>.<\/p>\n