Bill Shupp Software engineer, photographer, musician, space geek

Getting started with Kestrel from a PHP application

We've been using Twitter's kestrel queue server for a while now at work, but only from our service layer, which is written in python.  Now that we have some queueing needs from our application layer, written in PHP, I spent a few days this week adding queue support to our web application.  I thought I'd share what I learned, and how I implemented it.

Goals

The kestrel server itself was pretty straightforward to get up and running.  The only thing I would point out is that I recommend sticking to release branches, as master was fairly unstable when I tried to use it.  Regarding implementing the client, there were a few goals I had in mind when I started:

• Since kestrel is built on the memcache protocol, try and leverage an existing memcache client rather than build one from scratch
• Utilize our existing batch job infrastructure, which I covered previously here, and make sure our multi-tenant needs are met
• Keep the queue interface generic in case we change queue servers later
• Utilize existing kestrel management tools, only build out the the functionality we need

With these goals in mind, I ended up with 4 components: a kestrel client, a producer, a consumer, and a very small CLI harness for running the consumer.  But before I even coded anything, I set up kestrel web, a web UI for kestrel written by my co-worker Matt Erkkila.  Kestrel web allows you to view statistics on kestrel, manage queues, as well as sort and filter queues based on manual inputs.  Having this tool up and running from the get go made it easy to watch jobs get added and consumed from my test queue, and also easily flush out the queues as needed.