When developing a web application, it's common for your feature development to be ahead of any data administration tools you might need. While some frameworks, such as django, have built in admin tools for managing your application's data, many don't. At my current job we use Zend Framework, and have rolled our own lightweight model layer which does not have any fancy automatic admin tools. In addition to supporting database pools and selectors for partitioning of data, it also has transparent caching of data. While this is good for speed, it precludes you from making any data storage changes without going through this model layer or the cache will get out of sync. For now, rather than doing any DB queries directly, we do them through CLI jobs built for a specific task (batch updates, etc). This works fairly well, but often things come up that aren't yet supported in the application's admin tools, such as changing a user's status bits, etc.. Things that a CLI console would be useful for.
We've been using Twitter's kestrel queue server for a while now at work, but only from our service layer, which is written in python. Now that we have some queueing needs from our application layer, written in PHP, I spent a few days this week adding queue support to our web application. I thought I'd share what I learned, and how I implemented it.
The kestrel server itself was pretty straightforward to get up and running. The only thing I would point out is that I recommend sticking to release branches, as master was fairly unstable when I tried to use it. Regarding implementing the client, there were a few goals I had in mind when I started:
- Since kestrel is built on the memcache protocol, try and leverage an existing memcache client rather than build one from scratch
- Utilize our existing batch job infrastructure, which I covered previously here, and make sure our multi-tenant needs are met
- Keep the queue interface generic in case we change queue servers later
- Utilize existing kestrel management tools, only build out the the functionality we need
With these goals in mind, I ended up with 4 components: a kestrel client, a producer, a consumer, and a very small CLI harness for running the consumer. But before I even coded anything, I set up kestrel web, a web UI for kestrel written by my co-worker Matt Erkkila. Kestrel web allows you to view statistics on kestrel, manage queues, as well as sort and filter queues based on manual inputs. Having this tool up and running from the get go made it easy to watch jobs get added and consumed from my test queue, and also easily flush out the queues as needed.
I prefer managing my non-PEAR PHP dependencies (such as Zend Framework and PHPUnit) through PEAR channels when possible. It's much easier to manage component installation, upgrades, etc.. Managing and hosting a PEAR channel (using the Chiara_PEAR_Server) is not always that simple, and can add too much overhead for small projects. I like how Zend Framework leveraged google code to host its PEAR channel, and was able to update it simply through pushing to SVN, avoiding a web interface. Since I've been accumulating some components recently that really don't fit into pear.php.net, I figured it was time set up a PEAR channel for distributing them, but didn't want to host it myself or have to use a web interface. Turns out, it's really easy to do now.
An upcoming side project of mine requires the use of MogileFS and Zend Framework. MogileFS is an open source distributed file system, meant to scale up to many millions of files without a single point of failure. It's currently used by the likes of Digg and last.fm.
Though I was already familiar with using MogileFS clients in php and python at work, the operations team actually runs the servers. I wanted to get some experience with managing MogileFS itself (trackers, mogstored, mysql), and hopefully have a better understanding of how it all worked together. While I was already familiar with Zend Framework, I'd never used it to serve images. So, I figured I should build a quick prototype using both. But what to build? I decided to take inspiration from the notorious http://explosionsandboobs.com, but put my own spin on it. The result?