Getting started with Kestrel from a PHP application
We've been using Twitter's kestrel queue server for a while now at work, but only from our service layer, which is written in python. Now that we have some queueing needs from our application layer, written in PHP, I spent a few days this week adding queue support to our web application. I thought I'd share what I learned, and how I implemented it.
Goals
The kestrel server itself was pretty straightforward to get up and running. The only thing I would point out is that I recommend sticking to release branches, as master was fairly unstable when I tried to use it. Regarding implementing the client, there were a few goals I had in mind when I started:
- Since kestrel is built on the memcache protocol, try and leverage an existing memcache client rather than build one from scratch
- Utilize our existing batch job infrastructure, which I covered previously here, and make sure our multi-tenant needs are met
- Keep the queue interface generic in case we change queue servers later
- Utilize existing kestrel management tools, only build out the the functionality we need
With these goals in mind, I ended up with 4 components: a kestrel client, a producer, a consumer, and a very small CLI harness for running the consumer. But before I even coded anything, I set up kestrel web, a web UI for kestrel written by my co-worker Matt Erkkila. Kestrel web allows you to view statistics on kestrel, manage queues, as well as sort and filter queues based on manual inputs. Having this tool up and running from the get go made it easy to watch jobs get added and consumed from my test queue, and also easily flush out the queues as needed.
More on step debugging in PHP
Having recently wrote about PHP step debugging in VIM, I thought I'd share a couple of things I've come across since writing it.
First, I find that I often need step debugging when writing unit tests, as that's when I'm really scrutinizing logic. It turns out it's pretty easy to do. All you need is to set the XDEBUG_CONFIG environment variable like so:
XDEBUG_CONFIG="idekey=session_name" php bin/phpunit.php tests/FooTest.php
When debugging from VIM, you'll want to have two shells open. One running the debugger, and another executing the tests. Also, note that in a shared environment, you can still pass in xdebug ini settings like so:
XDEBUG_CONFIG="idekey=session_name" \
php -d xdebug.remote_host=10.0.0.1 bin/phpunit.php tests/FooTest.php
Second, for step debugging in VIM, I've settled on this plugin (there are many versions out there). It seems to solve problems with tab handling that I was seeing with older versions.
Cheers,
Bill
UPDATE: There's a new VIM plugin that's much better for step debugging. Check out the announcement here: http://joncairns.com/2012/08/vdebug-a-dbgp-debugger-client-for-vim-supporting-php-python-perl-and-ruby/
Preparing your application for serving static content efficiently
(Reposted from the Empower Campaigns blog)
Most web applications have static content such as CSS, JS, images, etc. To save on bandwidth costs and improve load times for your users, it's a good idea to tell the client to cache these items for a long period of time. But even though your static content changes infrequently, when it does, the cached content needs to be invalidated. In addition, high traffic sites often want to go a step further and take advantage of a CDN to offload static content delivery. Below I'll share the method we used here at Empower Campaigns to accomplish both of these things.
Organizing PHP Batch Jobs
This week at work I got the chance to address the growing number of batch oriented CLI scripts for our main web application. While they weren't quite unmanageable yet, they were heading in that direction. There was too much common code, especially with bootstrapping the application and parsing options. Also, the location of scripts didn't really make sense... ./bin/bar.php, ./cron/foo.php, etc. So I decided to carve out some time and clean it up.
The goals were pretty straight forward:
- Everything must use the application's model layer. This is mostly so that the built in caching will be consistent, but also to enforce that all data access goes through the same code.
- Centralize all CLI option parsing, application bootsrapping, error handling, and multi-tenant logic (this is a multi-tenant SaaS application)
- Keep the jobs themselves very simple.
With the above in mind, I ended up splitting things up into 3 parts:
PHP step debugging in VIM
When debugging problems in PHP, most of the time it's easiest to just add var_dump($foo); exit; in the middle of your script, and you can see the contents of $foo right in your browser. But if you have to do much more, this approach gets cumbersome pretty quickly. I've recently been using step debugging for harder to track down problems. It allows me to examine the state of things all the way through execution of a request, line by line, or skipping ahead to break points. This process also gives you more insight into everything else happening in a request, which can be useful when you're using frameworks or other 3rd party code in your application.
Follow me elsewhere
Github
LinkedIn
Facebook
TwitterCategories
- AppleTV (1)
- Architecture (1)
- Code (10)
- Digg (3)
- DNS (1)
- Foursquare (1)
- GitHub (1)
- House Repairs (2)
- Kestrel (2)
- Lean Startups (1)
- MariaDB (1)
- MySQL (2)
- PEAR (3)
- PHP (9)
- PHPUnit (3)
- Plumbing (2)
- PostGIS (2)
- SimpleGeo (1)
- Space (2)
- Technology (7)
- Travel (2)
- Unit Testing (3)
- VIM (2)
- Xdebug (2)
- Year In Review (3)
- Zend Framework (3)