Goals

The kestrel server itself was pretty straightforward to get up and running.  The only thing I would point out is that I recommend sticking to release branches, as master was fairly unstable when I tried to use it.  Regarding implementing the client, there were a few goals I had in mind when I started:

• Since kestrel is built on the memcache protocol, try and leverage an existing memcache client rather than build one from scratch
• Utilize our existing batch job infrastructure, which I covered previously here, and make sure our multi-tenant needs are met
• Keep the queue interface generic in case we change queue servers later
• Utilize existing kestrel management tools, only build out the the functionality we need

With these goals in mind, I ended up with 4 components: a kestrel client, a producer, a consumer, and a very small CLI harness for running the consumer.  But before I even coded anything, I set up kestrel web, a web UI for kestrel written by my co-worker Matt Erkkila.  Kestrel web allows you to view statistics on kestrel, manage queues, as well as sort and filter queues based on manual inputs.  Having this tool up and running from the get go made it easy to watch jobs get added and consumed from my test queue, and also easily flush out the queues as needed.

The Kestrel Client

I couldn't find any existing kestrel clients for PHP, so I started looking at the two memcache extensions: the older memcache, and Andrei Zmievski's memcached, the latter of which is based on the libmemcached library.  I started with memcache, and while it worked fine initially, I quickly found that I could not modify timeouts.  This interfered with the way  kestrel recommends you poll it for new jobs, and I would see timeout errors from the memcache extension if you tried to set the poll timeout to 1 second or higher (the memcache default).  The memcached extension does not have these issues, so I went with it.

The first gotcha I ran into was serialization.  You can use memcached's serializer for writing to kestrel, but when it reads the data back, it doesn't recognize that it is serialized.  So I just serialize the data manually in my client, and things work fine. One other thing to note is that you'll want to disable compression, or do it manually, as the memcached extension will automatically compress anything over 100 bytes by default, and will not decompress it when reading from kestrel.

The other issue is that if you want to use any custom kestrel commands, you can't.  Since the application layer doesn't need anything fancy, the memcached extension will work fine for it.  Once we need support for the upcoming monitor (batching) in kestrel 2, we may need to implement a kestrel client from scratch.  Kestrel web supplies everything else we need right now.

Once the decision was made to use memcached, I wrote a light decorator for it, EC_KestrelClient.  This handles instantiation of the memcached client, serialization, and helpers for some kestrel specific options to the GET command.  It also has support for passing memcached specific options through it.  The class ended up looking like this:

The Producer

The producer is very simple.  It just formats the data into a standard structure, including current tenant information, namespaces the queue so it doesn't collide with other projects, and adds it to the queue.  The producer looks like this:

The Consumer

The consumer has a bit more to it, though still pretty straightforward.  It's intended to be run from a monitoring tool like daemontools or supervisord, so there is a very small CLI harness that just passes the CLI arguments into EC_Consumer and runs it.  After parsing the CLI arguments, EC_Consumer polls kestrel for new jobs, and runs them through our standard batch job infrastructure.  Until we have more confidence in PHP's long running process ability, I added an optional maxium jobs argument, which will stop the consumer from processing more than X jobs and then terminate.  The monitoring service (supervisord) will then just restart it in a matter of seconds.  I also added an optional debug argument for testing, so you can see every action as it happens.  The CLI harness looks like this:

And the main consumer class, EC_Consumer, looks something like this:

Putting it together

Now that all the pieces are put together, let's take a look at in action. Adding example job "HelloWorld" to the queue "hello_world" from within our application looks something like this:

And finally, here's an example of running the consumer from the CLI harness, along with some example debug output of processing the job:

That's it! I'd be interested to hear how other folks are interfacing with kestrel from PHP.

First, I find that I often need step debugging when writing unit tests, as that's when I'm really scrutinizing logic.  It turns out it's pretty easy to do.  All you need is to set the XDEBUG_CONFIG environment variable like so:

XDEBUG_CONFIG="idekey=session_name" php bin/phpunit.php tests/FooTest.php

When debugging from VIM, you'll want to have two shells open. One running the debugger, and another executing the tests. Also, note that in a shared environment, you can still pass in xdebug ini settings like so:

XDEBUG_CONFIG="idekey=session_name" \
php -d xdebug.remote_host=10.0.0.1 bin/phpunit.php tests/FooTest.php

Second, for step debugging in VIM, I've settled on this plugin (there are many versions out there).  It seems to solve problems with tab handling that I was seeing with older versions.

Cheers,

Bill

Most web applications have static content such as CSS, JS, images, etc.  To save on bandwidth costs and improve load times for your users, it's a good idea to tell the client to cache these items for a long period of time.  But even though your static content changes infrequently, when it does, the cached content needs to be invalidated.  In addition, high traffic sites often want to go a step further and take advantage of a CDN to offload static content delivery.  Below I'll share the method we used here at Empower Campaigns to accomplish both of these things.

Version your content

The first step to getting control of cache invalidation is to manage the version of your content, usually included as part of the URL.  In the past I've seen things like incrementing a single counter for your JS and CSS files that effectively changed the URL of these files and invalidated the cache.  But this updates the URL for ALL of the files that use that counter, which is not very efficient.  A better approach is to version each file (or bundle, if you deliver files in groups) so that you control the cache invalidation of each item individually.  One way to do this is with a hash like CRC32.  If the content is stored on a file system, such as JS, CSS, or image files often are, you could hash the file upon the first read, and store that hash in a cache that gets purged during a production code push.  For PHP users, APC is a great tool for this.  This reduces the hashing overhead to one time per web node in your cluster per code release.  If your content is stored somewhere outside of your code base, you might consider storing the hash of the content at write time.  For example, when I worked at Digg, we stored user profile images in MogileFS.  So when a user added or changed their profile image, we would hash the image at that time and store the hash with that user's profile data in Cassandra.  This was even better because the content was hashed only once.  The same type of thing could be done with locally stored content, it just depends how fancy you'd like to get.  Note that when constructing your versioned URL, avoid using the version in the query string, as some CDNs strip query strings (ex: media.yourdomain.com/v/8bba8b90/css/960.min.css rather than media.yourdomain.com/css/960.min.css?v=8bba8b90)

Minify/Compress your content

In a production environment, you want to deliver static content as quickly as possible.  Two ways to help with this is minification and compression.  For minification, we use Google's Closure for JS, and YUI Compressor for CSS, both of which are run during the deployment process.  You'll also want to enable compression in your web server as well, such as mod_deflate in Apache, to make the files as small as possible during transit.

Increase cache time

Once you have your content versioning in place, you'll want to increase the cache time for those files.  We use a cache time of one year, and set that at the web server level like so using mod_rewrite:

Use a dedicated domain name

If you want to be ready to move to a CDN at any time, consider putting your static content on a dedicated virtual host, such as media.yourdomain.com.  This way, you can simply point that domain to a CDN at any time, and you'll be up and running.  Also, this has the added benefit of allowing the client's browser to make more parallel requests for your content, as they often limit the number of parallel requests they'll make to any one domain.  By spreading your content over more than one domain, the client can make more requests in parallel, speeding up the the time for the document to be ready.  One other benefit is that by using another domain for your media, you can reduce the number of cookies sent with media requests, which reduces the size of each request. Our rewrite rules for ignoring version numbers in our media host ended up looking like this:

With the above steps in mind, we wrote a view helper that handles hashing and url writing called mediaURL().  It can also look at our config and determine if SSL URLs should be used, if a given page in our application requires it.  The view helper looks like this:

And the call to the helper from our view looks like this:

Conclusion

By following the steps above, you'll have static content that is dynamically versioned, compressed, and cacheable by the client.  And if you're serving it on a dedicated domain, you'll speed up overall page load times and be ready to provision a CDN if you ever need it.

Cheers,

Bill Shupp, Lead Developer (twitter: http://twitter.com/shupp)

The goals were pretty straight forward:

• Everything must use the application's model layer.  This is mostly so that the built in caching will be consistent, but also to enforce that all data access goes through the same code.
• Centralize all CLI option parsing, application bootsrapping, error handling, and multi-tenant logic (this is a multi-tenant SaaS application)
• Keep the jobs themselves very simple.

With the above in mind, I ended up splitting things up into 3 parts:

• ./bin/ecli.php, the only PHP CLI script which does minimal bootstrapping, collects the options/arguments, and runs EC_CLI
• EC_CLI, a class that does the actual option parsing (via Zend_Console_Getopt), application bootstrapping, error handling, and multi-tenant runs (recursively calls ./bin/ecli.php for each tenant), and runs the actual job
• EC_Job_Abstract, the Job interface that collects job specific arguments and runs the job.

Now, when I need to create a new batch job EC_Job_Foo, I just put the contents in EC_Job_Foo::run(), and execute it like so:

./bin/ecli.php --environment production --instance clienta,clientb --job Foo

The ecli.php script ends up looking like this:

Pretty easy, huh?  It makes testing all the components really easy as well.

Finally, we made the decision to use Jenkins instead of the traditional cron scheduler for recurring jobs.  This has a few advantages: It's got a simple web interface for managing the job, you plug it into your existing Jenkins notifications (email, jabber, etc), console tailing, job dependencies, etc... It works really well.

With PHP, the common extension for this is Xdebug.  (See the remote debugging documentation here).  Since my main text editor is VIM, I had to find a suitable plugin to talk to Xdebug.  It turns out there are a few good blog posts already on this topic, so I won't duplicate the information here.  This post does a great job of covering installation and usage of the Xdebug plugin for VIM by Seng Woo Shin, though it links to an older version of it.  Here's a link to a newer one that Xdebug also links to.  The Xdebug site links to another tutorial that's worth a read too.

Beyond the links above, I've discovered a couple of things that are worth sharing.  First, there's a new FireFox extension, easy Xdebug, that is a real lifesaver.  You just click the green bug on the bottom right, and it will try to start an Xdebug session with any request (rather than having to add XDEBUG_SESSION_START=1 to the query string parameters myself).  This is particularly handy when dealing with POST or or AJAX requests.  The Xdebug docs also link to similar tools for Chrome, Safari, and Opera.

Now, my debugging work flow looks like this:

• Start the Xdebug session in FF by clicking the green bug
• Set breakpoints in VIM
• Press F5 in VIM to start listening for the request (it waits 5 seconds)
• Perform that action I want to debug in FF
• Press enter in VIM to enter the debug session
• Debug
• Press F6 to exit debugger, and fix the problem

The one gotcha that I keep running into is how to set breakpoints correctly.  When you use this VIM plugin, you must put a breakpoint on executable code.  If you put it on a blank line, or even on a line with just an opening try statement, your breakpoint will not work.  Instead, you'll end up with an "error 5".  This topic is briefly covered here and here.

Now, when I find myself using more than one or two var_dump() statements to debug something, I usually walk through the work flow above, and have things sorted out much more quickly than I used to.

Hosting using GitHub Pages

Since I prefer GitHub to Google Code for hosting my open source repositories, I decided to take a look at their new Pages feature for project web hosting.  It's super easy to set up.  You can host content at http://username.github.com by creating a repository username.github.com (master branch), and can also host project related content at http://username.github.com/project by creating a top level gh-pages branch in the project.  I was up and running in minutes.  See here for specific instructions on setting it up.

PEAR Channel Serving with Pirum

I wasn't sure what the ZF guys were using to manage their channel, so I started by looking at PEAR2's SimpleChannelServer.  While it looks promising, and you can even manage it directly from pyrus, I ran into enough errors that I thought I'd take a look Pirum, another simple channel server written by the Symfony project maintainer.  It worked for me right out of the box.  Setup was as simple as:


pear channel-discover pear.pirum-project.org
pear install pirum/Pirum
cd ~/git/pirum
# create pirum.xml file
pirum build .
# commit changes and push to github


So now releases through my PEAR channel look like this:


# update PEAR package.xml, then run:
pear package
cd ~/git/pirum
pirum add . ~/git/deneb/Deneb-0.3.0.tgz
# commit changes and push to github


Now I've got an easily managed PEAR Channel that I don't have to host:  http://shupp.github.com/pirum

When developing an application, one of the keys to maintaining high productivity and quality is to make sure that you not only have the right tool set, but that you have easy access to it.   With web applications, the usual needs are things like running Selenium tests, running unit tests and checking code coverage, profiling, and having links to documentation.  Though I've talked previously about the cost of maintaining 100% code coverage, I'm convinced that it's worth it if you not only get proficient at the tools, but make them very easy to use.  So when I started as the first developer here at Empower Campaigns a couple of months ago, I wanted to make sure these needs were addressed early on, before we had a large development team.

Inspired by the work that had been done at my previous job by Ian Eure and Matt Erkkila, I decided to start with the following:

• Use the tried and true GNU Make for building environments and executing tools
• Have a "build" virtual host in every development environment for accessing the tools' web resources

Using make, getting our project "enterprise" (we use space shuttle themed project names) up and running is as simple as this:

# clone into a directory named with your vhost name:
git clone review:enterprise.git enterprise.bshupp.dev.em.power
cd enterprise.bshupp.dev.em.power
# install into apache
make clean install

This not only provides a working application (or "shuttle" as we call them) at enterprise.bshupp.dev.em.power, but also a "build" virtual host for access to web resources at build.enterprise.bshupp.dev.em.power. Here's a screen shot of the initial build "dashboard", which provides navigation on the left, and some environment details and instructions on the main page.

Since our web tier is in PHP, the most common tool we need to run is PHPUnit, which you run via make test.  You'll see the standard output in your terminal like so (with some nice colors indicating the status - green for success and high coverage, yellow for success and medium coverage, and red for failing tests).

In addition, to view coverage details, you go to the "code coverage" link your build dashboard.

make test also generates TestDox output to see which tests are failing.

We keep a Selenium Remote Control (RC) server running on our development system for easy access by developers.  Want to see if you've broken the selenium tests?  Just run make selenium-smoke or make selenium-full, and Selenium RC will talk to your development environment.

And since developers need to run make test selenium-smoke phpcs to validate before any push to avoid breaking a build, I've wrapped them up into make validate for convenience.  We actually use these shortcuts with Hudson, which allows us to direct Selenium RC to different staging environments, or modify build steps without re-configuring Hudson.

Want to read the inline documentation in a web browser instead of your text editor?  Just run make phpdoc, then click on the "PHPDocs" link in the build dashboard.

One thing that's particularly useful is easy profiling with Facebook's XHProf.  Just enable profiling in your environment via make enable-profiling, perform the action in the application you want to profile, then open up the "XHProf Profiler" link in your build dashboard.  There you'll see a list of "runs" which you can drill down into and see exactly how each is performing.  When you're done, just run make disable-profiling.

To manage XHProf profile runs, I save them by environment, and wrote a helper that allows you to browse runs by date, and then "clean" them out when you're done.  This works easily in a shared xhprof.output_dir by namespacing the stored runs according to environment name.  Here is the example snippet for starting profiling and saving as runs:

And you can checkout the full code over on github.

If you want to get statistics on or content from APC, that's available too through the inspector that ships with the extension:

Lastly, I've also included links in the dashboard to all the tools we use: our Confluence wiki, Gerrit, gitweb, and Hudson.

This system makes it really convenient to deploy your development environment, access documentation, profile your code, run test tools and analyze results.  It's my hope that as our development team grows, it'll be relatively painless to maintain high quality, well tested code.

Cheers,

Bill Shupp, Lead Developer (twitter: http://twitter.com/shupp)

A broader PHP tutorial for using their API is available in their documentation, but for the impatient, here's a quick install of their PHP client.  It assumes you already have a working PEAR installation:


pear install HTTP_Request2-alpha
pear install HTTP_OAuth-alpha
pear install Net_URL2
git clone git://github.com/simplegeo/Services_SimpleGeo.git
cd Services_SimpleGeo
pear install package.xml

And here's the actual example in PHP, which you can run from CLI:

When running it, you should see progress:


$ php sg_4sq_example.php
Sending batch 1
Sending batch 2
Done!


Once it's complete, go to your SimpleGeo dashboard, click on the the explorer tab for the layer you're using, and you'll see something like this, which is a history of my check-ins in San Francisco:

My local foursquare checkins on SimpleGeo

I only ran into a couple of issues with getting this working in PHP.  First, getting the right syntax for extracting the coordinates from the RSS feed took some time to get right.  I first tried with PEAR's XML_Feed_Parser, but ended up just using SimpleXML directly.  Second, the blog example uses raw guid values, which break the API call when they contain a '/'.  Unfortunately, the error returned by the API was not that useful: 'dict' object has no attribute 'startswith'.  Once I noticed the slash in the value, wrapping the guid in base64_encode() fixed the problem.

Perhaps you'll find this useful, it was fun to finally use the SimpleGeo API!

Below are some things I keep in mind when coding and testing. I thought I'd share them here in case anyone else finds them useful.

Think about testability when you write code, not when you write the test

This may not be new to you, especially if you're familiar with dependency injection or test driven development.  Your unit tests should be able to run in a vacuum:  No database access, no memcache access, no headers can be sent, etc.  (Sure those things can be tested separately, but should be a small subset of your code, not the overall application logic.)  To do this, make sure you abstract those dependencies.  For example, instead of instantiating a cache driver in the middle of your method, make a getter for it which you can mock:

If you do this abstraction when you code, there's less refactoring to be done when writing your tests.  Also, if your dependency doesn't make a network call upon instantiation, you can still test the getter using assertType().  Memcached's addServer() is an example of such a dependency - no connection is made until you do an operation.  If this is not the case, and you can't properly test the getter, you probably want wrap the getter in @codeCoverageIgnore annotations like so:

It doesn't make sense for code you never intend to test to count against your coverage percentage.

Keep documentation handy

PHPUnit is a big framework.  There are a lot of things to remember, including all the assertion methods available, data providers, and mocking objects, among other things.  Rather than always copying some other example, or bugging your co-worker, spend an hour or two, and RTFM.  You'll be better off for it.  A co-worker of mine kept a printout of the PHPUnit_Framework_TestCase::getMock() documentation (probably the hardest to remember) taped to his monitor.  I prefer delicious bookmarking, but to each his own. Though I've been using PHPUnit for a couple of years now, I still refer to the documentation regularly.

Don't worry about test code re-usability

As good OOP programmers, we are often pre-occupied with writing elegant, re-usable code, even in tests. I have recently given up on this in my unit tests, mostly due to scope issues in PHPUnit which I'll describe later. Rather than try consolidate re-usable code, I'm more likely to copy and paste chunks throughout my test and know that they will work. I think the goal of tests is to write useful assertions, not necessarily elegant, re-usable components. Why do I care how many lines are in my test code? You might be thinking "Doesn't that make your tests harder to maintain?" Maybe so, but in the big picture, I don't think it adds much time, and once you've fixed something in one test, then you can quickly find it elsewhere. It might not be pretty, but it is faster, and more pragmatic, at least until the tools are better.

Mocking scope issues

As I mentioned above, I've found (and reported) some strange behavior I've seen with scope and mocked objects.  One pattern I see people use is a getMocks() helper in a test, which returns an array of commonly used objects, pre-mocked with the most used defaults already set.  Sounds like a reasonable approach to testing a large class with lots of dependency needs, right?  Wrong.  What I've found is that many times (and my test results have been inconsistent, unfortunately), you cannot modify the return value of a mocked method after its expectation has already been set.  No warning or error is thrown when you do this, but the original mocked value is returned instead of the second value you assigned.  Re-mocking return values does seem to work when it's used for throwing exceptions, though.  It appears to be a scope issue, but again, my tests have not always been consistent.  The usual result is that you spent an hour pulling your hear out trying to figure out why your test is failing.  I eventually said screw it, and just never set default behaviour for my mocked methods.

I now use a pattern where I have getters in my test for each mocked dependency, which mock each method I may need, but don't set any initial expectations yet.  I'll then pass them all into the getter for the main class I'm trying to test, which attaches them for me.  This way, on a per test basis, I set the expectations of each dependency as needed. This seems to work much more reliably:

Meaningful assertions

Another complaint I hear is that, especially when enforcing code coverage rules in your development shop, developers focus more on the required coverage than on the quality of the tests.  It's important to think about what it is you're testing.  If you just call a method to make sure you get your code coverage, you're doing it wrong.  There are lots of assertions you can do:  assert the type of the response, assert the content of the response, assert the message of the exception  thrown, the exception code, the type of exception, how many times a mocked method was called, etc.  But you want to be smart about it as well.  We had one case where the original test for a Form framework was testing the exact template output.  Since there was markup and JS in the ouput, every time a designer or JS developer modified it, the test would break.  After a few breaks, I modified the test to use DomDocument parsing, and make sure the relevant form ids and attributes were present, which is really what we wanted to assert.  If you don't write meaningful tests, don't bother writing them at all.

Useful tools

Make sure it's easy to write and run your tests.  PHPUnit has options for generating skeletons, for example, to get you started quickly.  You can use --skeleton-class to generate a class from your test (if you're practicing TDD), or the reverse, --skeleton-test, to generate a test from your class.

Running tests should be easy too.  One thing to keep in mind is that your include path must be correct.  For example, when I'm developing a PEAR package, I may have a previous version already installed, but am developing in my repository checkout.  I need to make sure my local files are not only in my include path, but that they are at the beginning.  A handy wrapper like this should allow you to always run your development code easily:

If I put this in the root of my repository, I can then call it like so:

php runTests.php --coverage-html coverage OpenID_AllTests

(If this is a PEAR package, you'll want to ignore runTests.php when generating your package.xml file.)

At Digg, our development environment tools are based on make, and it's super easy to run not only your PHPUnit tests and coverage, but also chain together running PHP_CodeSniffer for coding standards compliance:

make coverage phpcs

Facebook has a good post about hiring in general, which includes a section on writing good tools for your team.  This idea certainly applies to your testing tools. The easier you make it for your team to write good unit tests, the better.

Conclusion

Unit testing in PHP is not that easy, in my opinion (compared to python, anyway).  Until you get good at it, that is, and learn to work around quirks in the tools. Then it's really not that hard. But everyone  values unit testing differently.  If you choose to invest in testing your code, perhaps you'll find some of the above useful. I used to not think much about unit testing, until I had a few experiences where unit tests caught problems in my code before I had run into them myself. At this point, I think unit testing my PHP code 100% is worth it, especially now that I can do it quickly and efficiently.

Though I was already familiar with using MogileFS clients in php and python at work, the operations team actually runs the servers.   I wanted to get some experience with managing MogileFS itself (trackers, mogstored, mysql), and hopefully have a better understanding of how it all worked together.  While I was already familiar with Zend Framework, I'd never used it to serve images.  So, I figured I should build a quick prototype using both.  But what to build?  I decided to take inspiration from the notorious http://explosionsandboobs.com, but put my own spin on it.  The result?

http://boobsandkittens.com

Bear in mind that the above URL is on my inexpensive Slicehost VPS, and can be slow at times.  This is a really simple application that allowed me to do a few things:

• scrape some content from google images search
• store them in MogileFS
• render the stored images through ZF

The source is available here.  The scraper is pretty straight forward.  I just called it from the command line to save images to a directory.  Once saved to disk, I used the mogrify tool from Imagemagick to scale the images down to a height of 400 px max, and visually removed the worst of the images (it's relatively SFW).  Next, I inserted those images into MogileFS with insertImage.php.  There are only two real action controllers, the index page and image renderer.  The latter was really the only thing to learn on the php side of this project.  The key there was disabling the layout and view rendering, as well as setting the appropriate image information in the response object (content-type, content-length, and the content itself).  Since most of the logic is in the model, the action controllers are pretty skinny.  I used memcache to store db queries and the images from MogileFS to help offset the performance of a cheap VPS.  Since Zend Framework doesn't ship with a MogileFS client, I used the PEAR one.

That's it.  Enjoy!