Welcome to my third annual Year in Review!  2011 was the year of NASA Tweetups for me, attending 4 in total (plus a JPL open house), including the launch of the Juno mission to Jupiter.  It was also a year of some firsts including buying an electric car and a hot air balloon ride.

Juno Mission ready for launch!

• January
• I started out the year continuing to work as lead developer for Empower Campaigns.
• February
• I was fortunate enough to attend the NASA Ames Tweetup on "Planet Hunting".  There I heard from a number of great speakers, but the standout one to me was the Kepler Mission's co-investigator Natalie Batalha.  Not only is Kepler a fascinating mission (that ended up having a stellar year), but Batalha is a remarkably passionate and inspiring speaker.  The event also included a tour of the research center, which was fascinating.
• I also wrote about PHP step debugging in VIM.
• March
• I wrote about organizing PHP batch jobs and also serving static content efficiently from an application.
• March also included my attendance of the NASA Sun Earth Day Tweetup at Goddard Space Flight Center.  It was a blast to tour Goddard!  Since the event was in the the DC area, I was able to visit family and attend my youngest niece's second birthday party.
• The month wrapped up with a trip to New Orleans for the state-side wedding reception of my sister in law, who was married in New Zealand the previous November.
• April
• On Easter I made this.  You're welcome.
• May
• In early May I decided to take the plunge in electric cars and reserved a Nissan Leaf.  The wait was expected to be 4 months.
• Though it wasn't a NASA Tweetup, I did make it down to JPL's annual open house in mid May, which was a bit crowded, but still spectacular.  I got to see the Mars rover Curiosity in a clean room before its launch the following November.  I also got to walk through the Space Flight Operations Center.
• I wrote about working with Kestrel from a PHP application since there didn't seem to be any other documentation on the topic.
• We also ran into some further plumbing issues at home that month, but decided to tackle the problem ourselves.
• June
• In June my wife and I celebrated our 4th wedding anniversary!
• We also documented another small home repair, replacing the garbage disposal.
• July
• July included a second trip back to VA, this time just to visit family.
• Unfortunately, soon after that we got the bad news that one of our beloved cats, Linus, suffers from chronic kidney disease.  He's currently doing well at a stable weight (roughly 3/4 of his previous weight) through medication and diet, and we're hoping to keep him as comfortable as possible.  There is not a known cure.
• I also finally broke down and bought my first serious telescope, a Celestron NexStar 8SE.  It's pretty rad.  I'm hoping to get into astrophotography this year.
• August
• In August I was lucky enough to attend the launch of NASA's Juno mission during my 3rd tweetup of the year.  We heard from lots of great folks on the Juno teams as well as NASA Administrator Charles Bolden, who was a really inspiring speaker.
• Later that month, a book that myself and a few other former Digg employees contributed to nearly 2 years before  finally came out.
• That month I attended my first TEDxNASA event "Extreme Green", which was a blast.  Lots of great speakers.   One of the more impressive talks of the day was Ed Lu's talk about how inexpensive it really would be to set up an asteroid monitoring system to help prevent a major collision, perhaps the worst natural disaster imaginable.
• Another TED talk on 30 day challenges inspired me to try cutting out Diet Coke (and all sodas) for 30 days.  Four and a half months later, I still haven't gone back.
• September
• I bough my first 100% electric car, a Nissan Leaf, which was delivered right on schedule.  The car is such a dream to drive.  It's nearly silent.  I can't wait until electric vehicles are the norm!
• Later that month I wrote about speeding up satellite predictions in PHP with hiphop which coincided with my release of Predict, a library for modeling and predicting satellite orbits.  The use case for this is a project of mine called "Where the ISS at?", which is currently in the beta testing phase and should be public very soon.
• October
• October included attending my final NASA Tweetup of the year, the SOFIA Telescope Tweetup back at NASA Ames.  We got a chance to tour the telescope, which was parked there at Ames.  It too was pretty rad.
• November
• In November we decided to take a third trip back to VA for Thanksgiving, and I finally got a chance to visit Monticello.
  

  Hot Air Balloon Ride in Sonoma!

• December
• December saw us both reaching our 40th birthdays.  For some reason, I kind of dreaded this milestone.  But once I reached it, I was actually pretty relieved.  It seems the anticipation of turning 40 was much worse than the actual birthday.  The good news is that my wife arranged our first ride in a hot air balloon up in Sonoma to celebrate.  Once you got used to the sound and awesome heat of each of the 15 million BTU burners powering the balloon, the views were nothing short of breathtaking.
• In December I also accepted a job as Engineering Manager at OpenDNS, something I'm very excited start in the beginning of January.

ISS flyover data plotted on Google maps

Background

Seeing something in space fly overhead with the naked eye is pretty awesome.  It's not only a technical marvel to enjoy, but for me it helps keep things in perspective and my thoughts on the big picture.  Ever since the first time I saw the International Space Station fly by, I've felt that it is something everyone should experience.   I started looking at existing software to help remind me (and others) when the next pass of the ISS would be, and while there are a few good websites out there for satellite prediction, none of them did exactly what I wanted.  So a few months ago I decided to try and build my own.  The biggest piece of my project was going to be getting the actual satellite data.  So I looked around for existing data APIs that I could use, but couldn't find anything other than scraping existing sites (which is not only fragile, but some have no published data policy).  So, the first step was to figure out how to to generate the predictions myself.

How satellite predictions work

The process involves calculating the coordinates (position and velocity) of the observer, the satellite, and the sun at a given point in time using a common coordinate system called Earth Centered Intertial (ECI).  Then, you need to convert those coordinates to a topocentric-horizon or azimuth-elevation coordinate system that can be used to pinpoint the object in the sky.  But before you can do this, you first need to know the orbital path of the satellite.  NORAD, which has responsibility for tracking all man-made objects in orbit around Earth, uses the the Space Surveillance Network for collecting observational data of these objects.  It then distributes these observations as Keplerian orbital elements in the  Two Line Element Set (TLE) format.    (You can download these very small TLE files through sites like Celestrak).  The TLE data can then be used to  predict a satellite's position and velocity using Simplified Perturbations Models (SGP/SGP4/SDP4).  (NOTE: Since these models trade computational speed for accuracy, you need to update your TLE files every few days to make sure your predictions are accurate).  Dr. T.S. Kelso, who runs Celestrak, has a great series of columns  explaining all of this in detail.  I highly recommend them.

The library and performance problems

I couldn't find a suitable  or mature library in python or PHP implementing the SGP models, so I decided to port code from Gpredict, a real time satellite tracker application written in C.  Gpredict had almost everything I needed, the SGP model implementation, the prediction logic, etc..  The only thing I needed to add was the ability to determine the apparent magnitude (brightness) of an object, and I'd be set.  Though it took a few weeks, I finally got a web friendly PHP port of Gpredict working that produced prediction data which matched not only what the Gpredict application produced, but what other websites produced as well (including apparent magnitude for the ISS that I added).  While generating a snapshot or single pass was fast enough for my needs (under 50 milliseconds), generating predictions for the next 10 days was excruciatingly slow - in the order of several seconds.  After doing some profiling, I found there really wasn't much left to optimize.  PHP just wasn't fast at doing a LOT of math.  So this left me with a few options:  put the code into a php extension (which would be a fair amount of work), try a different language implementation of the SGP models (such as porting the Gpredict code to a web service, or creating a web service out of existing java code), or try Facebook's PHP to C++ transformation tool, HipHop for PHP.

HipHop for PHP to the rescue

I decided to try HipHop first since it would likely be the least amount of work and accomplish the biggest performance boost.  And though setting it up took some time to get right, the difference was very impressive.  A 10 day prediction that took 1866 milliseconds on my macbook pro in PHP took only 221 milliseconds when transformed and compiled with HipHop.  That's 88% less time!  Initially I was concerned that, since HipHop does not yet support PHP 5.3, I might have to do some refactoring.  But luckily only a few lines needed changing.  Further, HipHop does a great job of pointing out problems in the code.  Code that it doesn't understand (such as newer PHP 5.3 only code), as well as code that does not execute, or other general problems, are clearly logged during compile time.  It found a couple of bugs from my port of Gpredict that I hadn't run into yet, which was really helpful.  But as I mentioned before, HipHop for PHP is not terribly easy to set up.  It took me several hours from start to finish to get it installed and a program compiled and working.  Here's a quick list of things I ran into and you might want to watch out for:

• HipHop for PHP doesn't work on Mac OS X, only Linux or FreeBSD.  If you run a mac, install Linux via VirtualBox or another VM tool and work there.
• It won't compile on 32 bit Linux, only 64 bit is supported.
• The README file is not up to date (it even says so).   Use the wiki pages on github instead.
• The wiki includes some installation instructions for common distributions.  Do yourself a favor and use them.  There are also wiki pages covering how to use HipHop, which you'll need to read as well.
• Make sure you have sufficient memory if you're running your compiled php program as a daemon.  I was using a small 512MB VPS, and it would routinely crash until I upped the memory to 1GB.

Had I known the above tips in advance, I probably could have done my install and had a program up and running in less than half the time that it took me.  But still, I'm pretty impressed with HipHop.  I have it running as a daemon locally, and make HTTP requests to it only for the long running prediction processes.  One cool thing is that I can return serialized php from the HipHop web service, deserialize the results in the client PHP process, and continue using those objects.  It made moving the predictions process from local to HipHop  a trivial change in my existing application.

Conclusion

Learning about how satellite tracking works and how to track them myself has been a blast.  But running into such a performance problem after spending time porting the prediction code to PHP was disappointment.  However, HipHop for PHP was a perfect fit for solving this, and I'm happy with how well it works for my needs.

The application I set out to build isn't public just yet, but I did put up a prototype of real time ISS tracking using my newly ported Predict library, which you can view here.  Since I was new to satellite tracking, I needed some help understanding what needed to be done.  I would like to thank a few people who graciously lent their time to this process: David Lowe for helping to steer me in the right direction from the beginning (and putting up with a lot of questions), Dan Copeland for helping work through the math for calculating apparent magnitude as well as being a helpful second set of eyes, and T.S. Kelso for not only running Celestrak, but for being responsive and helpful in answering my questions and pointing me to additional information on the topic.

Cheers!

The new and improved in-sink-erator, with sound insulation

Ever since we moved into this place over three years ago, our garbage disposal has been a bit noisy, and more recently has started to smell bad.  The model we had, the In-Sink-Erator Badger 5 Plus, was not easy to clean since the black rubber flap at in the drain was not removable.  Since the disposal was pretty old, we decided to replace it.

The old noisy, stinky Badger 5 Plus

The disposal we chose as a replacement is the In-Sink-Erator Evolution Space Saver w/ sound insulation.  The sound insulation is partially achieved by using a rubber connector between the waste tube and the disposal, rather than connecting it directly to the unit.  Further, the rubber flap in the sink is not only removable (and thus easier to clean), but it's designed to absorb sound coming into the sink with it's sharp angles.  Pretty slick.

Installation would require plumber's putty and adjustable pliers, which I picked up as well.  The whole operation only took about an hour and a half, and probably 30 minutes of that was spent getting the old flange separated from the sink.  It also assumes you are replacing an existing disposal, as you have to wire the old power cable to the new disposal.  It turns out I didn't need the adjustable pliers, as our plumber recently replaced the trap with one that can be tightened by hand.  (The old disposal initially was installed BELOW the height of the drain vent, allowing water to sit in the disposal itself rather than the trap.  Our home inspector caught this, so we had the drain lowered by our plumber).  One thing I did learn, though, is that it's a bad a idea to leave your instructions under the trap pipe when you disconnect it.  There's water in there.  By design.

Disposal removed. You can see the original drain above the water valves, the lowered one below.

Once installation was complete, we were quite pleased with how quiet the new disposal is - a big difference.  And I gotta say, these kinds of do it yourself improvements are kind of fun to do.

The new quiet, odor-free Evolution Space Saver

When developing a web application, it's common for your feature development to be ahead of any data administration tools you might need.  While some frameworks, such as django, have built in admin tools for managing your application's data, many don't.  At my current job we use Zend Framework, and have rolled our own lightweight model layer which does not have any fancy automatic admin tools.  In addition to supporting database pools and selectors for partitioning of data, it also has transparent caching of data.  While this is good for speed, it precludes you from making any data storage changes without going through this model layer or the cache will get out of sync.  For now, rather than doing any DB queries directly, we do them through CLI jobs built for a specific task (batch updates, etc).  This works fairly well, but often things come up that aren't yet supported in the application's admin tools, such as changing a user's status bits, etc.. Things that a CLI console would be useful for.

I was playing with django's shell recently, and I realized that with the way we have our bootstrapping setup, it's actually pretty trivial to get full access to our data models from PHP's interactive shell.  Because we have multiple entry points to our application (primary entry point, media entry point, testing from CLI, etc), we already consolidated the generic bootstrapping into an init.php file that is used by all of the entry points.  This file contains most of the usual things you'll find in a Zend Framework application's default index.php, such as setting up environment definitions, setting the include path, setting up autoload, etc.  The only thing the entry points do really is instantiate the application and run it.  With our current setup, I can start the interactive shell, require the init.php file, and instantiate and bootstrap the application.  From that point on, I have full access to the application's models, and any changes will keep caching in sync with its data.

Here's an example of changing my password from the console:

Here's an example of looping over a collection of users and modifying their status bits:

The interactive shell supports tab completion of file names for things like require_once, which is nice. It also has history support between sessions, so you can use the up arrow to easily navigate to previous lines you've typed. Pretty handy!

Last week our drain in the garage got stopped up during a washing machine cycle, spilling water onto the floor. While the dishwasher was also affected, the rest of the house (kitchen sink and bathroom) were unaffected. This is good news since just a few months ago we had some major plumbing work done.  It meant that the sewer line was clear, and the blockage was somewhere in our internal pipes.  We decided to try out a new plumber since we had some minor issues with the previous work.  But since they wanted $225 to snake the drain, we decided to take a stab at fixing it ourselves.  If we failed, all we would lose most likely was some time and effort.

Peg guiding the rented powered auger

The first step was to pick up a hand powered drain auger (snake).  At around $25, it wasn't too big of an investment, and a handy thing to have around.  Unfortunately, we couldn't get it to go into the drain from the exposed intake for the washer.  Instead, it insisted on going up a different pipe with led to the air intake from the roof.  So it was time to get direct access to the cleanout.

The cleanout is underneath a wooden platform that holds our washer and dryer about 6" above the floor.  Whoever built the platform left an access hole in the platform from the front so that you wouldn't have to move the washer and dryer to get to it.  Awesome idea, right?  Unfortunately, this design was useless for two reasons.  First, they didn't bring the cleanout to the edge of the platform.  Instead, they left it about 18" in, making it impossible to work with.  (We tried buying a telescoping basin wrench, but that was useless.)    Second, once we unhooked our washer and dryer and removed the platform, the cleanout cap wouldn't budge anyway.  Luckily, it was attached to the cast iron pipe with flex coupler that was easily removed.  So now we were able to get our auger into the drain.  But we soon found we were having no luck clearing it.  The hand powered wire auger wasn't going to cut it for this clog.  It was time to get serious.

We went down to the Home Depot Pro shop and rented a powered auger, or "rooter" as they call it.  For a total of about $57 including tax and insurance, we rented it for 4 hours (overnight since we got it late in the day), and with only minimum cursing learning to operate the thing, we got our drain cleared!  The hardest part was actually cleaning the rooter before returning it.  Those things are wily!

The new extended cleanout

With the drain clear, we decided to not only replace the cleanout with one that worked, but to extend it out 18" so that the next time the drain got clogged, we wouldn't have to move the washer and dryer to get to it.  So we headed off to Lowes and $32 and about 100 questions later, we had the parts we needed:

• 2' length of 2" diameter  ABS pipe
• Cleanout cap and coupler
• Flex coupler to connect it to the original pipe
• Hack saw to cut the pipe (Lowe's won't cut it under 10' for some reason)
• ABS cleaning solution
• ABS cement
• Caulk to fix some leaky joints we found along the way

It didn't take too long to build the new cleanout and caulk the joints, but we had to wait until the next day for them to dry and be ready for testing.  So we took the opportunity to set up some fans and dry out the entire area over night, which would also make it easier to see if our handy work was successful.

The extended cleanout is now flush with the platform and accessible

Our test the next day was completely successful!  No evidence of leaking from the new cleanout or the joints we caulked.  When we put the platform back in place and re-connected the washer and dryer, the cleanout was flush with the access point in the platform.  If we encounter any problems in the future, it will be much easier to deal with.  And since the existing rug got damaged, we replaced it with something more festive.

All in all, it was a fair amount of work, but we only spent around $125.  And now we have a few extra tools, an improved cleanout, and a lot more experience (or "character" as Calvin's dad would call it) to apply to future problems.

Let the washing begin!

The finished product

We've been using Twitter's kestrel queue server for a while now at work, but only from our service layer, which is written in python.  Now that we have some queueing needs from our application layer, written in PHP, I spent a few days this week adding queue support to our web application.  I thought I'd share what I learned, and how I implemented it.

Goals

The kestrel server itself was pretty straightforward to get up and running.  The only thing I would point out is that I recommend sticking to release branches, as master was fairly unstable when I tried to use it.  Regarding implementing the client, there were a few goals I had in mind when I started:

• Since kestrel is built on the memcache protocol, try and leverage an existing memcache client rather than build one from scratch
• Utilize our existing batch job infrastructure, which I covered previously here, and make sure our multi-tenant needs are met
• Keep the queue interface generic in case we change queue servers later
• Utilize existing kestrel management tools, only build out the the functionality we need

With these goals in mind, I ended up with 4 components: a kestrel client, a producer, a consumer, and a very small CLI harness for running the consumer.  But before I even coded anything, I set up kestrel web, a web UI for kestrel written by my co-worker Matt Erkkila.  Kestrel web allows you to view statistics on kestrel, manage queues, as well as sort and filter queues based on manual inputs.  Having this tool up and running from the get go made it easy to watch jobs get added and consumed from my test queue, and also easily flush out the queues as needed.

The Kestrel Client

I couldn't find any existing kestrel clients for PHP, so I started looking at the two memcache extensions: the older memcache, and Andrei Zmievski's memcached, the latter of which is based on the libmemcached library.  I started with memcache, and while it worked fine initially, I quickly found that I could not modify timeouts.  This interfered with the way  kestrel recommends you poll it for new jobs, and I would see timeout errors from the memcache extension if you tried to set the poll timeout to 1 second or higher (the memcache default).  The memcached extension does not have these issues, so I went with it.

The first gotcha I ran into was serialization.  You can use memcached's serializer for writing to kestrel, but when it reads the data back, it doesn't recognize that it is serialized.  So I just serialize the data manually in my client, and things work fine. One other thing to note is that you'll want to disable compression, or do it manually, as the memcached extension will automatically compress anything over 100 bytes by default, and will not decompress it when reading from kestrel.

The other issue is that if you want to use any custom kestrel commands, you can't.  Since the application layer doesn't need anything fancy, the memcached extension will work fine for it.  Once we need support for the upcoming monitor (batching) in kestrel 2, we may need to implement a kestrel client from scratch.  Kestrel web supplies everything else we need right now.

Once the decision was made to use memcached, I wrote a light decorator for it, EC_KestrelClient.  This handles instantiation of the memcached client, serialization, and helpers for some kestrel specific options to the GET command.  It also has support for passing memcached specific options through it.  The class ended up looking like this:

The Producer

The producer is very simple.  It just formats the data into a standard structure, including current tenant information, namespaces the queue so it doesn't collide with other projects, and adds it to the queue.  The producer looks like this:

The Consumer

The consumer has a bit more to it, though still pretty straightforward.  It's intended to be run from a monitoring tool like daemontools or supervisord, so there is a very small CLI harness that just passes the CLI arguments into EC_Consumer and runs it.  After parsing the CLI arguments, EC_Consumer polls kestrel for new jobs, and runs them through our standard batch job infrastructure.  Until we have more confidence in PHP's long running process ability, I added an optional maxium jobs argument, which will stop the consumer from processing more than X jobs and then terminate.  The monitoring service (supervisord) will then just restart it in a matter of seconds.  I also added an optional debug argument for testing, so you can see every action as it happens.  The CLI harness looks like this:

And the main consumer class, EC_Consumer, looks something like this:

Putting it together

Now that all the pieces are put together, let's take a look at in action. Adding example job "HelloWorld" to the queue "hello_world" from within our application looks something like this:

And finally, here's an example of running the consumer from the CLI harness, along with some example debug output of processing the job:

That's it! I'd be interested to hear how other folks are interfacing with kestrel from PHP.

Having recently wrote about PHP step debugging in VIM, I thought I'd share a couple of things I've come across since writing it.

First, I find that I often need step debugging when writing unit tests, as that's when I'm really scrutinizing logic.  It turns out it's pretty easy to do.  All you need is to set the XDEBUG_CONFIG environment variable like so:

XDEBUG_CONFIG="idekey=session_name" php bin/phpunit.php tests/FooTest.php

When debugging from VIM, you'll want to have two shells open. One running the debugger, and another executing the tests. Also, note that in a shared environment, you can still pass in xdebug ini settings like so:

XDEBUG_CONFIG="idekey=session_name" \
php -d xdebug.remote_host=10.0.0.1 bin/phpunit.php tests/FooTest.php

Second, for step debugging in VIM, I've settled on this plugin (there are many versions out there).  It seems to solve problems with tab handling that I was seeing with older versions.

Cheers,

Bill

(Reposted from the Empower Campaigns blog)

Most web applications have static content such as CSS, JS, images, etc.  To save on bandwidth costs and improve load times for your users, it's a good idea to tell the client to cache these items for a long period of time.  But even though your static content changes infrequently, when it does, the cached content needs to be invalidated.  In addition, high traffic sites often want to go a step further and take advantage of a CDN to offload static content delivery.  Below I'll share the method we used here at Empower Campaigns to accomplish both of these things.

Version your content

The first step to getting control of cache invalidation is to manage the version of your content, usually included as part of the URL.  In the past I've seen things like incrementing a single counter for your JS and CSS files that effectively changed the URL of these files and invalidated the cache.  But this updates the URL for ALL of the files that use that counter, which is not very efficient.  A better approach is to version each file (or bundle, if you deliver files in groups) so that you control the cache invalidation of each item individually.  One way to do this is with a hash like CRC32.  If the content is stored on a file system, such as JS, CSS, or image files often are, you could hash the file upon the first read, and store that hash in a cache that gets purged during a production code push.  For PHP users, APC is a great tool for this.  This reduces the hashing overhead to one time per web node in your cluster per code release.  If your content is stored somewhere outside of your code base, you might consider storing the hash of the content at write time.  For example, when I worked at Digg, we stored user profile images in MogileFS.  So when a user added or changed their profile image, we would hash the image at that time and store the hash with that user's profile data in Cassandra.  This was even better because the content was hashed only once.  The same type of thing could be done with locally stored content, it just depends how fancy you'd like to get.  Note that when constructing your versioned URL, avoid using the version in the query string, as some CDNs strip query strings (ex: media.yourdomain.com/v/8bba8b90/css/960.min.css rather than media.yourdomain.com/css/960.min.css?v=8bba8b90)

Minify/Compress your content

In a production environment, you want to deliver static content as quickly as possible.  Two ways to help with this is minification and compression.  For minification, we use Google's Closure for JS, and YUI Compressor for CSS, both of which are run during the deployment process.  You'll also want to enable compression in your web server as well, such as mod_deflate in Apache, to make the files as small as possible during transit.

Increase cache time

Once you have your content versioning in place, you'll want to increase the cache time for those files.  We use a cache time of one year, and set that at the web server level like so using mod_rewrite:

Use a dedicated domain name

If you want to be ready to move to a CDN at any time, consider putting your static content on a dedicated virtual host, such as media.yourdomain.com.  This way, you can simply point that domain to a CDN at any time, and you'll be up and running.  Also, this has the added benefit of allowing the client's browser to make more parallel requests for your content, as they often limit the number of parallel requests they'll make to any one domain.  By spreading your content over more than one domain, the client can make more requests in parallel, speeding up the the time for the document to be ready.  One other benefit is that by using another domain for your media, you can reduce the number of cookies sent with media requests, which reduces the size of each request. Our rewrite rules for ignoring version numbers in our media host ended up looking like this:

With the above steps in mind, we wrote a view helper that handles hashing and url writing called mediaURL().  It can also look at our config and determine if SSL URLs should be used, if a given page in our application requires it.  The view helper looks like this:

And the call to the helper from our view looks like this:

Conclusion

By following the steps above, you'll have static content that is dynamically versioned, compressed, and cacheable by the client.  And if you're serving it on a dedicated domain, you'll speed up overall page load times and be ready to provision a CDN if you ever need it.

Cheers,

Bill Shupp, Lead Developer (twitter: http://twitter.com/shupp)

This week at work I got the chance to address the growing number of batch oriented CLI scripts for our main web application.  While they weren't quite unmanageable yet, they were heading in that direction.  There was too much common code, especially with bootstrapping the application and parsing options.  Also,  the location of scripts didn't really make sense... ./bin/bar.php,  ./cron/foo.php, etc.  So I decided to carve out some time and clean it up.

The goals were pretty straight forward:

• Everything must use the application's model layer.  This is mostly so that the built in caching will be consistent, but also to enforce that all data access goes through the same code.
• Centralize all CLI option parsing, application bootsrapping, error handling, and multi-tenant logic (this is a multi-tenant SaaS application)
• Keep the jobs themselves very simple.

With the above in mind, I ended up splitting things up into 3 parts:

• ./bin/ecli.php, the only PHP CLI script which does minimal bootstrapping, collects the options/arguments, and runs EC_CLI
• EC_CLI, a class that does the actual option parsing (via Zend_Console_Getopt), application bootstrapping, error handling, and multi-tenant runs (recursively calls ./bin/ecli.php for each tenant), and runs the actual job
• EC_Job_Abstract, the Job interface that collects job specific arguments and runs the job.

Now, when I need to create a new batch job EC_Job_Foo, I just put the contents in EC_Job_Foo::run(), and execute it like so:

./bin/ecli.php --environment production --instance clienta,clientb --job Foo

The ecli.php script ends up looking like this:

Pretty easy, huh?  It makes testing all the components really easy as well.

Finally, we made the decision to use Jenkins instead of the traditional cron scheduler for recurring jobs.  This has a few advantages: It's got a simple web interface for managing the job, you plug it into your existing Jenkins notifications (email, jabber, etc), console tailing, job dependencies, etc... It works really well.

When debugging problems in PHP, most of the time it's easiest to just add var_dump($foo); exit; in the middle of your script, and you can see the contents of $foo right in your browser.  But if you have to do much more, this approach gets cumbersome pretty quickly.  I've recently been using step debugging for harder to track down problems.  It allows me to examine the state of things all the way through execution of a request, line by line, or skipping ahead to break points.  This process also gives you more insight into everything else happening in a request, which can be useful when you're using frameworks or other 3rd party code in your application.

With PHP, the common extension for this is Xdebug.  (See the remote debugging documentation here).  Since my main text editor is VIM, I had to find a suitable plugin to talk to Xdebug.  It turns out there are a few good blog posts already on this topic, so I won't duplicate the information here.  This post does a great job of covering installation and usage of the Xdebug plugin for VIM by Seng Woo Shin, though it links to an older version of it.  Here's a link to a newer one that Xdebug also links to.  The Xdebug site links to another tutorial that's worth a read too.

Beyond the links above, I've discovered a couple of things that are worth sharing.  First, there's a new FireFox extension, easy Xdebug, that is a real lifesaver.  You just click the green bug on the bottom right, and it will try to start an Xdebug session with any request (rather than having to add XDEBUG_SESSION_START=1 to the query string parameters myself).  This is particularly handy when dealing with POST or or AJAX requests.  The Xdebug docs also link to similar tools for Chrome, Safari, and Opera.

Now, my debugging work flow looks like this:

• Start the Xdebug session in FF by clicking the green bug
• Set breakpoints in VIM
• Press F5 in VIM to start listening for the request (it waits 5 seconds)
• Perform that action I want to debug in FF
• Press enter in VIM to enter the debug session
• Debug
• Press F6 to exit debugger, and fix the problem

The one gotcha that I keep running into is how to set breakpoints correctly.  When you use this VIM plugin, you must put a breakpoint on executable code.  If you put it on a blank line, or even on a line with just an opening try statement, your breakpoint will not work.  Instead, you'll end up with an "error 5".  This topic is briefly covered here and here.

Now, when I find myself using more than one or two var_dump() statements to debug something, I usually walk through the work flow above, and have things sorted out much more quickly than I used to.