I use Gearman as a queue/job server. An application gives it a job to do, and Gearman passes the job along to a worker that can finish it. Handling both synchronous and asynchronous tasks, the workers can be running anywhere – the same server as Gearman, a server across the country, or even a workstation at a local office.
This makes things a bit complicated when it comes time to push out software or configuration changes to workers. When controlling workers locally, PHP’s gearman module doesn’t have a built-in way to terminate a worker without possibly interrupting a running job. And by design, Gearman cannot broadcast a job to every worker, nor send a generic job to a specific worker. I wanted a way where I could:
- ask a worker to stop in the middle of its task (standard
- ask a worker to stop after its current task
- remotely terminate a worker
- remotely terminate all workers
Even after doing a bit of research and reading posts there didn’t seem to be a fully agreeable, developed solution. So, I took an afternoon to figure things out, with the working result ending up in a gist and some of the background below.
For the first part, it was simply a matter of handling a
SIGTERM signal with PHP’s pcntl module and setting a
termination flag. The main worker loop could then check the flag every time it finished a job and cleanly exit. The
Gearman library complicated things a bit though because while it’s waiting for a job, none of the signals are
acknowledged. The workaround was to use its non-blocking alternative. Although it still seemed to do some blocking, it
was at least a configurable duration. Abbreviated, worker.php looks like:
When sent a
SIGTERM while running a job, it would wait to finish before exiting:
Sometimes it’s easier to remotely terminate workers when they need new code or configuration (and allowing a process manager to restart them). Since Gearman doesn’t support sending a job to every single worker, an alternative is to have a terminate function for every worker (as mentioned in this response). Assuming every worker has a unique identifier, this becomes trivial:
From the console, it looks like:
Batch Remote Termination
So now I can remotely terminate workers as needed. However, during deploys it’s much more common to ask all the workers
to restart. Using Gearman’s protocol to find running workers I can distribute the termination job and then wait
until all workers have received it. The result was
terminate.php, working something like.
The result is an extra bit of code, but it makes automating tasks, especially around deploys, much easier. This really just demonstrates one method of creating an internal workers API - termination is just one possibility. Other more complex possibilities could be self-performing updates, lighter config reloads (instead of full restarts), or dynamically registering/unregistering functions depending on application load.