A recent upgrade to Ruby 2 left us with some servers to retire. Servers meant to service web requests were easy; remove them from the load balancer and shut them down. However, a dedicated task server on the project couldn’t simply be taken offline, nor could we risk running two of them with different Ruby versions. Instead, we needed to simultaneously start running tasks on a new server while stopping them on the existing one. Only then could we delete the old instance. What we found was that our existing task scheduling and deployment automation were flexible enough to fulfill these requirements.
We use the Whenever Ruby gem to manage scheduling of background tasks. Whenever provides a Ruby API to describe cron jobs, then converts them to cron syntax and inserts the rules into the server’s crontab. We were also using its integration with our deployment automation tool Capistrano to limit background jobs to running only on our task server and not the web servers. After looking closely at this aspect of our configuration we realized that we could use Capistrano’s concept of servers fulfilling roles in an application to perform our cutover. This is what it looks like:
deploy.rb:
server '0.0.0.0', roles: %w{ retire } # Old app server is given a role of 'retire'
server '0.0.0.1', roles: %w{ app db } # New app server gets the existing roles
set :whenever_roles, %w{ retire app }
schedule.rb:
every 1.hour, roles: [:app] do
# Do some work in here
end
Our Whenever schedule has many more tasks than this, but they all define the roles option. This turned out to be a great thing because that meant we could expand Whenever’s set of roles without having those tasks defined on more servers. With that option set, Whenever limits the servers on which it will define those tasks. By configuring our new and old servers to exist in their own Capistrano roles in the deploy configuration we primed Whenever to perform the seamless cutover we required.
The first step is to clear all cronjobs from both servers (the new server’s crontab is empty, and that’s okay). The second is to actually write the crontab to both servers based on Whenever’s configuration.
cutover_tasks.sh:
$ cap production whenever:clear_crontab
$ cap production whenever:update_crontab
Remember that our tasks are limited by the server’s role. We will indeed clear all tasks managed by Whenever on both servers, but the tasks only get added back to the new server because of our clever configuration from before. Now that old server can be safely retired and removed from your deployment automation.
Development of this procedure was borne out of necessity for us. We were not confident that we’d be able to upgrade this server without extraordinary effort and significant downtime. The unsung hero here are the hosting options available to us today. Rather than buying and shipping an expensive server to a datacenter, we clicked our way through a web app and had a new instance available to us right away. I don’t foresee needing to do this often, but it’s nice to know that all the tools are in place to make replacing pieces of our infrastructure quick and easy.
