Self-Upgrading Packages in BOSH Releases, Part 2

October 21, 2016

Last year I wrote a post about how the process of updating BOSH release blobs could be better automated. The post relied on some scripts which could be executed to check and download new versions of blobs. The scripts were useful, but they still required manual execution and then testing to verify compatibility. My latest evolution of the idea further automates this with Concourse to check for new versions, download new blobs, build test releases, and then send pull requests for successful upgrades.

Existing Scripts as Concourse Resources

In the previous post, I relied on one script to monitor versions and another script to download a specific version. This is very similar to Concourse resources which utilize a check and get script, but it would be a lot of work to manually maintain a full Concourse-friendly resource for every blob package. Instead, I created a task which wraps the simple version/download scripts into a Docker image that Concourse can execute as a regular resource. For each self-upgrading blob, I have a job which looks like...

{ "plan": [
  { "get": "repo",
    "trigger": true },
  { "task": "prepare-buildroot",
    "file": "repo/ci/images/release-blob/prepare/task.yml",
    "params": {
      "blob": "{{blob}}" } },
  { "put": "ci-release-blob-{{blob}}-image",
    "params": {
      "build": "buildroot" } } ] }

Once the naïve scripts are wrapped by resource-friendly scripts, I can then use the Docker images as a custom resource type in my pipelines. By geting them as a trigger, tasks will have access to whatever files were downloaded. For example, I can have a job which adds the new package blobs to my release's blobstore before continuing to push somewhere...

{ "plan": [
  { "get": "blob",
    "resource": "release-blob-{{blob}}",
    "trigger": true },
  { "get": "repo" },
  { "task": "bump-release-blob",
    "file": "repo/ci/tasks/bump-release-blob/task.yml",
    "params": {
      "blob": "{{blob}}" } },
  ... ] }

Running Tests

Whenever something changes in a release, you typically want to run all of the tests. Historically I maintained this logic in my main pipeline which meant I either needed to immediately push the new (potentially broken) blobs to my main branch, or I needed to duplicate my test plans. For a third option, I utilized jq to extract my integration tests into a reusable function...

def run_integration_tests:
  [ { "aggregate": [
        { "get": "bosh-lite-stemcell" },
        { "put": "bosh-lite" } ] },
    { "put": "bosh-lite-integration-deployment",
      "params": {
        "target_file": "bosh-lite/target",
        "manifest": "repo/ci/tasks/integration-test/deployment.yml",
        "stemcells": [
          "bosh-lite-stemcell/*.tgz" ],
        "releases": [
          "release/*.tgz" ] } },
    { "task": "integration-test",
      "file": "repo/ci/tasks/integration-test/task.yml" } ] ;

With my tests refactored into a separate function I can add this run_integration_tests function after I execute bump-release-blob from above...

{ "plan": [
  ...,
  { "task": "bump-release-blob",
    "file": "repo/ci/tasks/bump-release-blob/task.yml",
    "params": {
      "blob": "{{blob}}" } },
  create_dev_release,
  run_integration_tests[] ] }

This allows me to immediately test my release changes without introducing them on a public branch. I'm also able to continue using the exact same integration tests in my main pipeline.

Making it Easy to Merge

Once tests are successful, it is more reasonable for someone to spend time reviewing and merging the new version. To that end, I configured the final blob-upgrade-tester job to push the changes to a blob-specific branch and send a pull request. The pull request includes a reminder about what exactly needs to happen.

By making this pull request, the upgrade becomes something I can easily finish, merge, and cleanup from the GitHub web UI, even without a terminal or laptop.

Automate

By automating package upgrades, it is easier to stay up to date with security patches which affect my releases. Refactoring my test tasks out into reusable functions helps provide better confidence in the upgrades before they hit any main branches. Utilizing pull requests for applying the changes reduces friction and provides a reminder in my inbox that something needs to happen. Although this process requires a bit of development overhead, the ability to rely on bots for upgrading, testing, and reminding me about dependency changes lets me focus on more creative tasks in the releases I have been testing this approach in.