New Concourse Resource for BOSH Releases on December 23, 2018

As a “continuous thing-doer” Concourse is great for documenting workflows and making sure they run. One of the workflows I frequently automate is consuming and publishing BOSH releases. Existing resources had some shortcomings for my needs, so I created the bosh-release resource to support those workflows. This post discusses more of the background and decisions that went into the resource.

Watching Upstream Binaries with Concourse on December 2, 2018

When building software packages, it’s easy to accumulate dependencies on dozens of other, upstream software components. When building the first version of something, it’s easy to blindly download the source of the latest version off the packages’ website. However, once you’re past prototypes and need to deal with auditing or maintenance, it becomes important to have some [automated] processes in place.


Self-Upgrading Packages in BOSH Releases, Part 2 on October 21, 2016

Last year I wrote a post about how the process of updating BOSH release blobs could be better automated. The post relied on some scripts which could be executed to check and download new versions of blobs. The scripts were useful, but they still required manual execution and then testing to verify compatibility. My latest evolution of the idea further automates this with Concourse to check for new versions, download new blobs, build test releases, and then send pull requests for successful upgrades.

Composing Configurations with JQ on April 26, 2016
When managing configurations for services there are often variables which need to be changed depending on the environment or use case. Different tools deal with that sort of parameterization slightly differently. For example… AWS CloudFormation - stack templates have a high level Parameter type which can contain user-supplied values. There are built-in functions to concatenate and do some other primitive transformations. BOSH - manifests are actually an ERB template, allowing for dynamic inclusion of environment variables, file contents, settings from configuration files, or complicated logic.

Experimenting with BOSH Links and Consul on January 11, 2016

With BOSH, I use deployments to segment various services. For example, TLE has several services like web and database servers, WordPress blogs, the main e-commerce application, statistics, and internal services. Many of them are interconnected in some way. Historically I’ve used a combination of hard-coded IP addresses in the deployment properties and dynamic service discovery with consul. With a small bit of tweaking and an extra pre-parser, I’m now able to emulate much of the proposed links features, but from a more dynamic, distributed perspective.


Tempore limites: BOSH Veneer on November 12, 2015

For all the low-level handling of things, BOSH is a good tool for system administration. But when it comes to configuring everything, I think it leaves something to be desired for the average Joe. Opening my text editor, making changes to the YAML, copying and pasting security groups from AWS Console, git diffing to make sure I did what I think, git commiting in case things go bad, bosh deploying to make it so… it can become quite the process. For me, I’m much more a visual person and prefer a browser-based tool. Since I’ve had a bit extra free-time lately, I’ve spent some time experimenting on ideas to help improve my BOSH-quality-of-life.

Pruning Blobs from BOSH Releases on August 6, 2015
Over time, as blobs are continually added to BOSH releases, the files can start consuming lots of disk space. Blobs are frequently abandoned because newer versions replace them, or sometimes the original packages referencing them are removed. Unfortunately, freeing the disk space isn’t as simple as rm blobs/elasticsearch-1.5.2.tar.gz because BOSH keeps track of blobs in the config/blobs.yml file and uses symlinks to cached copies. To help keep a lean workspace, I remove references to blobs which are no longer needed in my release.

Self-Upgrading Packages in BOSH Releases on August 3, 2015

Outside of BOSH world, package management is often handled by tools like yum and apt. With those tools, you’re able to run trivial commands like yum info apache2 to check the available versions or yum update apache2 to upgrade to the latest version. It’s even possible to automatically apply updates via cron job. With BOSH, it’s not nearly so easy since you must monitor upstream releases, manually downloading the sources before moving on to testing and deploying. Personally, this repetitive sort of maintenance is one of my least favorite tasks; so, to avoid it, I started automating.

New BOSH Release for OpenVPN on June 3, 2015

I’m a big fan of OpenVPN - both for personal and professional VPNs. Seeing as how I’ve been deploying more things with BOSH lately, an OpenVPN release seemed like a good little project. I started one about nine months ago and have been using development releases ever since, but last week I went ahead and created a “final” release of it.


Simplifying My BOSH-related Workflows on September 17, 2014
Over the last nine months I’ve been getting into BOSH quite a bit. Historically, I’ve been reluctant to invest in BOSH because I don’t entirely agree with its architecture and steep learning curve. BOSH describes itself with… BOSH installs and updates software packages on large numbers of VMs over many IaaS providers with the absolute minimum of configuration changes. BOSH orchestrates initial deployments and ongoing updates that are: