New Concourse Resource for BOSH Releases on December 23, 2018

As a “continuous thing-doer” Concourse is great for documenting workflows and making sure they run. One of the workflows I frequently automate is consuming and publishing BOSH releases. Existing resources had some shortcomings for my needs, so I created the bosh-release resource to support those workflows. This post discusses more of the background and decisions that went into the resource.

Watching Upstream Binaries with Concourse on December 2, 2018

When building software packages, it’s easy to accumulate dependencies on dozens of other, upstream software components. When building the first version of something, it’s easy to blindly download the source of the latest version off the packages’ website. However, once you’re past prototypes and need to deal with auditing or maintenance, it becomes important to have some [automated] processes in place.


Data Processing with Concourse on October 19, 2016

Recently I needed to focus on a project that regularly processed datasets with typical extract, transform, and load stages. Historically it was using Amazon SQS to queue up the tasks for each stage, some supervisor-managed processes to work off the queue, and Amazon S3 to store the results of each stage. The original implementation was struggling because it was inefficient, difficult to detect problems, and more difficult to replay a stage whenever unexpected, bad data showed up.

Composing Configurations with JQ on April 26, 2016
When managing configurations for services there are often variables which need to be changed depending on the environment or use case. Different tools deal with that sort of parameterization slightly differently. For example… AWS CloudFormation - stack templates have a high level Parameter type which can contain user-supplied values. There are built-in functions to concatenate and do some other primitive transformations. BOSH - manifests are actually an ERB template, allowing for dynamic inclusion of environment variables, file contents, settings from configuration files, or complicated logic.