Over the last nine months I've been getting into BOSH quite a bit. Historically, I've been reluctant to invest in BOSH because I don't entirely agree with its architecture and steep learning curve. BOSH describes itself with...

BOSH installs and updates software packages on large numbers of VMs over many IaaS providers with the absolute minimum of configuration changes.

BOSH orchestrates initial deployments and ongoing updates that are:

• Predictable, repeatable, and reliable
• Self-healing
• Infrastructure-agnostic

With continued use and experience necessitated from the logsearch project, I saw ways it would solve more critical problems for me than it would create. For that reason, I started experimenting and migrating some services over to BOSH to better evaluate it for my own uses. To help bridge the gap between BOSH inconveniences and some of my architectural/practical differences I've been making a tool called cloque.

You might find the ideas more useful rather than the cloque code itself – it is, after all, experimental and written in PHP (since that's why I'm most productive in) whereas bosh is more Ruby/Go-oriented.

Infrastructure First

Generally speaking, BOSH needs some help with infrastructure (i.e. it can't create its own VPC, network routing tables, etc). Additionally, sometimes deployments don't even need the BOSH overhead. Within cloque, I've split management tasks into two components:

• Infrastructure – this is more of the "physical" layer defining the networking layer, some independent services (e.g. NAT gateways, VPN servers), security groups, and other core or non-BOSH functionality.
• BOSH – everything related to BOSH (e.g. director, deployment, snapshots, releases, stemcells) which is deployed onto the infrastructure somewhere.

Since BOSH depends on some infrastructure, we'll get started with that first. One key to a cloque-managed environment is that each environment has its own directory which includes a network.yml in the top-level. The network may be located in a single datacenter, or it could span multiple countries. The file defines all the basics about the network including subnets, reserved IPs, basic cloud properties, and some logical names.

I've committed an example network to the share directory within cloque and will use that in the examples here. To get started, we'll copy the example and work with it...

If you take a look at the sample network.yml, you'll see a couple regions with their individual network segments, VPN networks, and a few reserved IP addresses which can be referenced elsewhere. Once network.yml is created, the utility:initialize-network task can take care of bootstrapping the following:

• create stub folders for your different regions; e.g. aws-apne1/core, global/private)
• create a new SSH key (in global/private/cloque-{yyyymmdd}*.pem) and upload it to the AWS regions being used
• create a new IAM user, access key, and EC2 policy for BOSH to use
• create a certificate authority for OpenVPN usage
• create both client/server certificates for the inter-region VPN connections (requires interactive prompts for passwords/confirmations)
• create an S3 bucket for shared configuration storage

When run, it assumes AWS credentials can be discovered from the environment...

The top-level global directory is intended for configuration which applies to all areas. With the example I use it to create an additional IAM role which allows VPN gateways to securely download their VPN keys and configuration files...

The infra:put is the core command responsible for managing the low-level, infrastructure-related resources. The command looks for an infrastructure.json file (see the example) and since I'm focused on AWS, the files are CloudFormation scripts.

Now that the global resources have been created, we can create our "core" resources for the us-west-2 region. If you take a look at the infrastructure.json file, you'll see it creates a VPC, multiple subnets for each availability zone, a couple base security groups, and a gateway instance which will function as a VPN server to allow inter-region communication. You'll also notice it's using Twig templating to load network.yml and simplify what would be a lot of repeated resources. We'll use the infra:put command again, but this time within the aws-usw2/core directory...

Once the infra:put command finishes, the aws-usw2 part of the environment is complete which means the OpenVPN server is ready for a client. First we'll need to create and sign a client certificate though...

Since I'm using example.com in the share scripts as the domain, DNS won't resolve it. For now, the easiest solution is to manually add an entry to /etc/hosts...

Once /etc/hosts is updated, I can connect with an OpenVPN client like Tunnelblick and ping the network...

BOSH Director

Now that we have a VPC and a private network to deploy things into, we can start a BOSH Director. Here it's important to note that I'm using "region", "network segment", and "director" interchangeably. Typically you'll have a single BOSH Director within an environment's region, and since that Director will tag it's deployment resources with a "director" tag, I decided to make them all synonyms. The effect is twofold:

• when you see a "director" name (whether it's in the context of BOSH or not) it refers to where resources are provisioned
• you can consistently use a "director" tag (BOSH or not) to identify where something is deployed which makes AWS resource management much simpler (and AWS Billing reports by tag much more valuable).

Back to getting BOSH deployed though. First, we'll create some additional BOSH-specific, region-specific infrastructure (specifically, security groups for the director and agents)...

Once the security groups are available, we can create the BOSH Director. The boshdirector:* commands deal with the Director tasks (i.e. they don't depend on a specific deployment). To get started, the boshdirector:inception:start command takes care of provisioning the inception instance (it takes a few minutes to get everything installed and configured)...

Before continuing, there's still a manual process of finding the correct stemcell. If we were in us-east-1, we could use the "light-bosh" stemcell (which is really just an alias to a pre-compiled AMI that Cloud Foundry publishes). Unfortunately, we need to take the slower route of compiling our own AMI for us-west-2. To do this, we need to lookup the latest stemcell URL from the published artifacts, then we pass that URL to the next command...

Once the provisioning step is complete, I can login and talk to BOSH...

Since BOSH Director is successfully running, it's safe to terminate the inception instance. Whenever there's a new BOSH version I want to deploy, I can just rerun the two start and provision commands (with an updated stemcell URL) and it will take care of upgrading it.

More on Stemcells

While inception was deploying the BOSH Director, it ended up making a stemcell that I can reuse for our BOSH deployments. Unfortunately, the Director doesn't know about it. The following command takes care of publishing it...

Another task I frequently need to do is convert the standard stemcells (which only support the PV virtualization) into HVM stemcells that I can use with AWS's newer instance types. This next command takes care of all those steps and, once complete, there will be a new *-hvm stemcell ready for use on the Director.

BOSH Deployment

Now that the BOSH Director is running, I can deploy something interesting onto it. Let's use logearch as an example. First I'll need to clone the repository...

Since I've changed directories away from our environment, cloque will no longer know where to find its environment information. To help, I'll use a .env file...

Now I can upload the release...

The example has the configuration files for infrastructure (EIP and security groups) and BOSH (deploy manifest), but I still need to generate a certificate locally...

Once uploaded, I can use the infra:put and mirrored bosh:put command to push the infrastructure and BOSH deployment (-n meaning non-interactive, just like with bosh)...

Once complete, I can see the elasticsearch service running...

And I can see the ingestor listening on its EIP:

And I can SSH into the instance...

Package Development

When I need to create a new package, I started using a convention where I'd add the origin URL where I found a blob/file. This provides me with more of an audit over time, but also allows me to automate a spec file which looks like:

Into a series of wgets with the boshutil:package-downloads command...

When I'm working on a packaging script I use Docker images to emulate the build environment. Since 99% of my build issues come from configure arguments and environment variables, this is normally sufficient. This also lets me iteratively debug my packaging scripts as opposed to the slow, guess and check method of re-releasing and deploying the whole thing to BOSH to test fixes. The boshutil:package-docker-build command helps me here...

Snaphots

One easy feature that BOSH has is snapshotting to get a full backup of its persistent disks. You can run its take snapshot command for a particular job or for an entire deployment. Or, if "dirty" snapshots are okay, the Director can schedule them automatically. To manage all those snapshots, I created a few commands. The first command takes care of snapshots that the BOSH Director creates of itself...

The command for individual deployment snapshots is a bit more intelligent. It allows writing logic which, when passed a given snapshot, determines whether it should be retained or deleted. For example...

Revitalizing

In the past I've typically used local VMs with VirtualBox or VMWare Fusion for personal development. Unfortunately they always seemed to drift from production servers, which made things inconvenient, at best. With BOSH, it became trivial for me to start/stop deployments and guarantee they have a known environment. When my VMs were local I always had scripts which would pull down backups, restore them, and clean up data for development. With cloque I've been using a revitalize concept which allows me to restore data from snapshots or run arbitrary commands. For example, I can add the following to my database job to restore data from a slave's most recent snapshot...

Whenever I want to reload my dev deployment with more recent production data (or after I've sufficiently polluted my dev data), I can just run the bosh:revitalize task...

Configuration Transformations

I mentioned earlier that configuration files are templates. In addition to basic templating capabilities, I added some transformation options. Transformations allow a processor to receive the current state of the configuration, do some magic to it, and return a new configuration. The easiest example of this is with logging – I want to centralize all my log messages and collectd measurements. Here I'll use logsearch-shipper-boshrelease, but regardless of how it's done, it typically requires adding a new release to your deployment, adding the job template to every job, and adding the correct properties. When you have multiple deployments, this becomes tedious and this is where a transformation shines. The transform could take care of the following:

• adding the logsearch properties (SSL key, bosh_director field to messages, EIP lookup for the ingestor)
• add the logsearch-shipper release to the deployment
• add the logsearch-shipper job template to every job

And raw code for that transform could go in aws-usw2/logsearch/shipper-transform.php:

And then whenever I want a deployment to forward its logs with logsearch-shipper, I only need to add the following to the root level of my bosh.yml deployment manifest...

Tagging

Since starting with BOSH, I've used AWS tags more heavily. I consistently use the director tag to represent the {network_name}-{region_name} (e.g. acme-dev-aws-usw2) and the deployment tag to represent the logical set of services (regardless of whether BOSH is managing them or not). I made another command which can enumerate relevant resources and ensure they have the expected tags:

Conclusion

BOSH is far from perfect, in my mind, but with a little help it is enabling me to be more productive and effective than other tools I've tried in the areas which are most important to me.