Continuous Deployment for Static Sites with Docker, AWS, and Ansible

Introduction

Static sites are popular because they are easy to work with, highly performant, and easily deployed. Static sites are also a nice test bed for continuous deployment practices due to their simplicity. This tutorial demonstrates how to use Docker, Middleman, Ansible, and AWS to build a continuous deployment system so your site’s infrastructure and content are deployed on every commit.

Middleman and Docker

Let’s start with Middleman. Middleman provides many features to static site authors. Specifically, it shares features that you find in complex web frameworks such as JS/CSS minification, support for SaSS, and other features for friendlier frontend work. Middleman is written in Ruby, so it includes a fair amount of work to set up a Ruby environment if you don’t have one already.

We’ll use Docker to encapsulate the Middleman environment. This is not strictly necessary but demonstrates a powerful Docker use case. Bootstrapping our Middleman environment takes two steps. First, run middleman init. This will ask you some questions about what you’d like to use and create the directory structure. This creates two important files: Gemfile and Gemfile.lock. These two files lists all the dependencies. If you opt for compass when initializing the project, the Gemfile and Gemfile.lock will include the compass library and any supporting dependencies. We use these two files to build the project-specific environment.

Let’s use make to coordinate this process. We’ll also need two independent Dockerfiles, one to build the generic middleman environment to run middleman init and the other to run middleman build with our project specific dependencies.

The middleman init Docker image needs Ruby, the Middleman gem, and some git config. The git config is somewhat abnormal. middleman init does a git clone to pull in some of its dependencies. This command fails if some get config things are not set appropriately. We’ll use the official Ruby image. Here is the Dockerfile.init:

	FROM ruby:2.3

	# Install nodejs because middleman requires a JavaScript runtime
	RUN apt-get update -y && apt-get install -y nodejs

	RUN gem install middleman

	# Set git-config things to middleman can use git clone over HTTPS
	RUN git config --system user.name docker \
		&& git config --system user.email docker@localhost.com

Now we can use the Dockerfile.init to build a Docker image for middleman init. make coordinates the process. The high level process is as follows:

1. Build the Docker image,
2. Start a Docker container for middleman init,
3. Use docker cp to copy files from the container file system onto the host, and
4. Stop and remove the container.

docker cp is key since the files are generated in a container. Docker containers run as root by default. Using a volume mount (something like -v "${PWD}:/data") results in root files written back to the host. This is problematic for a few reasons. First, having root owned files where unexpected will break things. This could be fixed by setting container’s user (-u) to match our users ID. However, this creates problems when the container requires elevated permissions. Middleman falls into this category by default because it runs bundle install during middleman init. These problems are resolved by using the create, start, cp, stop, and rm Docker commands. It requires more effort than using a shared file system volume but it will work with any Docker context.

Let’s create a make init the implements the previously mentioned process.

	tmp:
		mkdir -p $@

	.PHONY: init
	init: | tmp
		docker build -t middleman -f Dockerfile.init .
		docker create -it -w /data middleman middleman init > tmp/init_container
		docker start -ai $(cat tmp/init_container)
		@docker cp $(cat tmp/init_container):/data - | tar xf - -C $(CURDIR) --strip-components=1 > /dev/null
		@docker stop $(cat tmp/init_container) > /dev/null
		@docker rm -v $(cat tmp/init_container) > /dev/null
		@rm tmp/init_container

First, the middleman image is built from the Dockerfile.init. Next, a new container is created to run middleman init. The -it keeps standard input open and allocates a TTY so colors work. -w sets the current directory to /data. This creates a known directory to copy files from. docker create prints the container ID to standard out. The output is redirected to a temporary file for future use. The docker cp writes a tar archive to standard out which is piped to tar xf. tar extracts the contents to the current directory (-C $(CURDIR)). --strip-components=1 removes data/ from the path name. The end result is that all the files created by middleman init are written to current directory. Finally, the container is stopped, removed, and the temporary file is deleted.

Run make init and answer the prompts accordingly. Now, we have a fully functional Middleman project. We’re not entirely out of the woods yet. Let’s set up the middleman build structure before bootstrapping AWS and deploying.

The middleman build step is similar to middleman init. We’ll build a Docker image that includes the dependencies and source code. Since Middleman sites are standard Ruby applications, we can use the “onbuild” image to automatically install dependencies and add all source files. Then, we can add any specific customizations on top of that. Here’s the Dockerfile:

	FROM ruby:2.3-onbuild

	# Install the nodejs as Middleman's JavaScript runtime.
	RUN apt-get update -y && apt-get install -y nodejs

	CMD [ "bundle", "exec", "middleman", "--help" ]

We’ll configure make dist, which runs middleman build. The structure is the same as make init with some path names changed.

	# NOTE: you can replace this whatever you like!
	IMAGE:=slashdeploy/static-site-tutorial

	.PHONY: dist
	dist: Gemfile.lock | tmp
		mkdir -p build
		docker build -t $(IMAGE) .
		docker create $(IMAGE) middleman build > tmp/dist_container
		docker start $(cat tmp/dist_container)
		@docker cp $(cat tmp/dist_container):/build - | tar xf - -C build --strip-components=1 > /dev/null
		@docker stop $(cat tmp/dist_container) > /dev/null
		@docker rm -v $(cat tmp/dist_container) > /dev/null
		@rm tmp/dist_container

Note that build is used in docker cp. This is because middleman build outputs assets to ./build. Run make dist and check ./build for the complete site. Now that we can generate the site, it’s time turn our attention to deployments.

Bootstrap AWS with CloudFormation

We’ll use CloudFormation to create the S3 Bucket, CloudFront CDN, and Route53 DNS entry. CloudFormation templates are JSON files that tell AWS how to create or update resources. The templates are parameterizable as well. We’ll provide the bucket name, subdomain, and TLD. Then, we’ll make a S3 bucket and configure it for static site access. Next, we’ll create a CloudFront CDN to serve the site. Finally, we’ll create a Route53 CNAME for the CDN’s hostname.

Writing the CloudFormation template may be a bit daunting, so we’ll approach it piece by piece. Each section is part of a larger JSON object. Refer to the source for the final version.

The first section states which version this template is in, a description, and the input Paramters. Declared Parameters must be a valid type. CloudFormation uses String as a general purpose type. Note that all these parameters are required since no default value is specified.

	"AWSTemplateFormatVersion": "2010-09-09",
	"Description": "Static Website: S3 bucket, Cloudfront, & DNS",
	"Parameters": {
		"AppName": {
			"Type": "String"
		},
		"Subdomain": {
			"Type": "String"
		},
		"TLD": {
			"Type": "String"
		}
	}

We’ll declare the resources. The Resources key describes all the “physical” AWS resources in the template. Each key in the Resources object is an individual resource. The object must include a Type and Properties key. The Type sets AWS resources, e.g. S3 bucket, EC2 instance, Elastic Load Balanacer. The Properties keys sets all relevant information. The content inside Properties varies by the specified Type. The CloudFormation documentation lists all properties for all types.

Our static website starts with an S3 bucket. The following snippet creates the resource named Bucket. It sets the index document to index.html and error.html for anything in the 4xx/5xx range.

	"Resources": {
		"Bucket": {
			"Type": "AWS::S3::Bucket",
			"Properties": {
				"BucketName": { "Ref": "AppName" },
				"WebsiteConfiguration": {
					"ErrorDocument": "error.html",
					"IndexDocument": "index.html"
				}
			}
		}

Notice the Ref built in function. CloudFormation functions are specially named keys since there is no “function” in JSON. Ref, short for reference, connects various pieces. Ref gets the value for the AppName parameter and assigns that to the BucketName. Ref is also used to get both names and ID’s of resources in the template, which we’ll see shortly.

The previously mentioned S3 bucket needs a policy that allows anyone on the internet to read from it. The following snippet uses Ref to connect the policy to the S3 bucket in the template. It also uses the Fn::Join function to create a path matching all items in the bucket /*.

	"Policy": {
		"Type": "AWS::S3::BucketPolicy",
		"Properties": {
			"Bucket": { "Ref": "Bucket" },
			"PolicyDocument": {
				"Version": "2012-10-17",
				"Statement": [{
					"Sid": "PublicReadGetObject",
					"Effect": "Allow",
					"Principal": "*",
					"Action": [ "s3:GetObject" ],
					"Resource": [{
						"Fn::Join": [ "", [
							"arn:aws:s3:::",
							{ "Ref": "Bucket" },
							"/*"
						]]
					}]
				}]
			}
		}
	}

The template includes the S3 bucket and appropriate policy at this point. It’s time to put that behind a CloudFront CDN. CloudFront is key because it provides a domain name we can use for a future CNAME entry.

The CloudFront CloudFormation resource is the most complicated one so far. We will:

1. Register an “alias” based on the Subdomain and TLD input parameters. This makes the CDN behave correctly when accessed over our own domain,
2. Create an origin that reads from the previously created S3 bucket. The Fn::Join function creates the appropriate URL, and=
3. Configure caching behavior that lasts in the edge nodes for 300 seconds (5 minutes), adds gzip support, and ignores cookies and the query string.

Here’s the resource snippet:

	"Distribution": {
		"Type": "AWS::CloudFront::Distribution",
		"Properties": {
			"DistributionConfig": {
				"Aliases": [
					{
						"Fn::Join": [ ".", [
							{ "Ref": "Subdomain" },
							{ "Ref": "TLD" }
						]]
					}
				],
				"Comment": { "Ref": "AppName" },
				"DefaultRootObject": "index.html",
				"Enabled": "true",
				"Origins": [
					{
						"Id": { "Fn::Join": [ "-", [
							{ "Ref": "AppName" },
							"s3-website"
						]]},
						"DomainName": {
							"Fn::Join": [ ".", [
								{ "Ref": "Bucket" },
								{ "Fn::Join": [ "-", [ "s3-website", { "Ref": "AWS::Region" } ]] },
								"amazonaws.com"
							]]
						},
						"CustomOriginConfig": {
							"OriginProtocolPolicy": "http-only"
						}
					}
				],
				"DefaultCacheBehavior": {
					"DefaultTTL": 300,
					"Compress": "true",
					"ForwardedValues": {
						"Cookies": {
							"Forward": "none"
						},
						"QueryString": "false"
					},
					"TargetOriginId": { "Fn::Join": [ "-", [
						{ "Ref": "AppName" },
						"s3-website"
					]]},
					"ViewerProtocolPolicy": "allow-all"
				},
				"PriceClass": "PriceClass_All",
				"ViewerCertificate": {
					"CloudFrontDefaultCertificate": "true"
				}
			}
		}
	}

Last but not least, we hit the DNS entry. This resource is straightforward. The HostedZonedId is a special hard-coded value on AWS that indicates CloudFront. AWS Route53 behaves a bit differently compared to normal DNS providers when integrating its own offerings. This is why Type is to A instead of CNAME as you may expect. Note that the hosted zone is not configured through the CloudFormation template. This is done to avoid giving CloudFormation ownership of the Route53 hosted zone so other CloudFormation stacks in the account may use the hosted zone. This also means that deleting one stack cannot delete the hosted zone and thus all associated DNS records.

	"DNS": {
		"Type": "AWS::Route53::RecordSet",
		"Properties": {
			"HostedZoneName": {
				"Fn::Join": [ "", [ { "Ref": "TLD" }, "." ] ]
			},
			"Name": {
				"Fn::Join": [ ".", [
					{ "Ref": "Subdomain" },
					{ "Ref": "TLD" }
				]]
			},
			"Type": "A",
			"AliasTarget": {
				"HostedZoneId": "Z2FDTNDATAQYW2",
				"DNSName": { "Fn::GetAtt": [ "Distribution", "DomainName" ] }
			},
			"Comment": "CloudFront Distribution alias"
		}
	}

Finally, we declare the Outputs. Our template outputs the public URL, combination of the Subdomain and TLD parameters, and a few other items for deployment and debugging purposes. The Bucket output will be used when uploading assets to S3. PublicURL will be used for deploy verification.

	"Outputs": {
		"Bucket": {
			"Description": "S3 Bucket",
			"Value": { "Ref": "Bucket" }
		},
		"OriginURL": {
			"Description": "S3 Website URL",
			"Value": { "Fn::GetAtt": [ "Bucket", "WebsiteURL" ] }
		},
		"DistributionURL": {
			"Description": "Cloudfront URL",
			"Value": { "Fn::GetAtt": [ "Distribution", "DomainName" ] }
		},
		"PublicURL": {
			"Description": "Public URL",
			"Value": { "Fn::Join": [ ".", [
				{ "Ref": "Subdomain" },
				{ "Ref": "TLD" }
			]]}
		}
	}

Deploying with Ansible

Deploying our static site is a straightforward process. First, the CloudFormation stack must be deployed. This provides the place to upload the assets. We’ll pass the stack output values to other components where needed. Next, we’ll generate assets with middleman build. After that, we’ll use aws command to sync that directory the S3 bucket.

Ansible works well for this purpose because of its built in support for CloudFormation templates — it can create/update the CloudFormation stack accordingly. It also works well for invoking commands and coordinating other resources. Ansible runs “playbooks”. Playbooks are YML files configured to run against specific hosts, e.g. localhost, list of DB servers, application servers, etc. “Playbooks” contain “tasks”. Each task uses a module to do something. Let’s start building a deploy playbook step by step.

This playbook will run on localhost since there are no external hosts to run against. This effectively turns our playbook into a locally executed program. Refer to the complete source to view the complete file.

The following snippet is the “playbook” header. It states the playbook runs on `localhosta via direct command execution. You can think of this as running commands in your terminal. Second, it defines variables for use throughout the playbook.

	- hosts: localhost
		connection: local
		gather_facts: False
		vars:
			aws_region: eu-west-1
			app_name: your-site-name
			tld: your-domain-name.com

Let’s declare tasks. The first task is to deploy the CloudFormation stack. We’ll use the built-in cloudformation module as illustrated below.

	tasks:
	- name: Deploy stack
		cloudformation:
			state: present
			stack_name: "{{ app_name }}"
			region: "{{ aws_region }}"
			disable_rollback: true
			template: "cloudformation.json"
			template_parameters:
				BucketName: "{{ app_name }}"
				Subdomain: "{{ app_name }}"
				TLD: "{{ tld }}"
		register: cf

Each task may have an optional name. The name is printed out when you run the playbook. Each task uses a single Ansible module. This one uses CloudFormation. All the keys under cloudformation set individual settings. The text inside {{ }} references variables. The register keywords saves the task output in the cf variable. We will use this variable to get the stack output. This task will wait for the stack to create and update accordingly. The assets are uploaded afterwards.

Ansible includes the command module for invoking individual commands. We’ll use this to invoke a make target to build and upload the assets to specified S3 bucket. First, we need to generate the assets. Let’s optimize the build for the best performance using Middleman’s build in features. Let’s enable JavaScript, CSS minification, and assets hashes. We’ll provide these flags via environment variables. This gives us a way to toggle settings depending our context, e.g. production vs. development build. We’ll come back to the implementation later. For now, let’s assume we can set some environments and things will work as expected. We’ll use make again to upload the assets. make accepts variables on the command line. We’ll use the CloudFormation outputs to pass along the information. Here are the playbook tasks:

	- name: Generate assets
		command: make dist
		environment:
			MIDDLEMAN_MINIFY_JS: true
			MIDDLEMAN_MINIFY_CSS: true
			MIDDLEMAN_HASH_ASSETS: true
		changed_when: False

	- name: Upload assets
		command: make deploy-site REGION={{ aws_region }} BUCKET={{ cf.stack_outputs.Bucket }}
		changed_when: False

The tasks rely on the code we haven’t created yet. We’ll need to update the Makefile and config.rb to handle environment variable. Let’s consider config.rb first. These changes are straightforward. We’ll set the appropriate Middleman configuration option when an environment variable is set.

	# Build-specific configuration
	configure :build do
		activate :minify_css if ENV['MIDDLEMAN_MINIFY_CSS'] == 'true'

		activate :minify_javascript if ENV['MIDDLEMAN_MINIFY_JS'] == 'true'

		activate :asset_hash if ENV['MIDDLEMAN_HASH_ASSETS'] == 'true'
	end

Now, we need to update make dist to pass along enviornment variables.

	.PHONY: dist
	dist: Gemfile.lock | tmp
		mkdir -p build
		docker build -t $(IMAGE) .
		docker create \
			-e MIDDLEMAN_MINIFY_JS \
			-e MIDDLEMAN_MINIFY_CSS \
			-e MIDDLEMAN_HASH_ASSETS \
			$(IMAGE)
			middleman build > tmp/dist_container
		docker start $(cat tmp/dist_container)
		@docker cp $(cat tmp/dist_container):/build - | tar xf - -C build --strip-components=1 > /dev/null
		@docker stop $(cat tmp/dist_container) > /dev/null
		@docker rm -v $(cat tmp/dist_container) > /dev/null
		@rm tmp/dist_container

Only docker create command changes. The -e option sets that environment variable on the Docker container. Docker uses the current value on the host when a value is not specified. This means the environment variables set in the playbook task are forwarded to the Docker container.

Let’s configure make deploy-site.

	.PHONY: deploy-site
	deploy-site:
		aws --region $(REGION) s3 sync build/ s3://$(BUCKET)

This target calls the aws s3 sync with the appropriate arguments. sync is somewhat like rsync. In this case it takes all the files in build and creates and overwrites files in the S3 bucket.

We’re almost ready for the first run! Before we can do that, we must configure our CI system to talk to AWS and to run the appropriate commands. We’ll use Semaphore for CI. It supports secret environment variables so you can safely add your AWS keys.

There are some best practices to consider when integrating AWS with other providers. First and foremost, you should create a separate IAM user with specific permissions required for the individual use case. Our cases requires access to CloudFormation and to anything used in template itself (Route53, CloudFront, and S3 in our case). You can create a specific policy that restricts based on names and things like that. This sort of fine grained access control is outside the scope of this tutorial but you must keep these things in mind. In this case, it’s good enough to create a new IAM account and grant it the built in PowerUserAcess policy. This will grant access to everything expect the ability to manage IAM users/permissions/policies. Generate the access keys and set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables in the Semaphore project settings. Now, add ansible-playbook deploy.yml build step.

Next, push your code. The initial deploy may take up to 30 minutes since creating CloudFront distributions takes sometime. Hit the domain name you used (Subdomain.TLD) in your browser and you should see the Middleman landing page.

Deploy Verification

Continous deployment is not possible without testing whether things went according to plan. So how can we do this? It’s likely that you refreshed the browser to test if the web page showed up. This is easy to automate. We’ll use a specific file known as a sentinel to test the specific deploy. Thus, the the sentinel file must be unique to each deploy. We’ll use the git commit to upload a file to the S3 bucket and test it’s readable the PublicURL output. This tests that:

1. The S3 bucket read policies are configured correctly,
2. The CloudFront distribution is configured correctly, and
3. The DNS entires are working as expected..

Let’s add tasks to the deploy playbook to verify the deploys. First we’ll use the command module to run and capture the git SHA.

	- name: Generate release tag
		command: git rev-parse --short HEAD
		register: release_tag
		changed_when: False

Now, create a temporary directory to store the sentinel file.

	- name: Create scratch dir
		command: "mktemp -d"
		register: tmp_dir
		changed_when: False

Use the previously captured SHA to create the file in the temporary directory.

	- name: Create sentinel artfiact
		copy:
			content: "{{ release_tag.stdout }}"
			dest: "{{ tmp_dir.stdout }}/sentinel.txt"
		changed_when: False

Next, upload the file to S3 to unique path using the SHA.

	- name: Upload sentinel artifact
		s3:
			mode: put
			region: "{{ aws_region }}"
			bucket: "{{ cf.stack_outputs.Bucket }}"
			src: "{{ tmp_dir.stdout }}/sentinel.txt"
			object: "_sentinel/{{ release_tag.stdout }}.txt"

Finally, make a GET request to the PublicURL to the previously mentioned path.

	- name: Test Deploy
		get_url:
			url: "http://{{ cf.stack_outputs.PublicURL }}/_sentinel/{{ release_tag.stdout }}.txt"
			dest: "{{ tmp_dir.stdout }}/verification.txt"

Commit and push the changes. Your next deploy will be verified.

Wrap Up

This is it for the first iteration. There are some areas we can improve in the second interation — e.g. we can add soome tests. Here are some things to consider testing:

• Is the CloudFormation template valid?
• Do the generated assets include all required documents, such as index.html or error.html?
• Do all the links to other pages work?

This is not an exhaustive list. Instead, it opens a dialogue on what other aspects to test before doing the deploy.

For now, you have your static site online and deployed with every commit.

Conclusion

We’ve covered a lot of ground in this tutorial. Firstly, we learned how to dockerize Middleman. Secondly, we covered how to create fast and cheap static site infrastructure using S3, CloudFront, and Route53. Then, we learned how to deploy and verify each deploy with Ansible.

If you have any questions and comments, feel free to leave them in the section below. Happy shipping!

P.S. Want to continuously deliver your applications made with Docker? Check out Semaphore’s Docker support.