Continuous Integration and Continuous Deployment

Overview

Each git commit to the master branch in GitHub triggers both the TravisCI automated tests as well as the AWS CodeBuild building. Each git tag applied to a git commit triggers a CodeBuild build.

Travis CI

Travis CI runs tests on the MozDef code base with each commit to master. The results can be seen on the Travis CI MozDef dashboard

The Test Sequence

  • Travis CI creates webhooks when first setup which allow commits to the MozDef GitHub repo to trigger Travis.
  • When a commit is made to MozDef, Travis CI follows the instructions in the .travis.yml file.
  • .travis.yml installs docker-compose in the before_install phase.
  • In the install phase, Travis runs the build-tests make target which calls docker-compose build on the docker/compose/docker-compose-tests.yml file which builds a few docker containers to use for testing.
  • In the script phase, Travis runs the tests make target which

AWS CodeBuild

Enabling GitHub AWS CodeBuild Integration

Onetime Manual Step

The steps to establish a GitHub CodeBuild integration unfortunately require a onetime manual step be done before using CloudFormation to configure the integration. This onetime manual step need only happen a single time for a given AWS Account + Region. It need not be performed with each new CodeBuild project or each new GitHub repo

  1. Manually enable the GitHub integration in AWS CodeBuild using the dedicated, AWS account specific, GitHub service user.
    1. A service user is needed as AWS CodeBuild can only integrate with GitHub from one AWS account in one region with a single GitHub user. Technically you could use different users for each region in a single AWS account, but for simplicity limit yourself to only one GitHub user per AWS account (instead of one GitHub user per AWS account per region)
    2. To do the one time step of integrating the entire AWS account in that region with the GitHub service user
      1. Browse to CodeBuild in AWS and click Create Project
      2. Navigate down to Source and set Source Provider to GitHub
      3. For Repository select Connect with a GitHub personal access token
      4. Enter the persona access token for the GitHub service user. If you haven’t created one do so and grant it repo and admin:repo_hook
      5. Click Save Token
      6. Abort the project setup process by clicking the Build Projects breadcrumb at the top. This “Save Token” step was the only thing you needed to do in that process

Grant the GitHub service user access to the GitHub repository

  1. As an admin of the GitHub repository go to that repositories settings, select Collaborators and Teams, and add the GitHub service user to the repository
  2. Set their access level to Admin
  3. Copy the invite link, login as the service user and accept the invitation

Deploy CloudFormation stack creating CodeBuild project

Deploy the mozdef-cicd-codebuild.yml CloudFormation template to create the CodeBuild project and IAM Role

The Build Sequence

  • A branch is merged into master in the GitHub repo or a version git tag is applied to a commit.
  • GitHub emits a webhook event to AWS CodeBuild indicating this.
  • AWS CodeBuild considers the Filter Groups configured to decide if the tag or branch warrants triggering a build. These Filter Groups are defined in the mozdef-cicd-codebuild.yml CloudFormation template. Assuming the tag or branch are acceptable, CodeBuild continues.
  • AWS CodeBuild reads the buildspec.yml file to know what to do.
  • The install phase of the buildspec.yml fetches packer and unzips it.
    • packer is a tool that spawns an ec2 instance, provisions it, and renders an AWS Machine Image (AMI) from it.
  • The build phase of the buildspec.yml runs the cloudy_mozdef/ci/deploy script in the AWS CodeBuild Ubuntu 14.04 environment.
  • The deploy script calls the build-from-cwd target of the Makefile which calls docker-compose build on the docker-compose.yml file, building the docker images in the AWS CodeBuild environment. These are built both so they can be consumed later in the build by packer and also for use by developers and the community.
  • deploy then calls the docker-push-tagged make target which calls
    • the tag-images make target which calls the cloudy_mozdef/ci/docker_tag_or_push tag script which applies a docker image tag to the local image that was just built by AWS CodeBuild.
    • the hub-tagged make target which calls the cloudy_mozdef/ci/docker_tag_or_push push script which
      • Uploads the local image that was just built by AWS CodeBuild to DockerHub. If the branch being built is master then the image is uploaded both with a tag of master as well as with a tag of latest.
      • If the branch being built is from a version tag (e.g. v1.2.3) then the image is uploaded with only that version tag applied.
  • The deploy script next calls the packer-build-github make target in the cloudy_mozdef/Makefile which calls the ci/pack_and_copy script which does the following steps.
    • Calls packer which launches an ec2 instance, executing a bunch of steps and and producing an AMI
    • Shares the resulting AMI with the AWS Marketplace account
    • Copies the resulting AMI to a list of additional AWS regions
    • Copies the tags from the original AMI to these copied AMIs in other regions
    • Shares the AMIs in these other regions with the AWS Marketplace account
    • Creates a blob of YAML which contains the AMI IDs. This blob will be used in the CloudFormation templates
  • When ci/pack_and_copy calls packer, packer launches an ec2 instance based on the configuration in cloudy_mozdef/packer/packer.json
  • After packer completes executing the steps laid out in packer.json inside the ec2 instance, it generates an AMI from that instance and continues with the copying, tagging and sharing steps described above.
  • Now back in the AWS CodeBuild environment, the deploy script continues by calling the publish-versioned-templates make target which runs the ci/publish_versioned_templates script which
    • injects the AMI map yaml blob produced earlier into the mozdef-parent.yml CloudFormation template so that the template knows the AMI IDs of that specific branch of code.
    • uploads the CloudFormation templates to S3 in a directory either called master or the tag version that was built (e.g. v1.2.3).