Continuous Integration and Continuous Deployment

Overview

Each git commit to the master branch in GitHub triggers both the TravisCI automated tests as well as the AWS CodeBuild building. Each git tag applied to a git commit triggers a CodeBuild build.

Travis CI

Travis CI runs tests on the MozDef code base with each commit to master. The results can be seen on the Travis CI MozDef dashboard

The Test Sequence

  • Travis CI creates webhooks when first setup which allow commits to the MozDef GitHub repo to trigger Travis
  • When a commit is made to MozDef, Travis CI follows the instructions in the .travis.yml file
  • .travis.yml installs docker-compose in the before_install phase
  • in the install phase, Travis runs the build-tests make target which calls docker-compose build on the docker/compose/docker-compose-tests.yml file which builds a few docker containers to use for testing
  • in the script phase, Travis runs the tests make target which

AWS CodeBuild

Enabling GitHub AWS CodeBuild Integration

  • Request that a github.com/mozilla GitHub Organization owner temporarily approve / whitelist the AWS CodeBuild integration in the github.com/mozilla GitHub Organization
  • Manually configure the GitHub integration in AWS CodeBuild which will create the GitHub webhooks needed using the dedicated, AWS account specific, GitHub service user. A service user is needed as AWS CodeBuild can only integrate with GitHub from one AWS account in one region with a single GitHub user. Technically we could use different users for each region in a single AWS account, but for simplicity we’re limiting to only one GitHub user per AWS account (instead of one GitHub user per AWS account per region)
    • For the infosec-prod AWS account use the infosec-prod-371522382791-codebuild GitHub user
    • For the infosec-dev AWS account use the infosec-dev-656532927350-codebuild GitHub user
  • Request that a GitHub Organization owner, re-deny the integration for github.com/mozilla

The Build Sequence

  • A branch is merged into master in the GitHub repo or a version git tag is applied to a commit
  • GitHub emits a webhook event to AWS CodeBuild indicating this
  • AWS CodeBuild reads the buildspec.yml file to know what to do
  • The install phase of the buildspec.yml fetches packer and unzips it
    • packer is a tool that spawns an ec2 instance, provisions it, and renders an AWS Machine Image (AMI) from it.
  • The build phase of the buildspec.yml runs the cloudy_mozdef/ci/deploy script in the AWS CodeBuild Ubuntu 14.04 environment
  • The deploy script calls the build-from-cwd target of the Makefile which calls docker-compose build on the docker-compose.yml file, building the docker images in the AWS CodeBuild environment. These are built both so they can be consumed later in the build by packer and also for use by developers and the community.
  • deploy then calls the docker-push-tagged make target which calls
    • the tag-images make target which calls the cloudy_mozdef/ci/docker_tag_or_push tag script which applies a docker image tag to the local image that was just built by AWS CodeBuild.
    • the hub-tagged make target which calls the cloudy_mozdef/ci/docker_tag_or_push push script which
      • Uploads the local image that was just built by AWS CodeBuild to DockerHub. If the branch being built is master then the image is uploaded both with a tag of master as well as with a tag of latest
      • If the branch being built is from a version tag (e.g. v1.2.3) then the image is uploaded with only that version tag applied
  • The deploy script next calls the packer-build-github make target in the cloudy_mozdef/Makefile which calls the ci/pack_and_copy script which does the following steps
    • Calls packer which launches an ec2 instance, executing a bunch of steps and and producing an AMI
    • Shares the resulting AMI with the AWS Marketplace account
    • Copies the resulting AMI to a list of additional AWS regions
    • Copies the tags from the original AMI to these copied AMIs in other regions
    • Shares the AMIs in these other regions with the AWS Marketplace account
    • Creates a blob of YAML which contains the AMI IDs. This blob will be used in the CloudFormation templates
  • When ci/pack_and_copy calls packer, packer launches an ec2 instance based on the configuration in cloudy_mozdef/packer/packer.json
  • After packer completes executing the steps laid out in packer.json inside the ec2 instance, it generates an AMI from that instance and continues with the copying, tagging and sharing steps described above
  • Now back in the AWS CodeBuild environment, the deploy script continues by calling the publish-versioned-templates make target which runs the ci/publish_versioned_templates script which
    • injects the AMI map yaml blob produced earlier into the mozdef-parent.yml CloudFormation template so that the template knows the AMI IDs of that specific branch of code.
    • uploads the CloudFormation templates to S3 in a directory either called master or the tag version that was built (e.g. v1.2.3)