In this blog, I will describe how\u00a0<\/span>to build<\/span>\u00a0the core infrastructure in Amazon Web Services (AWS) to support our Continuous Integration platform. <\/span><\/p>\n We will refer to this as the \u201cLanding Zone,\u201d which will consist of components like VPCs, Subnets, Gateways and Routing Tables, Security and Identity Management, and more.<\/span><\/p>\n The platform architecture I\u2019ll build to support the CI system is represented in the following diagram.\u00a0<\/strong><\/p>\n At the end of this series, the\u00a0final\u00a0platform\u00a0will consist of\u00a0a Jenkins cluster with a dedicated Master and multiple slaves configured in an autoscaling group,\u00a0a GitLab code repository deployed in a HA configuration with external Elastic File System (EFS) volumes, a hosted Redis cluster,\u00a0<\/strong>a multi-AZ Postgresql database, and an Elastic Container Registry to store Docker images.<\/strong> <\/span><\/p>\n The Jenkins Master and GitLab servers sit behind an application load balancer with public access, allowing developers to remotely manage the code base and manage the Jenkins pipelines.\u00a0<\/span>\u00a0<\/span><\/p>\n The cluster of application servers will consist of an autoscaling group of Jenkins masters, Jenkins slaves and Gitlab servers deployed in private subnets behind an application load balancer. The autoscaling configuration will maintain one Jenkins master, two Jenkins Slaves, and two GitLab instances at any given time. <\/span><\/p>\n During deployment and initial configuration, only a single GitLab server will be deployed. Once GitLab has been configured for high availability, we will scale up the autoscaling group from one to two servers to provide redundancy.<\/span>\u00a0<\/span><\/p>\n For secure external access to the internet from <\/span>instances within the\u00a0<\/span>private subnets, a NAT gateway will be deployed into a public subnet.\u00a0<\/span>The private instance will use the NAT gateway to get out to the internet\u00a0<\/span>for things such as package and patch installs.<\/span>\u00a0<\/span>\u00a0<\/span><\/p>\n Finally, a Linux bastion host will be deployed in a public subnet to allow access to EC2 instances in the private subnets from external users.<\/span>\u00a0<\/span><\/p>\n The entire platform will be scripted and stored as infrastructure as code so that it can be recovered quickly or deployed in another region in the event of a disaster.<\/span>\u00a0<\/span><\/p>\n The deployment\u00a0<\/span>I\u2019ll build in this initial part of the blog series\u00a0<\/span>will consist of<\/span>\u00a0the following services. This will provide the foundation for the service I\u2019ll be adding later in this blog series.<\/span>\u00a0<\/span><\/p>\n To get started building this environment, you will need the\u00a0AWS CLI<\/a>\u00a0and\u00a0terraform<\/a>\u00a0installed locally on your workstation and create an\u00a0IAM access key<\/a>\u00a0for an AWS user with administrator access.\u00a0<\/strong><\/p>\n We will not cover how to install these tools or get into details of terraform syntax. To learn more about terraform, see\u00a0<\/span>terraform.io<\/span><\/a>. To learn more about AWS CLI, see\u00a0<\/span>AWS CLI<\/span><\/a>.<\/span>\u00a0We will focus on a practical application of these tools only.<\/span>\u00a0<\/span><\/p>\n An IAM user can be setup from the AWS console and will require programmatic access in order to use CLI commands. It is recommended to first create a group with Administrative access credentials, then add a new user to this group. As you create the group, save the generated access key which will be required when configuring the AWS CLI.<\/span>\u00a0<\/span><\/p>\n See below for a few screenshots modeling the creation of an administrative terraform user with programmatic access:\u00a0<\/strong><\/p>\n As mentioned before, save your Access key ID and Secret access key. A downloadable csv file containing the credentials is also provided when creating the AWS user. <\/span><\/span><\/p>\n These keys will be used when initializing your AWS CLI for access to your AWS account.<\/span><\/span>\u00a0<\/span><\/p>\n In addition to the administrative account, you will need\u00a0a<\/strong>\u00a0second user created with access to Elastic Container Registry used to push images from Jenkins to the Elastic Container Registry (ECR).\u00a0<\/strong> <\/span><\/p>\n Create another user using the same procedure as above, but do not add the user to a group.\u00a0 <\/span><\/p>\n Simple create a user with Programmatic access and attach two policies: A<\/span>mazonEC2ContainerServiceFullAccess<\/span>,\u00a0<\/span>AmazonEC2ContainerRegistryPowerUser<\/span>. Use the User ID: \u201cecr-remote-pusher\u201d. When complete download and save the Access Keys, which will later be used to create a Jenkins security credential.<\/span>\u00a0<\/span><\/p>\n Installation of the tools is beyond the scope of this series, but links to the software are provided. After installing the AWS CLI, GIT CLI and terraform CLI, complete the following tasks:<\/span>\u00a0<\/span><\/p>\n As a good practice, we would also recommend installing\u00a0<\/span>git<\/span><\/a>\u00a0command-line tools locally and have access to a code repository where you can store, protect and version control the scripts you will be developing.<\/span>\u00a0<\/span><\/p>\n![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-1.png\")
<\/a><\/p>\nA Look Ahead<\/h2>\n
Deployment\u00a0Part\u00a01<\/h2>\n
\n
![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-2.png\")
<\/a>\u00a0<\/span><\/p>\n![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-3.png\")
<\/a><\/p>\n![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-4.png\")
<\/a><\/p>\n![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-5.png\")
<\/a><\/p>\nInstall Terraform, AWS CLI and GIT<\/span>\u00a0<\/span><\/h2>\n
\n
Verify\u00a0<\/span><\/span>AWS CLI<\/span><\/span>\u00a0installation:<\/span><\/span>\u00a0<\/span><\/h4>\n
![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-6.png\")
<\/p>\nVerify terraform Installation:<\/span><\/span>\u00a0<\/span><\/h4>\n
![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-7.png\")
<\/a><\/p>\nVerify\u00a0<\/span><\/span>git<\/span><\/span>\u00a0Installation:<\/span><\/span>\u00a0<\/span><\/h4>\n
![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-8.png\")
<\/a><\/p>\nConfigure\u00a0<\/span><\/span>AWS CLI<\/span><\/span>\u00a0profile:<\/span><\/span>\u00a0<\/span><\/h4>\n
![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-9.png\")
<\/a><\/p>\n![\"Terraform\"](\"https:\/\/centricconsulting.com\/wp-content\/uploads\/2019\/02\/CI-Pipeline-Blog-1-Image-10.png\")
<\/a><\/p>\n