Setting Up a Private CI/CD Pipeline in Azure—Part 1

Listen to this article:

0:00

Note: This article was adapted from content originally written on October 19th, 2017, titled “Setting up a Private CI/CD Solution in Azure.” It has been simplified and split into four parts for easier reading.

Part 1: Introduction and Solution Overview

Key Takeaways

This article introduces a private CI/CD solution in Azure, tailored for companies adopting DevOps and Agile principles.
The solution architecture includes services, networks, and tools like GitLab, Jenkins, and Ephemeral Build Slaves for optimal functionality.
All virtual machines are manually created and configured for complete control, which may require extra effort compared to using native services.
Readers will continue to Part 2, focusing on setting up Azure infrastructure and networking for the CI/CD solution.

Introduction

Recently, I was tasked with creating a private CI/CD solution in the cloud. As more companies adopt DevOps, rapid iteration, Agile, and Lean Startup principles, having a versatile CI/CD solution that covers the entire spectrum of software development becomes essential.

One of my constraints was to create this solution in the cloud. I chose Azure for its simplicity and power. In this series, you’ll learn how to create an entire CI/CD solution in Azure that remains private to the outside world and discover how Azure makes building this solution straightforward.

Important Notes

This article will be updated to include information on setting up SonarQube for code analysis.
While the exact steps may change as Microsoft updates Azure, the concepts should be easily adaptable to any UI changes.
Please refer to my GitHub page for all the resources needed to implement this solution. Specific GitHub repositories will be referenced throughout the series.

Solution Overview

The architecture of this solution encompasses various services, networks, network security groups, load balancers, virtual machines, and more. The core CI/CD functionality comes from the combination of GitLab, Jenkins, and Ephemeral Build Slaves. All components have been configured to support a clustered, high-availability environment, which is crucial for enterprise deployments.

It’s important to note that to maintain full control, all virtual machines are created manually with software explicitly installed and configured. While this approach requires more work than using native services, it provides the benefit of ultimate control over your environment. You’re welcome to adapt these instructions to use native services if desired.

Main Components

Let’s explore the main components that comprise this CI/CD solution:

Azure Subscription

Whether you decide to use MSDN Azure credits or pay-as-you-go pricing, an Azure subscription is required to provision assets.

VPN Gateway

This component allows secure connection to the private virtual network via a VPN client.

Configuration Details:

Name: Spacely-Engineering-Private-Gateway
Public IP Address: [Configured during setup]

Private Networks & Address Space

Private networks and address spaces allow for network partitioning to increase security and logically organize network assets.

Network Configuration:

Virtual Network

Address Range: 10.0.0.0/20

VPN Gateway Subnet

Address Range: 10.0.255.224/27

VPN Clients

Address Range: 172.16.0.0/24

DMZ

Address Range: 10.0.250.0/24

Private Network #1

Address Range: 10.0.0.0/24

Private Network #2

Address Range: 10.0.1.0/24

Spacely Engineering Docker Overlay Network

Address Range: 172.16.255.0/24

Docker Swarm Manager VMs

Ubuntu VMs configured as Docker Swarm Managers provide high availability for the orchestration layer.

Manager Node 1:

Name: Spacely-Engineering-VM-001
VM Size: Standard DS2 v2
Private IP: 10.0.0.4
DNS Name: spacely-eng-vm-001.example.com
OS: Ubuntu 16.04 LTS
Username: spacely-eng-admin

Manager Node 2:

Name: Spacely-Engineering-VM-002
VM Size: Standard DS2 v2
Private IP: 10.0.0.5
DNS Name: spacely-eng-vm-002.example.com
OS: Ubuntu 16.04 LTS
Username: spacely-eng-admin

Manager Node 3:

Name: Spacely-Engineering-VM-003
VM Size: Standard DS2 v2
Private IP: 10.0.0.6
DNS Name: spacely-eng-vm-003.example.com
OS: Ubuntu 16.04 LTS
Username: spacely-eng-admin

Docker Swarm Worker VMs

Ubuntu VMs configured as Docker Swarm Workers, dedicated to running Jenkins jobs.

Worker Node 1:

Name: Spacely-Engineering-VM-004
VM Size: Standard DS3 v2
Private IP: 10.0.0.7
DNS Name: spacely-eng-vm-004.example.com
OS: Ubuntu 16.04 LTS
Username: spacely-eng-admin

Worker Node 2:

Name: Spacely-Engineering-VM-005
VM Size: Standard DS3 v2
Private IP: 10.0.1.4
DNS Name: spacely-eng-vm-005.example.com
OS: Ubuntu 16.04 LTS
Username: spacely-eng-admin

Load Balancers

Internal load balancers distribute traffic across services for high availability.

GitLab Load Balancer:

Type: Internal
Frontend IP: 10.0.250.10
Backend Pool: Docker Swarm Manager VMs
Health Probe Port: 10080
Load Balancing Port: 80 → 10080

Jenkins Load Balancer:

Type: Internal
Frontend IP: 10.0.250.11
Backend Pool: Docker Swarm Manager VMs
Health Probe Port: 18080
Load Balancing Port: 80 → 18080

Docker Registry Load Balancer:

Type: Internal
Frontend IP: 10.0.250.12
Backend Pool: Docker Swarm Manager VMs
Health Probe Port: 5000
Load Balancing Port: 443 → 5000

Internal Private DNS Zone

The internal DNS zone provides name resolution for services within the private network.

Configuration:

Zone Name: example.com
DNS Server: Docker Swarm Manager VMs running Bind9

DNS Entries:

gitlab.example.com → 10.0.250.10
jenkins.example.com → 10.0.250.11
docker-registry.example.com → 10.0.250.12

Core Services

These services form the backbone of the CI/CD pipeline:

GitLab Community Edition:

Version control system
Issue tracking and project management
Merge request workflow

Jenkins with Blue Ocean:

Continuous integration server
Pipeline orchestration
Build automation

Docker Registry v2:

Private container image storage
Secure image distribution

Portainer:

Docker Swarm management UI
Container monitoring and control

Bind9 DNS Server:

Internal DNS resolution
Service discovery

Next Steps

Now that we’ve covered the introduction and solution overview, you have a solid understanding of the architecture and components involved in this private CI/CD solution.

Continue to Part 2: Azure Infrastructure and Networking Setup, where we’ll dive into the detailed steps for setting up the Azure infrastructure, including resource groups, virtual networks, and VPN gateway configuration.

This is Part 1 of a 4-part series on setting up a private CI/CD solution in Azure.