90 Days of DevOps
Complete Guide: Setting Up Jenkins CI/CD Pipeline with Master-Agent Architecture

Complete Guide: Setting Up Jenkins CI/CD Pipeline with Master-Agent Architecture

Kanav Gathe's photo
Kanav Gathe
·

5 min read

Introduction

This guide is divided into two main parts:

  1. Part 1: Setting up Jenkins master node and deploying the application

  2. Part 2: Configuring a Jenkins agent node and migrating the deployment

Prerequisites

  • AWS account with EC2 access

  • Basic understanding of Jenkins, Docker, and Linux

  • GitHub account

  • Domain knowledge of CI/CD concepts

Part 1: Setting Up Jenkins Master and Initial Deployment

1. Master Node Setup

A. Launch EC2 Instance

  1. Launch a t2.medium Ubuntu instance

  2. Configure Security Group:

     - SSH (22)
 - Jenkins (8080)
 - Application (5000)
 - Agent communication (50000)

B. Install Required Software

# Update system packages
sudo apt-get update
sudo apt-get upgrade -y

# Install required packages
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common \
    git

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package database with Docker packages
sudo apt-get update

# Install Docker
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

# Start and enable Docker service
sudo systemctl start docker
sudo systemctl enable docker

# Add ubuntu user to docker group
sudo usermod -aG docker ubuntu

# Install Docker Compose
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

# Verify Docker installation
sudo docker --version > /home/ubuntu/docker_version.txt
sudo docker-compose --version >> /home/ubuntu/docker_version.txt

# Enable Docker system service to start on boot
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

# Install Jenkins
sudo apt install openjdk-11-jdk -y
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt update
sudo apt install jenkins -y

C. Configure Jenkins

  1. Access Jenkins at http://:8080

  2. Set up admin user and install suggested plugins

  3. Configure Docker permissions:

sudo usermod -aG docker jenkins
sudo systemctl restart jenkins

2. Initial Pipeline Setup

A. Create Pipeline

  1. Click "New Item"

  2. Select "Pipeline" and name it "Flask-App"

  3. Click OK

B. Configure Pipeline Script

pipeline {
    agent any

    stages {
        stage('Checkout') {
            steps {
                cleanWs()
                git branch: 'main', url: 'https://github.com/SlayerK15/Task-Manager.git'
            }
        }

        stage('Build with Docker Compose') {
            steps {
                script {
                    sh 'docker-compose build --no-cache'
                    sh 'docker-compose up -d'
                    sh 'sleep 30'
                }
            }
        }

        stage('Health Check') {
            steps {
                script {
                    sh '''
                        for i in {1..6}; do
                            if curl -f http://localhost:5000/; then
                                echo "Application is up and running!"
                                echo "Access the app at: http://YOUR-SERVER-IP:5000"
                                exit 0
                            fi
                            echo "Attempt $i failed, waiting..."
                            sleep 10
                        done
                        echo "Application failed to respond"
                        exit 1
                    '''
                }
            }
        }
    }

    post {
        failure {
            echo 'Pipeline failed! Check the logs for details.'
        }
        success {
            echo 'Pipeline succeeded!'
        }
    }
}

C. Test Initial Deployment

  1. Click "Build Now"

  2. Monitor build progress

  3. Access application at http://:5000

  4. Verify all functionality works

Part 2: Setting Up Jenkins Agent and Migration

1. Agent Node Setup

A. Launch Agent Instance

  1. Launch a new EC2 instance:

    • t2.micro or higher

    • Same VPC and security group as master

    • Same key pair as master

  2. Install Java:

sudo apt update
sudo apt install openjdk-11-jdk -y

  1. Create Jenkins workspace:
sudo mkdir -p /home/ubuntu/jenkins
sudo chown ubuntu:ubuntu /home/ubuntu/jenkins

2. SSH Configuration

A. Generate Keys on Master

  1. Switch to jenkins user:
sudo su - jenkins
  1. Generate SSH keys:
ssh-keygen -t rsa -b 4096
# Press Enter for default location
# Leave passphrase empty

B. Configure Agent Authentication

  1. Copy master's public key:
cat ~/.ssh/id_rsa.pub
  1. Set up on agent:
mkdir -p ~/.ssh
echo "copied_public_key" >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

3. Jenkins Agent Configuration

A. Add Node in Jenkins

  1. Go to "Manage Jenkins" → "Manage Nodes and Clouds"

  2. Click "New Node"

  3. Configure basic settings:

    • Name: Agent01

    • Type: Permanent Agent

B. Detailed Configuration

  1. Set agent details:

     - Name: Agent01
 - Description: Jenkins build agent
 - Number of executors: 1
 - Remote root directory: /home/ubuntu/jenkins
 - Labels: Agent01
 - Usage: Use this node as much as possible

  2. Configure SSH connection:

     - Launch method: Launch agents via SSH
 - Host: [Agent-EC2-IP]
 - Credentials: Add → Jenkins
   * Kind: SSH Username with private key
   * ID: agent1-ssh-key
   * Username: ubuntu
   * Private Key: Enter directly
 - Host Key Verification Strategy: Non verifying

4. Pipeline Migration

A. Update Pipeline for Agent

  1. Modify pipeline script:
pipeline {
    agent {
        label 'Agent01'
    }
    // Rest of the pipeline remains the same
}

B. Deploy on Agent

  1. Click "Build Now"

  2. Monitor build progress

  3. Verify agent connection in Node management

  4. Access application at http://:5000

5. Troubleshooting

A. Common Issues

  1. Agent Connection Issues:
# Check agent logs
tail -f /home/ubuntu/jenkins/agent.log

# Verify permissions
ls -la /home/ubuntu/jenkins
ls -la ~/.ssh
  1. Deployment Issues:
# Check Docker status
sudo systemctl status docker

# Check container logs
docker ps
docker logs <container_id>

B. Verification Steps

  1. Test SSH connection:
# From master as jenkins user
ssh ubuntu@agent-ip
  1. Verify application access:
curl http://localhost:5000

Conclusion

You now have a fully functioning Jenkins CI/CD pipeline with a master-agent setup. The application can be built and deployed on the agent node, providing better resource utilization and scalability.

Best Practices

  1. Regularly update Jenkins and plugins

  2. Monitor agent resource usage

  3. Implement proper backup strategies

  4. Maintain security best practices

  5. Document custom configurations

#90daysofdevops#90daysofdevops chanllenge#90daysofdevopschallengeJenkinsDockerAWS#TWSTrainWithShubhamDevops