Setting Up CI/CD Pipeline: Jenkins, Docker, and GitHub on AWS EC2

A step-by-step guide to automate your application deployment using Jenkins, Docker, and GitHub webhooks.

Initial Setup: AWS EC2 Instance

Start by launching a t2.medium EC2 instance on AWS. Configure your security groups to allow:

• SSH (Port 22)

• Jenkins (Port 8080)

• Application (Port 5000)

Step 1: Installing Required Tools

Connect to your EC2 instance and install docker, docker-compose, Jenkins :

git clone https://github.com/SlayerK15/Scripts.git
cd Scripts
chmod 777 *.sh
./setup.sh docker jenkins

Here i am using my scripts that i have created for fast installation of common used Devops tools.

Verifying the Docker and Jenkins installation.

Step 2: Jenkins Setup

Access Jenkins using your EC2 IP: http://<your-ec2-ip>:8080

Get your admin password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Select 'Install suggested plugins' for the best experience.

Wait for it to install.

Configure the user.

Step 3: Pipeline Creation

Create a new pipeline:

1. Click "New Item"

2. Name: "Task-Manager-app"

3. Choose "Pipeline"

Step 4: GitHub Integration

Configure your GitHub repository with Jenkins:

1. Add repo URL: https://github.com/YourUsername/Task-Manager.git

2. Set branch specification

3. Add Jenkinsfile path

Step 5: Webhook Configuration

1. Go to the GitHub repository

2. Select the setting from the above

3.Select the Webhooks tab from the left tabs and select add webhook

4. Add webhook in GitHub:

    Payload URL: http://<your-ec2-ip>:8080/github-webhook/
    Content type: application/json
    Events: Just the push event
   

5. After adding the webhook it will show that hook was successfully created and it has never been triggered.

Step 6: Pipeline Execution

If any changes are pushed to the GitHub repository then the pipeline execution is started. In my case I added/modified the Jenkinsfile and committed the changes.

All stages should show green checkmarks indicating successful execution.

Watch your pipeline execute through different stages:

• Checkout

• Build and Deploy

• Health Check

• Post Actions

Step 7: Application Verification

Access your deployed application: http://<your-ec2-ip>:5000

Your Task Manager application is now live and will automatically update whenever you push changes to GitHub.

Common Issues & Solutions in Jenkins CI/CD Pipeline

When troubleshooting your pipeline, here are the key areas to check and their solutions:

1. Jenkins Logs

sudo tail -f /var/log/jenkins/jenkins.log

Look for error messages related to:

• Build failures

• Plugin issues

• Permission problems

2. Docker Container Status

# Check running containers
docker ps

# Check all containers including stopped ones
docker ps -a

# View container logs
docker logs <container_id>

3. Webhook Connectivity

• Check GitHub repository settings → Webhooks

• Verify recent deliveries

• Look for green checkmarks indicating successful connections

• Test webhook manually using the "Test" button

4. Security Group Settings

Ensure these ports are open:

• 22 (SSH)

• 8080 (Jenkins)

• 5000 (Application)

• Any other ports your application needs

5. Docker Permissions

Critical step: Reboot after group changes!

# Add jenkins user to docker group
sudo usermod -aG docker jenkins

# Add your user to docker group
sudo usermod -aG docker $USER

# Reboot instance
sudo reboot

Without rebooting, you might encounter:

• "Permission denied" errors

• Docker socket access issues

• Jenkins unable to execute docker commands

Quick Verification

After reboot, verify permissions:

# Check group membership
groups jenkins
groups $USER

# Test docker access
docker ps

These steps should resolve most common pipeline issues. If problems persist, check your Jenkinsfile configuration and Docker compose files.

That's it! You now have a fully automated CI/CD pipeline. Happy coding!