Comprehensive Hands-On Guide for Linux and Bash Scripting for DevOps

This deep-dive guide will take you from a beginner to an advanced level in Linux and Bash scripting. Each step is packed with detailed explanations, practical examples, and hands-on projects.

---

Step 1: Setting Up Your Environment

Local Environment

1. Install VirtualBox or VMware on your system.

2. Download Linux distributions:

   • Beginner: Ubuntu or Linux Mint.

   • Intermediate: CentOS or Fedora.

   • Advanced: Arch Linux or Debian.

3. Allocate:

   • Minimum: 2GB RAM, 20GB storage.

   • Network: Bridged Adapter for external internet access.

4. Boot the VM and install the Linux OS.

Cloud Environment

1. Create an AWS account (free tier).

2. Launch an EC2 instance:

   • Ubuntu 20.04 or Amazon Linux 2.

   • Attach a 30GB EBS volume.

3. Connect via SSH:

    ssh -i "your-key.pem" ec2-user@<public-ip>
   

Tools to Install

Run these commands to prepare the environment:

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl wget git vim net-tools htop

---

Step 2: Linux Basics

File System and Directory Structure

1. Explore directories:

    ls /     # Root directory
    ls /var  # Variable files
    ls /etc  # Configuration files
   

2. Hands-On:

   • Navigate to /etc, list all files, and identify system configuration files.

File and Directory Commands

1. Create files and directories:

    touch file1.txt file2.txt
    mkdir project
   

2. Modify files:

    mv file1.txt project/
    cp project/file1.txt file1_copy.txt
    rm file2.txt
   

3. Hands-On:

   • Create a nested folder structure like:

       project/
       ├── src/
       └── logs/
     

   • Copy and move files into respective directories.

Permissions and Ownership

1. Understand permissions:

    ls -l  # Shows file permissions
    chmod 755 file.txt  # Change permissions
   

2. Change ownership:

    chown user:group file.txt
   

3. Hands-On:

   • Create a file and set specific permissions for a "read-only" user.

---

Step 3: Bash Scripting Basics

Your First Script

1. Create and run:

    nano hello.sh
   

   Inside the file:

    #!/bin/bash
    echo "Hello, World!"
   

   Run the script:

    chmod +x hello.sh
    ./hello.sh
   

Variables and Parameters

1. Declare variables:

    NAME="DevOps"
    echo "Welcome, $NAME"
   

2. Accept user input:

    read -p "Enter your name: " NAME
    echo "Hello, $NAME!"
   

3. Hands-On:

   • Write a script that asks for a file name and creates the file if it doesn’t exist.

Control Structures

1. Conditional statements:

    if [ -f file.txt ]; then
        echo "File exists"
    else
        echo "File does not exist"
    fi
   

2. Loops:

    for i in {1..5}; do
        echo "Iteration $i"
    done
   

---

Step 4: Intermediate Bash Scripting

System Administration Scripts

1. Automate system updates:

    #!/bin/bash
    sudo apt update && sudo apt upgrade -y
    echo "System updated successfully."
   

2. Monitor disk usage:

    df -h | grep '/dev/sd'
   

3. Hands-On:

   • Write a script that monitors CPU usage every minute and logs it.

Advanced Commands

1. Search files:

    find / -name "*.log"
   

2. Filter logs:

    grep "error" /var/log/syslog
   

3. Hands-On:

   • Write a script to search for specific patterns in a log file and extract the lines.

---

Step 5: Advanced Bash Scripting

Error Handling

1. Stop execution on error:

    set -e
   

2. Trap signals:

    trap 'echo "Script interrupted"; exit' INT
   

Functions

1. Define and use:

    function greet {
        echo "Hello, $1!"
    }
    greet "DevOps Engineer"
   

2. Hands-On:

   • Write a script with reusable functions for file management.

Networking Automation

1. Ping script:

    for host in 8.8.8.8 1.1.1.1; do
        ping -c 1 $host
    done
   

2. Check open ports:

    netstat -tuln
   

---

Step 6: Hands-On Projects

Project 1: System Health Check

Objective: Monitor server health.

1. Script requirements:

   • CPU and memory usage.

   • Disk space availability.

   • Running processes.

2. Example:

    #!/bin/bash
    echo "CPU Usage:"
    top -b -n1 | grep "Cpu(s)"
    echo "Memory Usage:"
    free -h
    echo "Disk Usage:"
    df -h
   

Project 2: Automated Backup

Objective: Automate daily backups.

1. Create a backup script:

    #!/bin/bash
    tar -czvf backup_$(date +%F).tar.gz /path/to/files
   

2. Schedule with cron:

    crontab -e
    0 2 * * * /path/to/backup.sh
   

Project 3: Log Aggregation

Objective: Fetch logs from remote servers.

1. Write a script to:

   • Use scp to fetch logs.

   • Compress and archive them.

    scp user@server:/var/log/syslog logs_$(date +%F).log
    tar -czvf logs_backup.tar.gz logs_*.log

Project 4: Web Server Automation

Objective: Install and configure a web server.

1. Automate Apache installation:

    #!/bin/bash
    sudo apt update
    sudo apt install apache2 -y
    sudo systemctl enable apache2
    sudo systemctl start apache2
    echo "Apache installed and running."
   

2. Configure virtual hosts:

   • Write a script to set up configuration files dynamically.

---

Step 7: Integrating Bash with DevOps Tools

CI/CD Pipelines

1. Use your scripts in Jenkins pipelines.

2. Example:

    pipeline {
        agent any
        stages {
            stage('Run Bash Script') {
                steps {
                    sh 'bash my_script.sh'
                }
            }
        }
    }
   

Ansible and Bash

1. Replace repetitive tasks with playbooks but use Bash for ad-hoc tasks.

2. Example:

   • Ansible can call Bash scripts for specialized configurations.

---

Continuous Practice

• Daily Tasks:

  • Automate any repetitive tasks you perform.

  • Write scripts for tasks like file cleanup, log monitoring, or package installations.

• Weekly Projects:

  • Create new scripts addressing common system admin tasks.

• Challenge Yourself:

  • Attempt real-world projects from GitHub.

By completing these hands-on tasks, projects, and integrations, you'll develop the skills needed to confidently handle Linux and Bash scripting tasks as a DevOps engineer. Let me know if you'd like detailed explanations or code samples for any specific part!

20 Hands-On DevOps Projects with Solutions

Here’s a curated list of 20 hands-on projects designed to take you from beginner to advanced in DevOps. These projects focus on tasks and challenges you’ll encounter daily in a DevOps role.

---

Beginner-Level Projects

1. System Health Monitoring Script

Objective: Create a script to monitor CPU, memory, and disk usage. Solution:

#!/bin/bash
echo "CPU Usage:"
top -b -n1 | grep "Cpu(s)"
echo "Memory Usage:"
free -h
echo "Disk Usage:"
df -h

---

2. Automated System Updates

Objective: Write a script to update system packages. Solution:

#!/bin/bash
sudo apt update && sudo apt upgrade -y
echo "System updated successfully."

---

3. User Account Management

Objective: Automate user creation and deletion. Solution:

#!/bin/bash
read -p "Enter username: " USERNAME
sudo adduser $USERNAME
echo "User $USERNAME created."

---

4. File Backup Script

Objective: Create a script to back up files and compress them. Solution:

#!/bin/bash
tar -czvf backup_$(date +%F).tar.gz /path/to/backup
echo "Backup created."

---

5. Log File Rotation

Objective: Write a script to rotate logs. Solution:

#!/bin/bash
LOG_FILE="/var/log/syslog"
mv $LOG_FILE $LOG_FILE.$(date +%F)
sudo systemctl restart rsyslog
echo "Logs rotated."

---

Intermediate-Level Projects

6. Web Server Installation

Objective: Automate the installation and configuration of Apache. Solution:

#!/bin/bash
sudo apt update
sudo apt install apache2 -y
sudo systemctl start apache2
sudo systemctl enable apache2
echo "Apache installed and running."

---

7. Disk Usage Report

Objective: Generate a report of disk usage and save it to a file. Solution:

#!/bin/bash
df -h > disk_usage_report.txt
echo "Disk usage report saved to disk_usage_report.txt"

---

8. Cron Job for Regular Backups

Objective: Automate backups using cron. Solution:

1. Create a backup script (backup.sh):

    tar -czvf /backup/backup_$(date +%F).tar.gz /path/to/data
   

2. Schedule with cron:

    crontab -e
    0 2 * * * /path/to/backup.sh
   

---

9. Service Status Checker

Objective: Monitor the status of critical services like Apache or MySQL. Solution:

#!/bin/bash
SERVICE="apache2"
if systemctl is-active --quiet $SERVICE; then
    echo "$SERVICE is running."
else
    echo "$SERVICE is not running."
fi

---

10. Network Connectivity Checker

Objective: Ping multiple servers and log results. Solution:

#!/bin/bash
for HOST in google.com aws.amazon.com; do
    ping -c 1 $HOST && echo "$HOST is reachable." || echo "$HOST is not reachable."
done

---

Advanced-Level Projects

11. CI/CD Pipeline with Jenkins

Objective: Automate deployment using Jenkins. Solution:

• Install Jenkins.

• Create a pipeline with a Jenkinsfile:

    pipeline {
        agent any
        stages {
            stage('Build') {
                steps {
                    echo "Building project..."
                }
            }
            stage('Test') {
                steps {
                    echo "Running tests..."
                }
            }
            stage('Deploy') {
                steps {
                    echo "Deploying project..."
                }
            }
        }
    }
  

---

12. Centralized Log Aggregation

Objective: Collect logs from multiple servers and compress them. Solution:

#!/bin/bash
scp user@server1:/var/log/syslog logs_server1.log
scp user@server2:/var/log/syslog logs_server2.log
tar -czvf aggregated_logs_$(date +%F).tar.gz logs_*.log

---

13. Kubernetes Cluster Setup

Objective: Automate the setup of a Kubernetes cluster. Solution:

• Use a tool like kubeadm:

    kubeadm init --pod-network-cidr=192.168.0.0/16
    kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
  

---

14. Dockerized Application Deployment

Objective: Containerize and deploy a web application. Solution:

1. Create a Dockerfile:

    FROM ubuntu
    RUN apt update && apt install apache2 -y
    CMD ["apache2ctl", "-D", "FOREGROUND"]
   

2. Build and run:

    docker build -t my-web-app .
    docker run -d -p 80:80 my-web-app
   

---

15. Load Balancer Setup

Objective: Set up an Nginx load balancer. Solution:

• Configure /etc/nginx/nginx.conf:

    upstream myapp {
        server server1:80;
        server server2:80;
    }
    server {
        listen 80;
        location / {
            proxy_pass http://myapp;
        }
    }
  

---

16. Infrastructure as Code with Terraform

Objective: Use Terraform to provision an AWS EC2 instance. Solution:

• Write a Terraform script (main.tf):

    provider "aws" {
        region = "us-west-2"
    }
    resource "aws_instance" "web" {
        ami           = "ami-0abcdef1234567890"
        instance_type = "t2.micro"
    }
  

• Deploy:

    terraform init
    terraform apply
  

---

17. Application Monitoring with Prometheus

Objective: Monitor server metrics using Prometheus and Grafana. Solution:

• Install Prometheus:

    wget https://github.com/prometheus/prometheus/releases/download/v2.0.0/prometheus-2.0.0.linux-amd64.tar.gz
    tar -xvzf prometheus-2.0.0.linux-amd64.tar.gz
    ./prometheus
  

• Configure Grafana to visualize metrics.

---

18. Automate SSL Certificate Renewal

Objective: Automate Let's Encrypt SSL renewal. Solution:

• Use Certbot:

    sudo certbot renew --quiet
  

---

19. Incident Response Automation

Objective: Create a script to restart a service automatically on failure. Solution:

#!/bin/bash
SERVICE="apache2"
if ! systemctl is-active --quiet $SERVICE; then
    systemctl restart $SERVICE
    echo "$SERVICE restarted at $(date)" >> /var/log/service_restart.log
fi

---

20. Advanced Monitoring Dashboard

Objective: Build a custom monitoring dashboard with ELK Stack. Solution:

1. Install Elasticsearch, Logstash, and Kibana.

2. Configure Logstash to parse logs:

    input {
        file {
            path => "/var/log/*.log"
        }
    }
    output {
        elasticsearch {
            hosts => ["localhost:9200"]
        }
    }
   

3. Visualize data in Kibana.

---

Next Steps

1. Practice Regularly: Deploy these solutions in real or simulated environments.

2. Enhance Projects: Add custom features like alerts, auto-scaling, or more robust error handling.

3. Document Your Work: Maintain a portfolio to showcase your skills.