funwithlinux guide

Streamlining DevOps Tasks with Bash Scripting

In the fast-paced world of DevOps, efficiency and automation are the cornerstones of success. DevOps engineers and SREs (Site Reliability Engineers) spend countless hours on repetitive tasks: deploying applications, monitoring logs, managing environments, backing up data, and more. These tasks, while critical, can be error-prone and time-consuming when done manually. Enter **Bash scripting**—a lightweight, powerful tool that has been a staple in the Unix/Linux ecosystem for decades. Bash (Bourne Again Shell) is not just a command-line interpreter; it’s a scripting language that lets you automate complex workflows with minimal overhead. Whether you’re orchestrating deployments, parsing logs, or provisioning infrastructure, Bash scripts can turn tedious manual steps into repeatable, reliable processes. This blog dives deep into how Bash scripting can streamline DevOps tasks. We’ll cover essential Bash concepts, walk through practical script examples for common DevOps workflows, and share best practices to ensure your scripts are robust, maintainable, and secure.

Table of Contents

  1. Why Bash for DevOps?
  2. Essential Bash Concepts for DevOps
  3. Practical Bash Scripts for DevOps Tasks
  4. Best Practices for DevOps Bash Scripts
  5. Advanced Tips for Power Users
  6. Conclusion
  7. References

Why Bash for DevOps?

Before diving into scripts, let’s clarify why Bash is a go-to tool for DevOps:

  • Ubiquity: Bash is pre-installed on nearly all Linux and macOS systems. No need for extra dependencies—just open a terminal and start scripting.
  • Simplicity: Bash syntax is straightforward for small to medium tasks. You don’t need to learn a full programming language to automate routine workflows.
  • Integration with CLI Tools: DevOps relies heavily on command-line tools (e.g., docker, kubectl, aws, git). Bash seamlessly pipes outputs between these tools, turning them into building blocks for automation.
  • Speed: Bash scripts execute quickly, making them ideal for time-sensitive tasks like log monitoring or deployment checks.
  • Flexibility: From one-liners to complex multi-step workflows, Bash scales to fit your needs. It’s equally useful for prototyping and production-grade automation.

Essential Bash Concepts for DevOps

To write effective DevOps scripts, you’ll need to master these core Bash concepts:

1. Variables

Store data (e.g., paths, API keys) for reuse. Use = to assign values (no spaces!), and $ to reference them.

APP_DIR="/opt/myapp"
LOG_FILE="$APP_DIR/app.log"
echo "Application logs stored at: $LOG_FILE"

2. Command Substitution

Capture the output of a command into a variable using $(command) or backticks `command` (prefer $() for readability).

CURRENT_DATE=$(date +%Y-%m-%d)  # Stores "2024-05-20"
GIT_COMMIT=$(git rev-parse --short HEAD)  # Stores the latest Git commit hash

3. Conditionals (if-else)

Control flow based on conditions (e.g., “if a file exists, skip download”). Use [ ] (POSIX) or [[ ]] (Bash-specific, supports regex and globbing).

if [[ -f "$LOG_FILE" ]]; then  # Check if log file exists
  echo "Log file found: $LOG_FILE"
else
  echo "Log file missing! Creating..."
  touch "$LOG_FILE"
fi

4. Loops

Repeat actions (e.g., “restart all services” or “backup all databases”). Common loop types: for, while, and until.

# For loop: Iterate over a list of services
SERVICES=("api" "db" "web")
for service in "${SERVICES[@]}"; do
  systemctl restart "$service"
  echo "Restarted $service"
done

# While loop: Tail logs until "success" is found
while read -r line; do
  if echo "$line" | grep -q "success"; then
    echo "Deployment succeeded!"
    break
  fi
done < "$LOG_FILE"

5. Pipes and Redirection

Chain commands or redirect output to files. Use | to pipe output from one command to another, > to overwrite a file, and >> to append.

# Pipe: Search logs for errors and count occurrences
grep "ERROR" "$LOG_FILE" | wc -l  # Output: Number of ERROR lines

# Redirect: Save deployment logs to a file
./deploy.sh > "deploy_$CURRENT_DATE.log" 2>&1  # 2>&1 redirects errors to the same file

6. Exit Codes

Every command returns an exit code (0 = success, non-zero = failure). Use $? to check the exit code of the last command, or set -e to make the script exit on any error.

set -e  # Exit script if any command fails
./build.sh  # Script exits here if build fails
./deploy.sh  # Only runs if build succeeded

7. Functions

Reuse code with functions. Define them with function_name() { ... } and call them like commands.

log_error() {
  local message="$1"
  echo "[$(date +%H:%M:%S)] ERROR: $message" >> "$LOG_FILE"
  exit 1  # Exit after logging critical errors
}

# Usage: log_error "Database connection failed"

Practical Bash Scripts for DevOps Tasks

Now, let’s apply these concepts to real-world DevOps scenarios. Each script includes a goal, code, key explanations, and usage instructions.

1. Automated Application Deployment

Goal: Deploy a Node.js app from Git: pull latest code, install dependencies, build, restart the service, and verify success.

#!/bin/bash
set -euo pipefail  # Exit on error, undefined variable, or failed pipe

# Configuration
APP_NAME="my-node-app"
APP_DIR="/opt/$APP_NAME"
GIT_REPO="https://github.com/your-username/$APP_NAME.git"
SERVICE_NAME="$APP_NAME.service"
LOG_FILE="/var/log/$APP_NAME/deploy.log"

# Ensure log directory exists
mkdir -p "$(dirname "$LOG_FILE")"

# Function to log messages
log() {
  echo "[$(date +%Y-%m-%d %H:%M:%S)] $1" >> "$LOG_FILE"
}

log "Starting deployment of $APP_NAME..."

# Pull latest code
log "Pulling latest code from Git..."
cd "$APP_DIR" || log_error "Directory $APP_DIR not found"
git pull origin main >> "$LOG_FILE" 2>&1

# Install dependencies
log "Installing dependencies..."
npm install --production >> "$LOG_FILE" 2>&1

# Build app (if needed)
log "Building app..."
npm run build >> "$LOG_FILE" 2>&1

# Restart service
log "Restarting $SERVICE_NAME..."
systemctl restart "$SERVICE_NAME" >> "$LOG_FILE" 2>&1

# Verify deployment
log "Verifying deployment..."
if systemctl is-active --quiet "$SERVICE_NAME"; then
  log "Deployment succeeded! $APP_NAME is running."
  echo "Deployment successful. Check logs at $LOG_FILE"
else
  log_error "Deployment failed! $SERVICE_NAME is not active."
fi

Key Features:

  • set -euo pipefail: Makes the script robust by exiting on errors, undefined variables, or failed pipes.
  • log() function: Centralizes logging with timestamps for debugging.
  • Idempotent: Safe to run multiple times (e.g., git pull works even if already up-to-date).
  • Verification step: Ensures the service is active after deployment.

Usage:
Save as deploy_app.sh, make executable with chmod +x deploy_app.sh, and run with sudo ./deploy_app.sh.

2. Log Monitoring and Alerting

Goal: Tail application logs, detect critical errors, and send alerts to Slack.

#!/bin/bash
set -uo pipefail

# Configuration
LOG_FILE="/opt/my-app/app.log"
ERROR_PATTERN="CRITICAL ERROR|FAILED TO CONNECT"  # Regex pattern to match
SLACK_WEBHOOK="https://hooks.slack.com/services/YOUR_SLACK_WEBHOOK"
CHECK_INTERVAL=5  # Seconds between checks

# Function to send Slack alert
send_slack_alert() {
  local error_message="$1"
  curl -X POST -H "Content-type: application/json" \
    --data "{\"text\":\"🚨 *Error Detected in $LOG_FILE* 🚨\n$error_message\"}" \
    "$SLACK_WEBHOOK"
}

log "Starting log monitor for $LOG_FILE..."
echo "Monitoring $LOG_FILE for errors (Ctrl+C to stop)..."

# Tail log and check for errors
tail -n 0 -f "$LOG_FILE" | while read -r line; do
  if echo "$line" | grep -qiE "$ERROR_PATTERN"; then  # -i: case-insensitive, -E: regex
    echo "Critical error found: $line"
    send_slack_alert "$line"
  fi
done

Key Features:

  • tail -f: Follows the log file in real time.
  • Regex pattern matching: Catches multiple error types (e.g., “CRITICAL ERROR” or “FAILED TO CONNECT”).
  • Slack integration: Uses curl to send alerts via Slack webhook.

Usage:
Replace SLACK_WEBHOOK with your Slack incoming webhook URL. Run with ./monitor_logs.sh.

3. Environment Setup and Dependency Management

Goal: Automatically set up a new DevOps workstation: install Docker, Kubernetes, Python, and configure Git.

#!/bin/bash
set -euo pipefail

# Configuration
USER="devops-user"
GIT_EMAIL="[email protected]"
GIT_NAME="DevOps Bot"

log() {
  echo "[$(date +%H:%M:%S)] $1"
}

log "Starting environment setup..."

# Update package lists
log "Updating packages..."
sudo apt update -y

# Install Docker
if ! command -v docker &> /dev/null; then
  log "Installing Docker..."
  sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
  sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
  sudo apt update -y
  sudo apt install -y docker-ce
  sudo usermod -aG docker "$USER"  # Add user to docker group
else
  log "Docker already installed. Skipping..."
fi

# Install Kubernetes (kubectl)
if ! command -v kubectl &> /dev/null; then
  log "Installing kubectl..."
  curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
  chmod +x kubectl
  sudo mv kubectl /usr/local/bin/
else
  log "kubectl already installed. Skipping..."
fi

# Install Python 3
if ! command -v python3 &> /dev/null; then
  log "Installing Python 3..."
  sudo apt install -y python3 python3-pip
else
  log "Python 3 already installed. Skipping..."
fi

# Configure Git
log "Configuring Git..."
git config --global user.email "$GIT_EMAIL"
git config --global user.name "$GIT_NAME"

log "Environment setup complete! Logout and back in for Docker group changes to take effect."

Key Features:

  • Idempotent: Checks if tools are already installed with command -v to avoid redundant installs.
  • User-specific config: Adds the user to the docker group and sets Git credentials.
  • Package manager agnostic: Modify for yum (RHEL/CentOS) by replacing apt with yum.

Usage:
Run with sudo ./setup_environment.sh.

4. Database Backup and Rotation

Goal: Backup a MySQL database, compress it, upload to AWS S3, and delete backups older than 7 days.

#!/bin/bash
set -euo pipefail

# Configuration
DB_NAME="myappdb"
DB_USER="backup-user"
DB_PASS="secure-password"  # Use environment variables in production!
BACKUP_DIR="/var/backups/mysql"
S3_BUCKET="myapp-backups"
RETENTION_DAYS=7  # Delete backups older than this

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Generate backup filename with timestamp
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_FILE="$BACKUP_DIR/$DB_NAME_$TIMESTAMP.sql.gz"

log() {
  echo "[$(date +%H:%M:%S)] $1"
}

log "Starting backup of $DB_NAME..."

# Dump MySQL database and compress
log "Creating database dump..."
mysqldump -u "$DB_USER" -p"$DB_PASS" "$DB_NAME" | gzip > "$BACKUP_FILE"

# Verify backup was created
if [[ ! -f "$BACKUP_FILE" || $(du -b "$BACKUP_FILE" | cut -f1) -eq 0 ]]; then
  log "ERROR: Backup file missing or empty!"
  exit 1
fi

# Upload to S3
log "Uploading to S3 bucket $S3_BUCKET..."
aws s3 cp "$BACKUP_FILE" "s3://$S3_BUCKET/mysql/"

# Delete old backups locally
log "Deleting local backups older than $RETENTION_DAYS days..."
find "$BACKUP_DIR" -name "$DB_NAME_*.sql.gz" -type f -mtime +"$RETENTION_DAYS" -delete

# Delete old backups in S3 (optional)
log "Deleting S3 backups older than $RETENTION_DAYS days..."
aws s3 ls "s3://$S3_BUCKET/mysql/" | grep "$DB_NAME" | while read -r line; do
  FILE_DATE=$(echo "$line" | awk '{print $1}')
  FILE_NAME=$(echo "$line" | awk '{print $4}')
  if [[ $(date -d "$FILE_DATE" +%s) -lt $(date -d "$RETENTION_DAYS days ago" +%s) ]]; then
    aws s3 rm "s3://$S3_BUCKET/mysql/$FILE_NAME"
  fi
done

log "Backup completed successfully! File: $BACKUP_FILE"

Key Features:

  • Compression: Uses gzip to reduce backup size.
  • S3 integration: Requires AWS CLI configured with permissions to upload to the bucket.
  • Retention policy: Cleans up old backups to save disk space and S3 costs.

Usage:
Replace credentials with environment variables (e.g., DB_PASS=$DB_PASSWORD) for security. Run with ./backup_db.sh, or schedule with cron for daily backups:

# Add to crontab (run daily at 2 AM)
0 2 * * * /path/to/backup_db.sh >> /var/log/db_backup.log 2>&1

Best Practices for DevOps Bash Scripts

To ensure your scripts are reliable and maintainable, follow these best practices:

  1. Start with a Shebang: Always use #!/bin/bash (not #!/bin/sh) to specify Bash as the interpreter.
  2. Enable Strict Mode: Use set -euo pipefail to catch errors early:
    • -e: Exit on any command failure.
    • -u: Treat undefined variables as errors.
    • -o pipefail: Exit if any command in a pipe fails.
  3. Comment Liberally: Explain why (not just what) the script does. Example: 
    # Wait 10s for DB to initialize (required for first run after provisioning)
sleep 10
  4. Handle Errors Gracefully: Use trap to clean up resources on exit, or log_error functions to exit with context: 
    trap 'log_error "Script failed at line $LINENO"' ERR
  5. Validate Inputs: Check for required arguments or dependencies before running: 
    if [[ $# -eq 0 ]]; then
  echo "Usage: $0 <environment> (e.g., production)"
  exit 1
fi
  6. Log Everything: Redirect output to a log file (e.g., >> /var/log/deploy.log 2>&1) for debugging.
  7. Test Scripts: Use tools like shellcheck to lint scripts, or write unit tests with bats.
  8. Version Control: Store scripts in Git (e.g., alongside your application code) for traceability.

Advanced Tips for Power Users

Take your Bash scripting to the next level with these pro tips:

  • Use getopts for CLI Arguments: Parse flags like --env production or -v (verbose) with getopts: 
    while getopts "e:v" opt; do
  case $opt in
    e) ENV="$OPTARG" ;;
    v) VERBOSE=1 ;;
    \?) echo "Invalid option: -$OPTARG" >&2 ;;
  esac
done
  • Source Config Files: Store environment-specific variables in a separate config.env file and source it: 
    source ./config.env  # Loads DB_USER, DB_PASS, etc.
  • Parallel Execution: Speed up tasks with xargs -P (parallel) or GNU Parallel: 
    # Backup 4 databases in parallel
echo -e "db1\ndb2\ndb3\ndb4" | xargs -I {} -P 4 ./backup_single_db.sh {}
  • JSON/YAML Parsing: Use jq (JSON) or yq (YAML) to parse config files: 
    # Get API URL from config.json
API_URL=$(jq -r '.api.url' config.json)

Conclusion

Bash scripting is a Swiss Army knife for DevOps engineers. It transforms repetitive, error-prone tasks into automated, reliable workflows—all without leaving the terminal. From deploying apps to backing up databases, Bash scripts save time, reduce human error, and ensure consistency across environments.

Start small: automate one task (e.g., log monitoring), then build a library of scripts for your team. Combine Bash with tools like jq, aws cli, or kubectl to tackle even more complex workflows. With the best practices outlined here, your scripts will be maintainable, secure, and ready for production.

References