How to Use Bash Scripting for Log Analysis and Monitoring

Table of Contents

1. Understanding Logs: Formats and Importance
2. Bash Scripting Basics for Log Analysis
3. Essential Bash Tools for Log Manipulation
4. Practical Bash Script Examples for Log Analysis & Monitoring
   • 4.1 Real-Time Error Monitoring
   • 4.2 Detecting Critical Errors and Sending Alerts
   • 4.3 Trend Analysis: Tracking HTTP Status Codes
   • 4.4 Generating Daily Log Summaries
5. Best Practices for Bash Log Scripts
6. References

1. Understanding Logs: Formats and Importance

Before diving into scripting, it’s critical to understand what logs contain and how they’re structured. Logs vary by application, but most follow a consistent format with timestamps, severity levels, and contextual details.

Common Log Types and Formats

Why Log Analysis Matters

• Troubleshooting: Identify root causes of application crashes or server downtime.
• Security: Detect brute-force attacks, unauthorized access, or suspicious activity.
• Performance: Track slow requests, resource bottlenecks, or high traffic periods.
• Compliance: Maintain audit trails for regulatory requirements (e.g., GDPR, HIPAA).

2. Bash Scripting Basics for Log Analysis

If you’re new to Bash scripting, start with these foundational concepts to write effective log analysis scripts.

Shebang and Script Structure

Every Bash script starts with a shebang (#!/bin/bash) to specify the interpreter. Follow this with variables, functions, and logic.

Example skeleton:

#!/bin/bash

# Define log file path (modify this for your use case)
LOG_FILE="/var/log/apache2/access.log"

# Function to print errors
error_exit() {
  echo "$1" 1>&2
  exit 1
}

# Check if log file exists
if [ ! -f "$LOG_FILE" ]; then
  error_exit "Error: Log file $LOG_FILE not found."
fi

# Add log analysis logic here...

Key Bash Concepts for Logs

• Variables: Store log paths, thresholds, or output filenames (e.g., THRESHOLD=10 for error limits).
• Loops: Iterate over log lines (e.g., while read line; do ... done < "$LOG_FILE").
• Conditionals: Check for errors or thresholds (e.g., if grep -q "ERROR" "$LOG_FILE"; then ... fi).
• Pipes (|): Chain commands to process logs (e.g., grep "ERROR" "$LOG_FILE" | wc -l to count errors).

3. Essential Bash Tools for Log Manipulation

Bash’s real power lies in combining lightweight command-line tools to parse, filter, and analyze logs. Here are the most useful ones:

grep: Search for Patterns

Use grep to filter log lines matching a keyword or regex.

Examples:

• Find all ERROR entries:

  grep "ERROR" /var/log/app.log

• Case-insensitive search for “failed”:

  grep -i "failed" /var/log/auth.log

• Count occurrences of “500” errors (server errors) in Apache logs:

  grep -c " 500 " /var/log/apache2/access.log

awk: Process Columnar Data

awk is ideal for parsing structured logs with columns (e.g., Apache access logs). Use it to extract fields, filter, or compute aggregates.

Examples:

• Extract IP addresses from Apache logs (1st column):

  awk '{print $1}' /var/log/apache2/access.log

• Count HTTP status codes (9th column in Apache logs):

  awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c

• Filter logs from the last 24 hours (requires timestamp in [DD/Mon/YYYY:HH:MM:SS] format):

  awk -v date="$(date -d '24 hours ago' +'%d/%b/%Y:%H:%M:%S')" '$4 > "["date {print}' /var/log/apache2/access.log

sed: Transform or Clean Logs

sed (stream editor) modifies log lines in-place or filters output (e.g., remove noise, redact sensitive data).

Examples:

• Replace “192.168.1.1” with “REDACTED” in a log file:

  sed -i 's/192.168.1.1/REDACTED/g' /var/log/app.log

• Delete lines containing “DEBUG” (non-critical logs):

  sed '/DEBUG/d' /var/log/app.log

tail: Real-Time Monitoring

Use tail -f to “follow” a log file and display new lines as they’re added (critical for real-time alerts).

Example:

tail -f /var/log/syslog | grep "sshd"  # Monitor SSH activity in real time

wc, sort, and uniq: Aggregate Data

• wc -l: Count lines (e.g., total requests in a log).
• sort: Order log lines (e.g., by timestamp or IP).
• uniq -c: Count unique occurrences (e.g., top IPs making requests).

Example: Top 5 IPs with the most requests (Apache logs):

awk '{print $1}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -5

4. Practical Bash Script Examples for Log Analysis & Monitoring

Let’s put these tools together with actionable scripts for common log analysis tasks.

4.1 Real-Time Error Monitoring

This script tails a log file and highlights ERROR or CRITICAL entries in red for immediate visibility.

#!/bin/bash
# realtime_error_monitor.sh
# Usage: ./realtime_error_monitor.sh /path/to/logfile

LOG_FILE="$1"

# Check if log file is provided
if [ -z "$LOG_FILE" ]; then
  echo "Usage: $0 /path/to/logfile"
  exit 1
fi

# Tail log and highlight errors in red
tail -f "$LOG_FILE" | while read -r line; do
  if echo "$line" | grep -qiE "error|critical|failed"; then
    # Use ANSI escape codes for red text
    echo -e "\033[0;31m[ALERT] $(date +'%Y-%m-%d %H:%M:%S'): $line\033[0m"
  else
    echo "$line"
  fi
done

How to Use:

chmod +x realtime_error_monitor.sh
./realtime_error_monitor.sh /var/log/app.log

4.2 Detecting Critical Errors and Sending Alerts

Automatically check for critical errors (e.g., “Database Down”) and send an email alert using mail (configure SMTP first, or use sendmail).

#!/bin/bash
# critical_error_alert.sh
# Usage: Run via cron to check logs periodically

LOG_FILE="/var/log/app.log"
ALERT_EMAIL="[email protected]"
CRITICAL_PATTERNS=("Database Down" "Out of Memory" "Connection Refused")

# Temporary file to store errors
TMP_ERRORS=$(mktemp)

# Check for critical patterns
for pattern in "${CRITICAL_PATTERNS[@]}"; do
  grep -i "$pattern" "$LOG_FILE" >> "$TMP_ERRORS"
done

# If errors found, send email
if [ -s "$TMP_ERRORS" ]; then
  echo "Critical errors detected in $LOG_FILE:" | cat - "$TMP_ERRORS" | mail -s "ALERT: Critical Errors in $(hostname)" "$ALERT_EMAIL"
  echo "Alerts sent to $ALERT_EMAIL"
else
  echo "No critical errors found."
fi

# Cleanup temporary file
rm "$TMP_ERRORS"

Automate with Cron:
Add this to crontab -e to run every 15 minutes:

*/15 * * * * /path/to/critical_error_alert.sh >> /var/log/alert_script.log 2>&1

4.3 Trend Analysis: Tracking HTTP Status Codes

This script generates a report of HTTP status code trends (e.g., 200 OK, 404 Not Found) for Apache/Nginx logs, helping identify broken links or server issues.

#!/bin/bash
# status_code_trend.sh
# Usage: ./status_code_trend.sh /path/to/access.log [output_file]

LOG_FILE="$1"
OUTPUT_FILE="${2:-status_trend_$(date +%Y%m%d).txt}"

# Check if log file exists
if [ ! -f "$LOG_FILE" ]; then
  echo "Error: Log file $LOG_FILE not found."
  exit 1
fi

# Generate report header
echo "HTTP Status Code Trend Report" > "$OUTPUT_FILE"
echo "Log File: $LOG_FILE" >> "$OUTPUT_FILE"
echo "Generated: $(date)" >> "$OUTPUT_FILE"
echo "==============================" >> "$OUTPUT_FILE"

# Count status codes and sort by frequency (descending)
awk '{print $9}' "$LOG_FILE" | sort | uniq -c | sort -nr >> "$OUTPUT_FILE"

echo "Report saved to $OUTPUT_FILE"

Example Output:

HTTP Status Code Trend Report
Log File: /var/log/apache2/access.log
Generated: Wed Oct 11 10:00:00 2023
==============================
  15000 200
   2500 404
    300 500
     50 403

4.4 Generating Daily Log Summaries

This script compiles a daily summary of key metrics (total requests, top IPs, errors) for Apache logs, useful for capacity planning or auditing.

#!/bin/bash
# daily_log_summary.sh
# Usage: ./daily_log_summary.sh /path/to/access.log

LOG_FILE="$1"
TODAY=$(date +'%d/%b/%Y')  # Match Apache timestamp format (e.g., 11/Oct/2023)
SUMMARY_FILE="daily_summary_$TODAY.txt"

# Check log file
[ ! -f "$LOG_FILE" ] && { echo "Log file $LOG_FILE not found"; exit 1; }

# Filter logs for today
TODAY_LOGS=$(mktemp)
awk -v date="$TODAY" '$4 ~ date {print}' "$LOG_FILE" > "$TODAY_LOGS"

# Generate summary
{
  echo "Daily Log Summary: $TODAY"
  echo "==================="
  echo "Total Requests: $(wc -l < "$TODAY_LOGS")"
  echo -e "\nTop 5 Requesting IPs:"
  awk '{print $1}' "$TODAY_LOGS" | sort | uniq -c | sort -nr | head -5
  echo -e "\nHTTP Status Codes:"
  awk '{print $9}' "$TODAY_LOGS" | sort | uniq -c | sort -nr
  echo -e "\n404 Errors (Not Found):"
  grep " 404 " "$TODAY_LOGS" | awk '{print $7}' | sort | uniq -c | sort -nr | head -5
} > "$SUMMARY_FILE"

# Cleanup
rm "$TODAY_LOGS"
echo "Daily summary saved to $SUMMARY_FILE"

5. Best Practices for Bash Log Scripts

To ensure your scripts are reliable, efficient, and secure:

1. Handle Large Logs Efficiently:

   • Avoid cat "$LOG_FILE" | grep ... (use grep ... "$LOG_FILE" instead to reduce I/O).
   • Use awk or sed for filtering instead of loops to process logs faster.

2. Validate Inputs:

   • Check if log files exist ([ -f "$LOG_FILE" ]) before processing.
   • Sanitize user input to prevent path traversal attacks (e.g., LOG_FILE=$(realpath "$1")).

3. Error Handling:

   • Use set -e to exit on errors, or set -eo pipefail to catch failed pipes.
   • Add trap 'rm -f "$TMP_FILE"' EXIT to clean up temporary files.

4. Log Rotation Awareness:

   • Many logs (e.g., /var/log/syslog) are rotated by logrotate. Use tail -F (capital F) instead of tail -f to follow the new file after rotation.

5. Avoid Hardcoding:

   • Use variables for paths (e.g., LOG_DIR="/var/log") and thresholds (e.g., MAX_ERRORS=100) for easy customization.

6. Test Thoroughly:

   • Test scripts with sample logs to avoid accidental data loss (e.g., when using sed -i).
   • Use bash -n script.sh to check for syntax errors before running.

6. References

• Bash Official Documentation
• GNU grep Manual
• AWK User’s Guide
• Logrotate Configuration
• Bash Scripting Best Practices

By leveraging Bash scripting, you can transform raw logs into actionable insights with minimal effort. Start small with simple scripts (e.g., real-time error monitoring) and gradually build up to complex reports. With practice, you’ll automate away tedious tasks and gain deeper visibility into your systems.