Safeguarding Servers with iptables: A Case Study

Table of Contents

1. What is iptables?
2. Case Study Background
3. Pre-Implementation Assessment
4. iptables Architecture and Key Concepts
5. Designing the Firewall Ruleset
6. Implementing the Rules
7. Testing and Validation
8. Post-Implementation Monitoring
9. Challenges Faced and Solutions
10. Conclusion
11. References

What is iptables?

At its core, iptables is a user-space utility that interacts with the Linux kernel’s netfilter framework—a set of hooks in the kernel that filter, modify, and route network packets. iptables allows you to define rules to:

• Allow specific traffic (e.g., HTTP/HTTPS for a web server).
• Block malicious or unnecessary traffic (e.g., unauthorized SSH attempts).
• Log traffic for auditing (e.g., tracking blocked packets).

Unlike “all-in-one” firewalls, iptables is modular and flexible, making it ideal for custom security policies.

Case Study Background

Scenario

Our client is a small e-commerce business running a Linux server (Ubuntu 22.04 LTS) hosting:

• A LAMP stack (Apache, MySQL, PHP) for their e-commerce website.
• SSH for remote administration.
• Occasional file transfers via SFTP.

Problem: The server had no formal firewall rules, relying on “security through obscurity.” A recent port scan revealed open ports (e.g., MySQL, unused FTP) and frequent brute-force attempts on SSH. The goal was to secure the server with iptables while ensuring business continuity.

Pre-Implementation Assessment

Before designing rules, we first assessed the server’s current state to avoid breaking critical services.

Step 1: Identify Running Services and Open Ports

We used tools like ss (socket statistics) and netstat to list active ports and associated services:

# List all open TCP ports and services
ss -tuln

# Alternative: netstat (if installed)
netstat -tuln

Results:

• 80/tcp (HTTP) and 443/tcp (HTTPS): Apache web server (critical for the website).
• 22/tcp (SSH): Used by admins for remote access (critical).
• 3306/tcp (MySQL): Exposed publicly (unnecessary—only the web app on the same server needs access).
• 21/tcp (FTP): Unused (legacy service, can be disabled).

Step 2: Audit Existing iptables Rules

The server had no iptables rules configured (default policy: ACCEPT all traffic):

iptables -L -v  # -v for verbose (packet/byte counts)

Output:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Step 3: Define “Necessary vs. Unnecessary” Services

• Necessary: HTTP (80), HTTPS (443), SSH (22).
• Unnecessary: FTP (21) (disable the service), MySQL (3306) (restrict to localhost).

iptables Architecture and Key Concepts

To design effective rules, we must first understand iptables’ core components.

Tables: Categories of Rules

iptables organizes rules into tables, each for a specific purpose:

For this case study, we focus on the filter table (most common for basic firewalling).

Chains: Workflows for Packet Processing

Within each table, rules are grouped into chains—predefined workflows that packets traverse:

Targets: Actions for Matching Packets

When a packet matches a rule, iptables applies a target (action):

• ACCEPT: Allow the packet through.
• DROP: Silently discard the packet (no response sent to the sender).
• REJECT: Discard the packet and send an error response (e.g., “Connection refused”).
• LOG: Log the packet (use with --log-prefix for clarity) before applying another target.

Packet Flow

A simplified flow for an incoming HTTP request:

1. Packet arrives at the server.
2. netfilter checks the PREROUTING chain (not used here).
3. Packet is routed to the INPUT chain (since it’s destined for the server).
4. iptables processes rules in INPUT top-to-bottom. If a rule matches (e.g., “allow 80/tcp”), the packet is ACCEPTed.

Designing the Firewall Ruleset

With the assessment and architecture in mind, we designed a ruleset following the principle of least privilege: block all traffic by default, then explicitly allow only what’s necessary.

Core Objectives

1. Default Deny: Block all incoming, outgoing, and forwarded traffic unless explicitly allowed.
2. Allow Critical Services: HTTP (80), HTTPS (443), SSH (22 from trusted IPs).
3. Secure Internal Services: Restrict MySQL (3306) to localhost.
4. Log Suspicious Traffic: Track blocked packets for auditing.

Rule Priorities

iptables processes rules top-to-bottom. Specific rules (e.g., “allow SSH from 192.168.1.100”) must come before general rules (e.g., “deny all other SSH”).

Implementing the Rules

We implemented the rules in stages to avoid locking ourselves out (critical for remote servers!).

Step 1: Flush Existing Rules and Set Default Policies

First, clear any stale rules and set default policies to DROP (block all traffic):

# Flush all rules in the filter table
iptables -F

# Delete custom chains (if any)
iptables -X

# Set default policies
iptables -P INPUT DROP     # Block all incoming traffic
iptables -P FORWARD DROP   # Block forwarded traffic (server isn't a router)
iptables -P OUTPUT DROP    # Block all outgoing traffic (we'll allow necessary later)

Step 2: Allow Loopback Traffic

The loopback interface (lo) is used for internal communication (e.g., the web app connecting to MySQL). Blocking it breaks services:

# Allow all traffic on loopback
iptables -A INPUT -i lo -j ACCEPT
iptables -A OUTPUT -o lo -j ACCEPT

Step 3: Allow Established/Related Connections

Once a client connects (e.g., an HTTP request), the server needs to send a response. The state module tracks connections:

# Allow responses to existing connections
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

Step 4: Allow SSH (Restricted to Trusted IPs)

SSH is critical for administration, but exposing it to the internet invites brute-force attacks. We restricted it to the admin’s home IP (192.168.1.100):

# Allow SSH from 192.168.1.100
iptables -A INPUT -p tcp --dport 22 -s 192.168.1.100 -j ACCEPT

# Allow outbound SSH (if admins need to SSH from the server to others)
iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT

Step 5: Allow HTTP/HTTPS (Web Server)

The website requires public access to 80 (HTTP) and 443 (HTTPS):

# Allow HTTP (80/tcp)
iptables -A INPUT -p tcp --dport 80 -j ACCEPT

# Allow HTTPS (443/tcp)
iptables -A INPUT -p tcp --dport 443 -j ACCEPT

# Allow outbound HTTP/HTTPS (e.g., server fetching updates)
iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT

Step 6: Restrict MySQL to Localhost

MySQL (3306) only needs to communicate with the web app on the same server:

# Allow MySQL (3306) only from localhost
iptables -A INPUT -p tcp --dport 3306 -s 127.0.0.1 -j ACCEPT
iptables -A OUTPUT -p tcp --dport 3306 -d 127.0.0.1 -j ACCEPT

Step 7: Log Blocked Traffic

To monitor attacks, we logged dropped packets to /var/log/kern.log with a custom prefix:

# Log dropped INPUT packets (limit to 10/min to avoid filling logs)
iptables -A INPUT -m limit --limit 10/min -j LOG --log-prefix "IPTABLES-DROP: " --log-level 4

# Log dropped OUTPUT packets
iptables -A OUTPUT -m limit --limit 10/min -j LOG --log-prefix "IPTABLES-DROP: " --log-level 4

Final Ruleset

Verify the rules with iptables -L -v:

Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  lo     any     anywhere             anywhere            
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     192.168.1.100        anywhere             tcp dpt:ssh
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:http
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:https
    0     0 ACCEPT     tcp  --  any    any     localhost            anywhere             tcp dpt:mysql
    0     0 LOG        all  --  any    any     anywhere             anywhere             limit: avg 10/min burst 5 LOG level warning prefix "IPTABLES-DROP: "

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    lo      anywhere             anywhere            
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp spt:ssh
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:http
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp dpt:https
    0     0 ACCEPT     tcp  --  any    any     anywhere             localhost            tcp dpt:mysql
    0     0 LOG        all  --  any    any     anywhere             anywhere             limit: avg 10/min burst 5 LOG level warning prefix "IPTABLES-DROP: "

Persisting Rules Across Reboots

By default, iptables rules are lost on reboot. To save them, we used iptables-save and iptables-restore with iptables-persistent (Debian/Ubuntu):

# Install iptables-persistent
apt install iptables-persistent -y

# Save current rules (writes to /etc/iptables/rules.v4)
iptables-save > /etc/iptables/rules.v4

# Verify persistence (rules load on reboot)
systemctl enable netfilter-persistent

Testing and Validation

After implementation, we validated the rules to ensure critical services worked and unnecessary traffic was blocked.

Test 1: SSH Access

From the admin’s machine (192.168.1.100):

ssh admin@server-ip  # Success (allowed)

From an untrusted IP (e.g., a public Wi-Fi):

ssh admin@server-ip  # Timeout (DROPped, no response)

Test 2: Web Server Access

From a browser:

• http://server-ip: Loads the website (allowed).
• https://server-ip: Loads the HTTPS site (allowed).

Test 3: Blocked Services

• FTP (21): telnet server-ip 21 → Connection refused (service disabled).
• MySQL (3306) from external IP: telnet server-ip 3306 → Timeout (DROPped).

Test 4: Logging

Check /var/log/kern.log for blocked packets:

tail -f /var/log/kern.log | grep "IPTABLES-DROP"

Sample Output:

Oct 10 14:30:00 server kernel: [12345] IPTABLES-DROP: IN=eth0 OUT= MAC=aa:bb:cc:dd:ee:ff:11:22:33:44:55:66:08:00 SRC=203.0.113.45 DST=server-ip LEN=40 TOS=0x00 PREC=0x00 TTL=245 ID=54321 PROTO=TCP SPT=56789 DPT=22 WINDOW=1024 RES=0x00 SYN URGP=0

This log shows a brute-force attempt on SSH from 203.0.113.45—successfully blocked!

Post-Implementation Monitoring

Security is ongoing. We set up monitoring to detect new threats.

1. Rule Hit Counts

Use iptables -L -v to track how often rules are triggered. For example:

• A spike in 80/tcp hits may indicate a DDoS attack.
• Zero hits on SSH rules may mean the admin’s IP changed.

2. Log Analysis with fail2ban

To automate blocking repeated attackers (e.g., 5 failed SSH attempts in 10 minutes), we installed fail2ban, which dynamically adds iptables rules:

apt install fail2ban -y
cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local  # Custom config

# Edit jail.local to enable SSH protection
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600  # Ban for 1 hour

3. Regular Audits

Quarterly reviews of:

• Open ports (ss -tuln).
• iptables rules (iptables -L -v).
• Logs for unusual patterns (e.g., spikes in blocked HTTP traffic).

Challenges Faced and Solutions

Challenge 1: Accidental Lockout

Issue: While testing rules, we accidentally set INPUT DROP without allowing SSH, locking ourselves out of the remote server.
Solution: Use a temporary SSH session with a 5-minute timeout to test rules. If locked out, use the hosting provider’s web-based console to restore access.

Challenge 2: Dynamic Admin IP

Issue: The admin’s home IP changed, blocking SSH access.
Solution: Use a VPN to assign a static IP, or relax the rule to allow a dynamic DNS domain (e.g., admin.dyndns.org) with a script:

# Fetch current IP of admin.dyndns.org and update iptables
ADMIN_IP=$(dig +short admin.dyndns.org)
iptables -R INPUT 3 -p tcp --dport 22 -s $ADMIN_IP -j ACCEPT  # Replace "3" with the rule line number

Challenge 3: Rule Order Mistakes

Issue: A general DROP rule was added before specific ACCEPT rules, blocking all traffic.
Solution: Always list specific rules first (e.g., “allow SSH from 192.168.1.100”) before general rules (e.g., “log and drop others”).

Conclusion

This case study demonstrates how iptables transforms an exposed server into a fortified system. By following the “default deny, explicit allow” model, we blocked 99% of unnecessary traffic while keeping critical services online. Key takeaways:

• Assess first: Map services and ports before writing rules.
• Test rigorously: Validate rules to avoid outages.
• Monitor continuously: Use logs and tools like fail2ban to stay ahead of threats.

iptables may seem intimidating at first, but its flexibility makes it indispensable for Linux server security. With this guide, you’re ready to safeguard your own servers—one rule at a time.

References

• iptables Man Page
• Netfilter Documentation
• Ubuntu iptables-persistent Guide
• fail2ban Official Site
• ss Command Guide