How to Redirect Command Output in Docker for Logging a Flask App: A Step-by-Step Guide

Table of Contents#

1. Understanding Docker Logging Basics
2. Prerequisites
3. Step 1: Setting Up a Sample Flask App
4. Step 2: Dockerizing the Flask App (Basic Setup)
5. Step 3: Redirecting Output in Docker
   • Method 1: Shell Redirection in CMD/ENTRYPOINT
   • Method 2: Redirecting File Logs to Docker
6. Step 4: Using Docker Logging Drivers for Advanced Logging
7. Step 5: Verifying Logs
8. Best Practices for Docker Logging in Flask Apps
9. Common Pitfalls to Avoid
10. References

1. Understanding Docker Logging Basics#

Docker logs work by capturing the stdout and stderr streams of the container’s PID 1 process (the main process running in the container). By default, Docker uses the json-file logging driver, which stores logs as JSON files on the host machine (typically in /var/lib/docker/containers/<container-id>/).

Key points:

• stdout/stderr: Docker automatically captures these streams if the app writes to them.
• File logs: If your app logs to files (e.g., app.log), Docker will not capture these unless explicitly redirected.
• Logging drivers: Docker supports drivers like json-file, syslog, journald, or third-party drivers (e.g., for ELK Stack, Datadog) to route logs to external tools.

2. Prerequisites#

Before starting, ensure you have:

• Docker installed (v20.10+ recommended).
• Docker Compose (optional, for multi-container setups).
• Basic knowledge of Flask and Python.
• A terminal/command prompt to run Docker commands.

3. Step 1: Setting Up a Sample Flask App#

Let’s create a simple Flask app with logging. We’ll start with a basic app that logs to both stdout and a file to demonstrate the need for redirection.

Project Structure#

flask-docker-logging/  
├── app.py  
├── requirements.txt  
└── Dockerfile  

requirements.txt#

flask==2.3.3  

app.py#

import logging  
from flask import Flask  
 
app = Flask(__name__)  
 
# Configure logging to stdout AND a file  
stdout_handler = logging.StreamHandler()  # Logs to stdout  
file_handler = logging.FileHandler("app.log")  # Logs to app.log  
 
# Set log format and level  
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")  
stdout_handler.setFormatter(formatter)  
file_handler.setFormatter(formatter)  
 
app.logger.addHandler(stdout_handler)  
app.logger.addHandler(file_handler)  
app.logger.setLevel(logging.INFO)  # Log INFO and above  
 
@app.route("/")  
def hello():  
    app.logger.info("User accessed the root endpoint!")  # Log info message  
    app.logger.error("This is a test error!")  # Log error message  
    return "Hello, Docker Logging!"  
 
if __name__ == "__main__":  
    app.run(host="0.0.0.0", port=5000)  # Bind to all interfaces (required for Docker)  

This app logs messages to both stdout (via StreamHandler) and app.log (via FileHandler).

4. Step 2: Dockerizing the Flask App#

Next, create a Dockerfile to containerize the app. We’ll start with a basic setup and later modify it to handle redirection.

Basic Dockerfile#

# Use Python 3.9 slim image  
FROM python:3.9-slim  
 
# Set working directory  
WORKDIR /app  
 
# Copy requirements and install dependencies  
COPY requirements.txt .  
RUN pip install --no-cache-dir -r requirements.txt  
 
# Copy app code  
COPY app.py .  
 
# Command to run the app  
CMD ["python", "app.py"]  

Build and Run the Container#

Build the image and run the container:

# Build the Docker image  
docker build -t flask-log-app .  
 
# Run the container (detached mode, name it 'flask-log-demo')  
docker run -d -p 5000:5000 --name flask-log-demo flask-log-app  

Check Initial Logs#

By default, Docker captures stdout, so let’s see what logs are available:

docker logs flask-log-demo  

Expected Output (truncated):

2023-10-01 12:00:00,000 - flask.app - INFO - User accessed the root endpoint!  
2023-10-01 12:00:00,001 - flask.app - ERROR - This is a test error!  

But wait—what about the app.log file inside the container? Let’s check:

# Exec into the container to view app.log  
docker exec -it flask-log-demo cat app.log  

You’ll see the same logs in app.log, but these are not captured by Docker by default. If your app only logs to app.log (common in legacy apps), Docker logs would be empty!

5. Step 3: Redirecting Output in Docker#

To ensure Docker captures all logs (including file logs), we need to redirect output streams to Docker’s monitored stdout/stderr. Below are two common methods.

Method 1: Shell Redirection in CMD/ENTRYPOINT#

Docker’s CMD or ENTRYPOINT can use shell redirection to route logs to /proc/1/fd/1 (stdout) and /proc/1/fd/2 (stderr). These paths point to the stdout/stderr of the container’s PID 1 process (the main process Docker monitors).

Update the Dockerfile#

Modify the CMD to redirect stdout and stderr:

# ... (previous steps)  
 
# Redirect stdout/stderr to Docker's logs  
CMD ["sh", "-c", "python app.py > /proc/1/fd/1 2> /proc/1/fd/2"]  

How It Works:#

• > /proc/1/fd/1: Redirects stdout to the container’s stdout.
• 2> /proc/1/fd/2: Redirects stderr to the container’s stderr.
• sh -c: Runs the command in a shell to support redirection (required for >/2> syntax).

Method 2: Redirecting File Logs to Docker#

If your app only logs to a file (e.g., app.log), use tail -f to stream the file to stdout. However, combine this with exec to ensure proper process handling.

Update the Dockerfile#

# ... (previous steps)  
 
# Run app and tail the log file to stdout  
CMD ["sh", "-c", "python app.py & tail -f app.log > /proc/1/fd/1 2> /proc/1/fd/2"]  

⚠️ Note: Avoid backgrounding the app (&) unless necessary. A better approach is to configure the app to log to stdout (see Best Practices).

Using Docker Compose (Optional)#

If using docker-compose.yml, define the command with redirection:

version: "3.8"  
services:  
  flask-app:  
    build: .  
    ports:  
      - "5000:5000"  
    command: sh -c "python app.py > /proc/1/fd/1 2> /proc/1/fd/2"  # Redirection here  

6. Step 4: Using Docker Logging Drivers for Advanced Logging#

For production, you may want to send logs to tools like Elasticsearch, Splunk, or AWS CloudWatch. Docker’s logging drivers simplify this.

Example: Limit Log File Size with json-file#

The default json-file driver stores logs on the host. To prevent disk bloat, set limits on log size and retention:

docker run -d \  
  -p 5000:5000 \  
  --log-driver json-file \  
  --log-opt max-size=10m \  # Max log file size per container  
  --log-opt max-file=3 \     # Max number of log files to retain  
  --name flask-log-demo-advanced \  
  flask-log-app  

Example: Send Logs to Syslog#

Route logs to the host’s syslog (Linux/macOS):

docker run -d \  
  -p 5000:5000 \  
  --log-driver syslog \  
  --log-opt syslog-address=unix:///dev/log \  
  --name flask-log-syslog \  
  flask-log-app  

Check logs with:

tail -f /var/log/syslog | grep flask-log-syslog  

7. Step 5: Verifying Logs#

After updating the Dockerfile, rebuild and test:

# Stop and remove old container  
docker stop flask-log-demo && docker rm flask-log-demo  
 
# Rebuild the image  
docker build -t flask-log-app .  
 
# Run the new container  
docker run -d -p 5000:5000 --name flask-log-demo flask-log-app  
 
# Access the app to generate logs  
curl http://localhost:5000  
 
# Check Docker logs  
docker logs flask-log-demo  

Expected Output (now includes all logs):

2023-10-01 12:30:00,000 - flask.app - INFO - User accessed the root endpoint!  
2023-10-01 12:30:00,001 - flask.app - ERROR - This is a test error!  

If you used Method 2 (tailing app.log), the output will include file logs as well.

8. Best Practices for Docker Logging in Flask Apps#

1. Log to Stdout/Stderr by Default: Configure Flask to log directly to stdout using StreamHandler (avoid file logs unless necessary).

   # In app.py, use only StreamHandler  
   app.logger.addHandler(logging.StreamHandler())  

2. Use exec for Process Management: When using shell scripts in ENTRYPOINT, use exec to replace the shell with the app process (ensures PID 1 and proper signal handling):

   ENTRYPOINT ["sh", "-c", "exec python app.py"]  

3. Limit Log Size: Use the json-file driver with max-size and max-file to prevent disk overflow (see Step 4).

4. Avoid Sensitive Data: Never log passwords, API keys, or PII. Use Flask’s logger.filter to redact sensitive fields.

5. Structured Logs: Use JSON-formatted logs (e.g., with python-json-logger) for easier parsing by tools like Elasticsearch:

   from pythonjsonlogger import jsonlogger  
    
   handler = logging.StreamHandler()  
   handler.setFormatter(jsonlogger.JsonFormatter())  
   app.logger.addHandler(handler)  

9. Common Pitfalls to Avoid#

• Forgetting 2>&1: Always redirect stderr (2>) to stdout (1) to capture errors: command > stdout.log 2>&1.
• Logging to Ephemeral Storage: Container filesystems are temporary—logs in app.log will be lost when the container is deleted.
• Overusing File Logs: Relying on file logs increases complexity; prefer stdout for Docker compatibility.

10. References#

• Docker Logging Documentation
• Flask Logging Guide
• Python logging Module
• Docker Logging Drivers
• 12 Factor App: Logs

By following this guide, you’ll ensure all Flask app logs are captured, centralized, and actionable—whether running locally or in production. Happy logging! 🚢📝