funwithlinux blog

How to Redirect Stdout with `find -exec` Without Creating a New Shell: Output Separate Files for Each Input

The find command is a workhorse for searching and processing files in Unix-like systems. A common task is to run a command on multiple files (e.g., converting images, extracting text, or filtering content) and save the output to a separate file for each input. However, redirecting stdout directly with find -exec often leads to unexpected behavior, such as all outputs merging into a single file. Worse, many workarounds involve spawning a subshell (e.g., sh -c), which can be inefficient for large datasets.

In this blog, we’ll explore a clean, efficient method to redirect output to separate files for each input using find -exec—without creating a new shell. We’ll break down the challenge, explain why traditional redirection fails, and demonstrate solutions using commands that natively support output file arguments.

2025-12

Table of Contents#

  1. Understanding the Challenge: Why Traditional Redirection Fails
  2. The Traditional Workaround: Subshells (and Their Drawbacks)
  3. The Solution: Use Commands with Output File Arguments
  4. Step-by-Step Examples
  5. Advanced: Custom Output Filenames (Avoiding Subshells)
  6. Troubleshooting Common Issues
  7. Conclusion
  8. References

Understanding the Challenge: Why Traditional Redirection Fails#

When using find -exec, you might naively try to redirect stdout with >, like this:

find . -name "*.txt" -exec grep "hello" {} > {}.out \;

Problem: This fails spectacularly. The shell parses the > {}.out redirection before find runs, creating a single literal file named {}.out (with curly braces). All output from grep (for every .txt file) gets appended to this one file, overwriting or merging results instead of creating separate outputs.

The Traditional Workaround: Subshells (and Their Drawbacks)#

To fix this, many users resort to spawning a subshell with sh -c:

find . -name "*.txt" -exec sh -c 'grep "hello" "$1" > "$1.out"' sh {} \;

How it works: The sh -c creates a subshell for each file. Inside the subshell, $1 expands to the filename (passed via {}), and > "$1.out" redirects output to a unique file (e.g., file.txt.out).

Drawbacks:

  • Overhead: Spawning a subshell (sh -c) for every file is inefficient, especially with thousands of files.
  • Complexity: Quoting and escaping filenames with spaces/special characters (e.g., my file.txt) requires careful handling to avoid syntax errors.

The Solution: Use Commands with Output File Arguments#

The key insight: Avoid shell redirection entirely by using commands that accept an output file as an explicit argument. Most tools (e.g., convert, gcc, rg) let you specify an output file directly, eliminating the need for >, subshells, or sh -c.

How It Works#

If your command has an option to define an output file (e.g., command input.txt -o output.txt), you can use find -exec to pass both the input file ({}) and the output file (e.g., {}.out) as arguments. This bypasses the shell’s redirection logic, so no subshell is needed.

Step-by-Step Examples#

Let’s walk through practical examples with common commands.

Example 1: Extract Text from PDFs with pdftotext#

The pdftotext tool (from poppler-utils) extracts text from PDFs and supports an output file argument:

pdftotext input.pdf output.txt  # Extracts input.pdf to output.txt

To run this on all PDFs in a directory and save outputs as file.pdf.txt:

find . -name "*.pdf" -exec pdftotext {} {}.txt \;

Result: For report.pdf, you get report.pdf.txt with the extracted text—no subshells!

Example 2: Resize Images with convert (ImageMagick)#

ImageMagick’s convert command resizes images and takes an output file:

convert input.jpg -resize 50% output.jpg  # Resizes input.jpg to 50% and saves as output.jpg

To resize all .png files and save as file.png.small.png:

find . -name "*.png" -exec convert {} -resize 50% {}.small.png \;

Result: photo.png becomes photo.png.small.png with reduced dimensions.

Example 3: Compile Source Code with gcc#

The gcc compiler accepts an output file via -o:

gcc main.c -o main.bin  # Compiles main.c to main.bin

To compile all .c files into binaries with .bin extensions:

find . -name "*.c" -exec gcc {} -o {}.bin \;

Result: app.c compiles to app.c.bin.

Example 4: Search with ripgrep (rg)#

ripgrep (rg) is a fast alternative to grep with an --output option for saving results:

rg "error" app.log --output app.log.errors  # Saves "error" matches to app.log.errors

To search all .log files and save matches:

find . -name "*.log" -exec rg "error" {} --output {}.errors \;

Result: server.log generates server.log.errors with matching lines.

Advanced: Custom Output Filenames#

What if you want to replace the input file extension (e.g., file.txt → file.out instead of file.txt.out)? Without a subshell, you can use find -execdir to simplify basename manipulation.

Example: Replace .txt with .out#

Use find -execdir to run the command in the file’s directory, making {} expand to the basename (e.g., file.txt instead of ./path/to/file.txt). Combine with a command that supports output filename templates:

find . -name "*.txt" -execdir sed "s/old/new/g" {} -i.bak \;  # In-place edit (not our goal)

For output files, if your command lacks extension replacement, you may need a subshell (but this is rare). Most tools let you construct filenames like {}.out (append) or use find -execdir for shorter paths.

Troubleshooting Common Issues#

1. Filenames with Spaces/Special Characters#

find automatically quotes {} when passing filenames to -exec, so tools like convert or rg handle spaces correctly:

# Works even for "my document.txt"
find . -name "*.txt" -exec rg "hello" {} --output {}.out \;

2. Permission Denied#

If the output file is in a read-only directory, the command will fail. Fix: Ensure the output directory is writable, or use sudo (if necessary).

3. Overwriting Existing Files#

Most commands overwrite existing files by default. To prevent this, use tools with a "no-clobber" option (e.g., rg --output has no such option, but you can pre-check with test:

# Skip if output exists (requires subshell, but useful for safety)
find . -name "*.txt" -exec sh -c 'test -f "$1.out" || rg "hello" "$1" > "$1.out"' sh {} \;

4. Commands Without Output File Options#

If your command lacks an output argument (e.g., cat, grep), you may need a subshell. For example, to use grep without rg:

# Subshell required for grep (no -o option for output files)
find . -name "*.txt" -exec sh -c 'grep "hello" "$1" > "$1.out"' sh {} \;

Conclusion#

By leveraging commands that accept output file arguments, you can use find -exec to create separate output files without subshells. This method is faster, cleaner, and avoids quoting headaches. For commands lacking output options, subshells are a fallback, but the approach above should handle most use cases.

References#