funwithlinux blog

How to Preserve Newlines in Shell Script Variables When Echoing Command Output

If you’ve ever written a shell script, you’ve likely encountered a frustrating issue: storing multi-line command output in a variable, only to have the newlines disappear when you echo it back. For example, running ls -l might produce neatly formatted lines, but when you save that output to a variable and print it, everything becomes a jumbled mess of spaces.

Newlines are critical for readability (e.g., log files, formatted reports) and data integrity (e.g., processing multi-line records). Losing them can break scripts that rely on line-by-line parsing. In this blog, we’ll demystify why newlines vanish, explore practical solutions to preserve them, and highlight common pitfalls to avoid.

2026-01

Table of Contents#

  1. Understanding the Problem: When Newlines Disappear
  2. Why Do Newlines Get Lost? The Role of Word Splitting and IFS
  3. Solutions to Preserve Newlines
  4. Common Pitfalls to Avoid
  5. Conclusion
  6. References

Understanding the Problem: When Newlines Disappear#

Let’s start with a concrete example. Suppose you run ls -l in a directory with three files:

$ ls -l
total 0
-rw-r--r-- 1 user user 0 Jan 1 12:00 file1.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file2.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file3.txt

The output has 4 lines (including the total 0 line). Now, save this output to a variable and echo it:

$ ls_output=$(ls -l)  # Store output in variable
$ echo $ls_output     # Echo without quotes
total 0 -rw-r--r-- 1 user user 0 Jan 1 12:00 file1.txt -rw-r--r-- 1 user user 0 Jan 1 12:00 file2.txt -rw-r--r-- 1 user user 0 Jan 1 12:00 file3.txt

The newlines are gone! The output is a single line with spaces separating what were once lines. Why does this happen?

Why Do Newlines Get Lost? The Role of Word Splitting and IFS#

The root cause lies in two shell behaviors: word splitting and the Internal Field Separator (IFS).

Word Splitting#

When a variable is expanded without quotes (e.g., echo $ls_output), the shell splits the expanded value into "words" based on the IFS. These words are then passed as separate arguments to the command (in this case, echo).

IFS (Internal Field Separator)#

IFS is a shell variable that defines the delimiters used for word splitting. By default, IFS includes:

  • Spaces ( )
  • Tabs (\t)
  • Newlines (\n)

Thus, when you run echo $ls_output, the shell treats newlines (and spaces/tabs) as word separators. It splits the variable’s content into individual words and passes them to echo, which prints them separated by a single space.

Solutions to Preserve Newlines#

Let’s explore proven methods to preserve newlines when working with shell variables.

1. Double-Quoting the Variable Expansion#

The simplest and most reliable fix is to double-quote the variable expansion. When you wrap a variable in double quotes (e.g., echo "$ls_output"), the shell skips word splitting, treating the entire variable content as a single argument.

Example:

$ ls_output=$(ls -l)
$ echo "$ls_output"  # Note the double quotes
total 0
-rw-r--r-- 1 user user 0 Jan 1 12:00 file1.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file2.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file3.txt

Why it works: Double quotes prevent word splitting, so the shell passes the variable’s content (including newlines) as a single string to echo, which prints it exactly as stored.

2. Using printf Instead of echo#

While echo works in most cases, its behavior is inconsistent across shells (e.g., BSD echo vs. GNU echo). For example, some echo implementations interpret backslashes (e.g., \n) as escape characters by default.

printf is a more portable and predictable alternative. Use printf "%s" "$var" to preserve newlines:

Example:

$ ls_output=$(ls -l)
$ printf "%s" "$ls_output"  # %s ensures the variable is printed as-is
total 0
-rw-r--r-- 1 user user 0 Jan 1 12:00 file1.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file2.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file3.txt

Why it works: printf "%s" takes the variable as a single argument and prints it verbatim, including newlines. Unlike echo, printf behavior is standardized by POSIX, making it safer for cross-shell scripts.

3. Here-Strings or Here-Documents#

If you need to pass the variable’s content to another command (not just print it), use here-strings (<<<) or here-documents to preserve newlines.

Here-Strings (<<<)#

A here-string feeds the variable’s content into a command’s standard input:

Example:

$ ls_output=$(ls -l)
$ cat <<< "$ls_output"  # Pass variable to cat via here-string
total 0
-rw-r--r-- 1 user user 0 Jan 1 12:00 file1.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file2.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file3.txt

Note: Here-strings are a bashism (not POSIX-compliant). For POSIX sh, use a here-document instead.

Here-Documents#

A here-document (<<EOF ... EOF) is a POSIX-compliant way to pass multi-line input:

Example:

$ ls_output=$(ls -l)
$ cat <<EOF
$ls_output
EOF
# Output: Same as above, with newlines preserved

Why it works: Here-strings and here-documents treat the variable expansion as a literal string, avoiding word splitting.

4. Modifying IFS (Internal Field Separator)#

You can temporarily modify IFS to disable word splitting for variable expansions. By setting IFS to an empty string, the shell stops splitting on spaces, tabs, or newlines.

Example:

$ ls_output=$(ls -l)
$ IFS=""  # Temporarily disable word splitting
$ echo $ls_output  # Now works without quotes (but quotes are still safer!)
total 0
-rw-r--r-- 1 user user 0 Jan 1 12:00 file1.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file2.txt
-rw-r--r-- 1 user user 0 Jan 1 12:00 file3.txt
$ unset IFS  # Reset IFS to default

Caveat: Modifying IFS affects all variable expansions in the current shell session. Use this sparingly, and always reset IFS afterward to avoid unintended side effects.

5. Handling Trailing Newlines in Command Substitution#

Command substitution ($(command) or `command`) strips trailing newlines from the command’s output. For example, if a command outputs line1\nline2\n\n (two trailing newlines), command substitution will store it as line1\nline2\n (one trailing newline).

To preserve all newlines (including trailing ones), use a workaround with printf to add a dummy character, then remove it:

Example:

# Command with trailing newlines: outputs "line1\nline2\n\n"
$ output=$(printf "line1\nline2\n\n"; printf "x")  # Add "x" to prevent stripping
$ output="${output%x}"  # Remove the "x"
$ echo "$output"
line1
line2
 
# Newlines are preserved!

Why it works: The extra printf "x" adds a non-newline character, preventing command substitution from stripping trailing newlines. We then remove the "x" with parameter expansion ${output%x}.

Common Pitfalls to Avoid#

  • Forgetting Quotes: Always use double quotes around variable expansions ("$var") unless you explicitly want word splitting.
  • Single Quotes: Single quotes (echo '$var') prevent variable expansion entirely—use double quotes instead.
  • Assuming echo is Consistent: Use printf for portability across shells (e.g., BSD/macOS vs. Linux).
  • Trailing Newline Stripping: Remember that command substitution removes trailing newlines. Use the printf "x" workaround if you need to preserve them.

Conclusion#

Preserving newlines in shell variables boils down to understanding word splitting and IFS. The most reliable solutions are:

  1. Double-quoting variable expansions (echo "$var").
  2. Using printf (printf "%s" "$var").
  3. Using here-strings/here-documents for passing multi-line data to commands.

By applying these techniques, you’ll ensure your shell scripts handle multi-line output correctly, avoiding messy formatting and data corruption.

References#