Shell scripts are powerful but debugging them can be challenging without proper logging. In this post, I’ll share some techniques for logging shell scripts.

I often start by adding a set -x to the script so that I can see what’s being executed, including variable expansion and substitution.

I usually also want to log the output to a file. When running a script interactively, this is easy using tee, which reads from standard input and writes to both standard output and one or more files simultaneously:

$ ./some/script.sh | tee /some/file.log

If I want both stdout and stderr in the file, I can redirect stderr to stdout first:

$ ./some/script.sh 2>&1 | tee /some/file.log

But most scripts I care about aren’t run interactively—they’re triggered by other scripts or events. So you need to add logging to the script itself. I used to add these lines to the top of my scripts:

#!/usr/bin/env bash

set -x
exec &> /tmp/some/file

# {the actual script}

The &> redirects both stdout and stderr to a file, and exec applies the change to the current shell process and its children (subprocesses).

The downside of this approach is that when you run the script interactively, you don’t see any output in the terminal—it’s only available in the log file. To get both, we’d want to use tee, but the challenge is that tee reads from standard input, not a file. This is exactly where shell process substitution becomes useful:

#!/usr/bin/env bash

set -x
exec &> >(tee /some/file.log)

# {the actual script}

What is process substitution?

Process substitution allows you to access the stdout or stdin of a command as a file. The syntax is <(command) or >(command):

  1. <(command)

This is the more commonly used construct. It allows you to access a command’s output as a file that you can read from. This is what normal | pipes do as well but sometimes having the output as a file is more useful or required. For example, some programs only accept file arguments (not stdin). Another example is when you need multiple outputs. diff demonstrates both these cases:

diff <(ls /path1) <(ls /path2)

In general, if you find yourself writing to a temporary file, chances are you can accomplish the same task using process substitution.

  1. >(command)

This gives you access stdin of a command as a file descriptor you can write to. This is particularly powerful when combined with tee. For example, you can send data to multiple commands simultaneously:

./script.sh | tee >(grep foo > matches.txt) >(wc -l > count.txt) > all.txt

Log to system log instead of a file

Finally, consider using your system’s logging capability via the logger command instead of writing to a file. This leverages your system’s log management infrastructure, which handles rotation, compression, and retention policies:

#!/usr/bin/env bash

set -x
exec &> >(logger --tag some_tag)

# {the actual script}

The logs can later be retrieved using journalctl -t some_tag on systems with systemd.