Redirecting Standard Output and Logging with Unix Tools

When working with command-line programs on Unix/Linux systems, it is often necessary to log the standard output (stdout) and standard error (stderr) for debugging and monitoring purposes. This article will walk you through using the tee command to duplicate stdout to a log file, and explain how to log stderr as well. We will also discuss a technique to prepend each log line with a timestamp for better readability.

Introduction to the tee Command

tee is a versatile utility that duplicates the input to the standard output and simultaneously writes a copy to files. This makes it an ideal tool for logging command outputs while also displaying them on the terminal. Here is an example of using tee to log the output of the echo command:

echo Hello | tee hello.log

Running the above command will display Hello on the terminal and simultaneously write it to hello.log file:

To view the contents of the log file, you can use the cat command:

cat hello.log

This will display:

Redirecting stderr to stdout and Logging

A common scenario is to redirect stderr to stdout to capture error outputs as well. This can be achieved using file redirection. Here is an example:

program 21  log

This command will run the program and write both stdout and stderr to the log file.

Adding Timestamps to Log Entries

For better readability and context, you might want to prepend each log entry with the timestamp. This can be accomplished using awk to format the timestamp and tee to log the output. Below are the steps to achieve this:

Step 1: Create a Timestamp Formatting Script

Create a small AWK script to format the timestamp. Save the following AWK code in a file named ts in your search path:

awk '{ print strftime() " " $0 }'

Make the file executable:

chmod  x ts

Step 2: Use the Script to Log with Timestamps

Now, you can run your program and pipe its output through the ts script and tee to create a properly timestamped log file. Here is an example:

program 21 | ts | tee log

This command will execute program with its stderr and stdout piped through the ts script, which adds the current timestamp to each line of the output, and then writes the entire stream to the log file.

Conclusion

Using the tee command for logging is a straightforward and effective method in Unix/Linux environments. By combining it with other tools such as awk and redirecting streams, you can create powerful logging mechanisms that provide context and detail in your logs.