– General Linux 1 – Use Streams, Pipes, and Redirects [3] (Linux Professional Institute Certification) a .˜. /V\ // \\ @._.@ Prepared by Andrew Eager geoffrey robertson geoffrey@zip.com.au $Id: gl1.103.4.slides.tex,v 1.2 2003/05/29 14:10:18 geoffr Exp $ c 2002,2003 Geoffrey Robertson. Permission is granted to make and distribute verbatim copies or modified versions of this document provided that this copyright notice and this permission notice are preserved on all copies under the terms of the GNU General Public License as published by the Free Software Foundation—either version 2 of the License or (at your option) any later version. a Copyright 1 (1.3) GNU and UNIX Commands [30] 1.103.1 Work on the command line [4] 1.103.2 Process text streams using filters [7] 1.103.3 Perform basic file management [2] 1.103.4 Use streams, pipes, and redirects [3] 1.103.5 Create, monitor, and kill processes [7] 1.103.6 Modify process execution priorities [2] 1.103.7 Search text files using regular expressions [3] 1.103.8 Perform basic file editing operations using vi [2] 2 Objective Candidate should be able to redirect streams and connect them in order to efficiently process textual data. Tasks include redirecting standard input, standard output, and standard error, piping the output of one command to the input of another command, using the output of one command as arguments to another command and sending output to both stdout and a file. 3 Key files, terms, and utilities tee xargs < << > >> | ‘‘ 4 Resources of interest 5 STDIN, STDOUT & STDERR • When a process is run it needs 3 things: – An input device (ie a keyboard) – An output device (ie a screen) – An error device - somewhere to send critical errors (normally the screen) • Every process has 3 file descriptors – fd 0 is for input – fd 1 is for normal output – fd 2 is for error/abnormal output • By default these devices all default to your current tty 6 Default File Descriptor Assignments process 1 2 stdout 0 stdin stderr • fd 0 == stdin (keyboard) • fd 1 == stdout (screen) • fd 2 == stderr (screen) 7 Redirection & Duplication Operators • There are 3 operators used for redirection: – File redirects: (<, >and >>operators) – Pipelines (|operator) – File descriptor duplication (>& operator) 8 File Redirect Operators Each of the 3 file descriptors can be redirected to/from files as follows: Operator Action From file to fd 0 From fd 1 to file From fd 2 to file 1 2 0 < > 2> (Replace > with >> to append to a file) Note that the redirect operators work with the file descriptors (0, 1 or 2) and not with the physical device itself (/dev/tty). 9 Pipeline Redirect Operator Consider the command cmd1 | cmd2. The pipe operator takes data sent to stdout by cmd1 and sends it to the input file descriptor (fd 0) of cmd2: cmd2 0 stdout 1 2 Note that the pipe connects stdout (which may or may not be associated with fd 1) to fd 0 of the next command. 10 Example of the Pipeline Operator cmd2 cmd1 1 2 1 2 stdout 0 0 stdin stderr stderr cmd1 | cmd 2 Piping output of cmd1 into input of cmd2 11 File Descriptor Duplication Operator A file descriptor can be made to be a copy of another descriptor. Consider the command cmd 2>&1. This will make fd 2 become a copy of fd 1. cmd 0 1 2 fd 2 becomes a copy of fd 1 cmd 2>&1 Duplicating a file descriptor 12 Examples of File Output Redirection For the following examples, we use the two example files good and nofile created thus: $ echo "This is good" > good ← $ rm nofile ← And to test what output is going where, we use the following command line: $ cat good nofile ← which will produce: This is good cat: nofile: No such file or directory (stdout) (stderr) 13 Standard File Output Redirection cat 0 out.txt 1 2 "cat: nofile: No such file or directory" "This is good" cat good nofile > out.txt "This is good" cat 0 1 2 err.txt "cat: nofile: No such file or directory" cat good nofile 2> err.txt Redirecting stdout & stderr to different files 14 Output Redirection - Two at once cat 0 out.txt 1 2 err.txt "cat: nofile: No such file or directory" "This is good" cat good nofile > out.txt 2> err.txt Redirecting stdout & stderr at the same time 15 Redirecting stdout & stderr cat 0 out−err.txt 1 2 cat good nofile > out−err.txt cat 0 out−err.txt 1 2 out−err.txt cat good nofile > out−err.txt 2>&1 fd 2 becomes a copy of fd 1 Redirecting stdout & stderr to the same file 16 Duplicating before & after redirection cat 0 out−err.txt 1 2 fd 2 becomes a copy of fd 1 cat good nofile 2>&1 > out−err.txt cat 0 out−err.txt 1 2 out−err.txt fd 2 becomes a copy of fd 1 cat good nofile > out−err.txt 2>&1 Just when you duplicate the fd is significant 17 Piping stdout to stdin cat 0 0 1 2 sed 1 2 out.txt "This is good" "cat: nofile: No such file or directory" cat good nofile | sed −n p > out.txt Normal pipe from cat to sed 18 Piping stdout & stderr to stdin 1 cat 0 1 2 2 cat 0 1 2 cat good nofile cat good nofile 2>&1 3 0 cat 0 1 2 sed 1 2 out−err.txt cat good nofile 2>&1 | sed −n −p > out−err.txt 19 Piping stderr to stdin 1 cat 0 1 2 2 cat 0 1 2 cat good nofile cat good nofile 2>&1 3 cat 0 1 2 /dev/null err.txt 0 sed 1 2 cat good nofile 2>&1 >/dev/null | sed −n p > err.txt 20 File Redirection Summary • Redirect stdin from file: – $ cat < input.txt ← • Redirect stdout to file: – $ cat good nofile > out.txt ← • Redirect stderr to file: – $ cat good nofile 2> err.txt ← • Redirect stdout & stderr to file: – $ cat good nofile > out-err.txt 2>&1 ← OR – $ cat good nofile 2> out-err.txt 1>&2 ← 21 File Redirection Summary To append to a file instead of overwriting, simply replace >with >> • Redirect stdout to file (append): – $ cat good nofile >> out.txt ← • Redirect stderr to file (append): – $ cat good nofile 2>> err.txt ← • Redirect stdout & stderr to file (append): – $ cat good nofile >> out-err.txt 2>&1 ← OR – $ cat good nofile 2>> out-err.txt 1>&2 ← 22 Pipe Redirection Summary • Pipe stdout to stdin: – $ cat good nofile | sed -n p ← • Pipe stdout & stderr to stdin – $ cat good nofile 2>&1 | sed -n p ← • Pipe stderr to stdin – $ cat good nofile 2>&1 >/dev/null | sed -n p ← 23 A cool example - Swap stdout & stderr In this example, we are going to swap stdout & stderr by using a temporary fd as a holding buffer: If we execute the command using a normal pipe: $ cat good nofile | sed -n -p > stdout.txt ← cat: nofile: No such file or directory Now if we swap stdout & stderr: $ cat good nofile 255>&1 1>&2 2>&255 | sed -n -p > stdout.txt ← This is good 24 The End 25