Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision
Previous revision
howto:redirection_tutorial [2010/10/16 13:26]
127.0.0.1 external edit
howto:redirection_tutorial [2017/02/04 20:55] (current)
twc [stdin, stdout, stderr] - change lsof pseudo terminal number to match that referenced in the rest of the page
Line 6: Line 6:
  
 This tutorial is not a complete guide to redirection,​ it will not This tutorial is not a complete guide to redirection,​ it will not
-cover here docs, here strings, name pipes etc... I just hope it'​ll ​make +cover here docs, here strings, name pipes etc... I just hope it'​ll ​help 
-you understand ​better ​what things like ''​3>&​2'',​ ''​2>&​1''​ or ''​1>&​3-''​ do.+you to understand what things like ''​3>&​2'',​ ''​2>&​1''​ or ''​1>&​3-''​ do.
  
  
Line 16: Line 16:
 standard error (''​stderr''​). standard error (''​stderr''​).
  
-For example, ​in Bashin a terminal emulator, ​on Linux you'll see:+For example, ​with Bash running ​in a Linux terminal emulator, you'll see:
  
 <​code>​ <​code>​
-lsof -a -p $$ -d0,1,2 +lsof +f g -ap $BASHPID ​-d 0,1,2 
-COMMAND ​  PID USER   ​FD ​  TYPE DEVICE SIZE NODE NAME +COMMAND ​  PID USER   ​FD ​  ​TYPE ​FILE-FLAG ​DEVICE SIZE/OFF NODE NAME 
-bash    ​24507 root    0u   ​CHR ​ 136,5         ​7 ​/​dev/​pts/​5 +bash    ​12135 root    0u   ​CHR ​    RW,LG 136,13      0t0   ​16 ​/​dev/​pts/​5 
-bash    ​24507 root    1u   ​CHR ​ 136,5         ​7 ​/​dev/​pts/​5 +bash    ​12135 root    1u   ​CHR ​    RW,LG 136,13      0t0   ​16 ​/​dev/​pts/​5 
-bash    ​24507 root    2u   ​CHR ​ 136,5         ​7 ​/dev/pts/5+bash    ​12135 root    2u   ​CHR ​    RW,LG 136,13      0t0   ​16 ​/dev/pts/5
 </​code>​ </​code>​
  
-This ''/​dev/​pts/​5'' ​pts is a pseudo terminal used to simulate ​a real +This ''/​dev/​pts/​5''​ is a pseudo terminal used to emulate ​a real 
-terminal ​connected to the computer. Bash reads (''​stdin''​) from this +terminal. Bash reads (''​stdin''​) from this terminal and prints 
-terminal and prints via ''​stdout''​ and ''​stderr''​ to this terminal.+via ''​stdout''​ and ''​stderr''​ to this terminal.
  
 <​code>​ <​code>​
Line 44: Line 44:
 </​code>​ </​code>​
  
-When a command, a compound command, a subshell etc... is executed, it inherits+When a command, a compound command, a subshell etc. is executed, it inherits
 these file descriptors. For instance ''​echo foo''​ will send the text ''​foo''​ to the file these file descriptors. For instance ''​echo foo''​ will send the text ''​foo''​ to the file
 descriptor ''​1''​ inherited from the shell, which is connected to ''/​dev/​pts/​5''​. descriptor ''​1''​ inherited from the shell, which is connected to ''/​dev/​pts/​5''​.
Line 55: Line 55:
 ''​echo foo > file''​ ''​echo foo > file''​
  
-the ''>​ file''​ after the command ​alter the files descriptors ​of the+the ''>​ file''​ after the command ​alters ​the file descriptors ​belonging to the
 command foo. It changes the file descriptor ''​1''​ (''>​ file''​ is the same as command foo. It changes the file descriptor ''​1''​ (''>​ file''​ is the same as
 ''​1>​file''​) so that it points to the file ''​file''​. They will look like: ''​1>​file''​) so that it points to the file ''​file''​. They will look like:
Line 77: Line 77:
 named ''​file''​. named ''​file''​.
  
-In the same way command ''​2>​ file''​ will change the standard error and +In the same waycommand ''​2>​ file''​ will change the standard error and 
-will make it point to ''​file''​. Standard error is often use by +will make it point to ''​file''​. Standard error is used by applications to print errors.
-applications to print... errors.+
  
 What will ''​command 3> file''​ do? It will open a new file descriptor What will ''​command 3> file''​ do? It will open a new file descriptor
-pointing to ''​file''​. The command will then starts ​with:+pointing to ''​file''​. The command will then start with:
  
 <​code>​ <​code>​
Line 102: Line 101:
 </​code>​ </​code>​
  
-What will the command do with this descriptor? ​ It depends, often +What will the command do with this descriptor? ​ It depends. Often nothing. 
-nothing. We will see later why we might want other file descriptors.+We will see later why we might want other file descriptors.
  
 ===== Input Redirection "n< file" ===== ===== Input Redirection "n< file" =====
  
-I hope that the following will be obvious, when you run a command +When you run a commandusing ​''​command < file'',​ it changes the 
-using ''​command < file'',​ it changes the file descriptor ''​0''​ so that +file descriptor ''​0''​ so that it looks like:
-it looks like:+
  
 <​code>​ <​code>​
                   ---       ​+-----------------------+                   ---       ​+-----------------------+
-standard input   ( 0 ) ---->| file                  |+standard input   ( 0 ) <----| file                  |
                   ---       ​+-----------------------+                   ---       ​+-----------------------+
  
Line 126: Line 124:
  
 If the command reads from ''​stdin'',​ it now will read from ''​file''​ and not If the command reads from ''​stdin'',​ it now will read from ''​file''​ and not
-from the terminal.+from the console.
  
 As with ''>'',​ ''<''​ can be used to open a new file descriptor for reading, As with ''>'',​ ''<''​ can be used to open a new file descriptor for reading,
-''​command 3<​file''​. ​We will see how this can be useful.+''​command 3<​file''​. ​Later we will see how this can be useful.
  
  
Line 135: Line 133:
  
 What does this ''​|''​ do? Among other things, it connects the standard output of What does this ''​|''​ do? Among other things, it connects the standard output of
-the left command to the standard input of the right command. +the command ​on the left to the standard input of the command ​on the right
-That is, it creates a special file, a pipe which is opened ​for writing ​ +That is, it creates a special file, a pipewhich is opened ​as a write destinaton 
-for the left command, and opened for reading ​for the command ​on the right.+for the left command, and as a read source ​for the right command.
  
 <​code>​ <​code>​
Line 156: Line 154:
 </​code>​ </​code>​
  
-This is possible because the redirections are set upby the shell +This is possible because the redirections are set up by the shell 
-before the commands are executed, and the commands inherit the file+**before** the commands are executed, and the commands inherit the file
 descriptors. descriptors.
  
Line 205: Line 203:
 </​code>​ </​code>​
  
-Note that the positions are also duplicated. If you have allready+Note that the positions are also duplicated. If you have already
 read a line of ''​n'',​ then after ''​n>&​m''​ if you read a line from ''​m'',​ you will read a line of ''​n'',​ then after ''​n>&​m''​ if you read a line from ''​m'',​ you will
 get the second line of the file. get the second line of the file.
Line 213: Line 211:
  
 While it doesn'​t matter where the redirections appears on the command While it doesn'​t matter where the redirections appears on the command
-line, their order does matter. They are setup from left to right.+line, their order does matter. They are set up from left to right.
  
   * ''​2>&​1 >​file''​   * ''​2>&​1 >​file''​
Line 219: Line 217:
 A common error, is to do ''​command 2>&1 > file''​ to redirect both ''​stderr''​ A common error, is to do ''​command 2>&1 > file''​ to redirect both ''​stderr''​
 and ''​stdout''​ to ''​file''​. Let's see what's going on. First we type the and ''​stdout''​ to ''​file''​. Let's see what's going on. First we type the
-command in our typical ​terminal, the descriptors look like this:+command in our terminal, the descriptors look like this:
  
 <​code>​ <​code>​
Line 311: Line 309:
  
 This is a common error, we want to modify a file using something that This is a common error, we want to modify a file using something that
-reads from a file and write the result to ''​stdout''​. To do this we redirect stdout+reads from a file and writes ​the result to ''​stdout''​. To do thiswe redirect stdout
 to the file we want to modify. The problem here is that, as we to the file we want to modify. The problem here is that, as we
 have seen, the redirections are setup before the command is actually have seen, the redirections are setup before the command is actually
 executed. executed.
  
-So **BEFORE** sed starts standard output has already been redirected, with +So **BEFORE** sed startsstandard output has already been redirected, with 
-the additional side effect that, because we use >, "​file" ​is truncated. When ''​sed''​ starts to+the additional side effect that, because we used >, "​file" ​gets truncated. When ''​sed''​ starts to
 read the file, it contains nothing. read the file, it contains nothing.
  
Line 349: Line 347:
 ran ''​myscript 2>​file''​. ran ''​myscript 2>​file''​.
  
-''​exec''​ can be used, for instance, ​if you want to log the errors ​that the+''​exec''​ can be used, if, for instance, you want to log the errors the
 commands in your script produce, just add ''​exec 2>​myscript.errors''​ at commands in your script produce, just add ''​exec 2>​myscript.errors''​ at
 the beginning of your script. the beginning of your script.
Line 393: Line 391:
 <​code>​ <​code>​
  exec 3<file  exec 3<file
- while read -u 3 line;​do ​echo echo "​$line";​ read -p "Press any key" -n 1;done+ while read -u 3 line;do echo "​$line";​ read -p "Press any key" -n 1;done
 </​code>​ </​code>​
  
Line 434: Line 432:
 good idea to close the file descriptors you open. For instance, if you good idea to close the file descriptors you open. For instance, if you
 open a file descriptor with ''​exec 3>​file'',​ all the commands afterwards open a file descriptor with ''​exec 3>​file'',​ all the commands afterwards
-will inherit it. It's probably ​nicer to do something like:+will inherit it. It's probably ​better ​to do something like:
  
 <​code>​ <​code>​
Line 447: Line 445:
  
 I've seen some people using this as a way to discard, say stderr, using something like: command 2>&​-. I've seen some people using this as a way to discard, say stderr, using something like: command 2>&​-.
-Though it might work, I'm not sure if you can expect all the applications to behave correctly with a closed stderr.+Though it might work, I'm not sure if you can expect all applications to behave correctly with a closed stderr.
  
-In doubt, I prefer to use 2>/​dev/​null ​for this.+When in doubt, I use 2>/​dev/​null.
    
  
Line 487: Line 485:
  
 We only made 2 copies of ''​stderr''​ and ''​stdout''​. ''​3>&​1 4>&​1''​ would have We only made 2 copies of ''​stderr''​ and ''​stdout''​. ''​3>&​1 4>&​1''​ would have
-produce ​the same result here because we run the command in a terminal +produced ​the same result here because we ran the command in a terminal 
-and thus ''​1''​ and ''​2'' ​goes to the terminal. As an exercise, you can start+and thus ''​1''​ and ''​2'' ​go to the terminal. As an exercise, you can start
 with ''​1''​ pointing to ''​file.stdout''​ and 2 pointing to ''​file.stderr'',​ you will with ''​1''​ pointing to ''​file.stdout''​ and 2 pointing to ''​file.stderr'',​ you will
 see why these redirections are very nice. see why these redirections are very nice.
Line 508: Line 506:
 </​code>​ </​code>​
  
-It inherits the previous file descriptors, ​close 3 and 4 and setup a pipe +It inherits the previous file descriptors, ​closes ​3 and 4 and sets up a pipe 
-for read. Now for the left part of the second pipe ''​{...} ​ 2>&1 >&4 4>&- |''​+for reading. Now for the left part of the second pipe ''​{...} ​ 2>&1 >&4 4>&- |''​
  
 <​code>​ <​code>​
Line 526: Line 524:
  
 First, The file descriptor ''​1''​ is connected to the pipe (''​|''​),​ then ''​2''​ is First, The file descriptor ''​1''​ is connected to the pipe (''​|''​),​ then ''​2''​ is
-made a copy of ''​1''​ and thus is made fd to the pipe (''​2>&​1''​),​ then ''​1''​ is+made a copy of ''​1''​ and thus is made an fd to the pipe (''​2>&​1''​),​ then ''​1''​ is
 made a copy of ''​4''​ (''>&​4''​),​ then ''​4''​ is closed. These are the file made a copy of ''​4''​ (''>&​4''​),​ then ''​4''​ is closed. These are the file
-descriptors of the inner ''​{}''​, let's go inside and have a look at the+descriptors of the inner ''​{}''​. Lcet's go inside and have a look at the
 right part of the first pipe: ''​| cmd2 2>&3 3>&​-''​ right part of the first pipe: ''​| cmd2 2>&3 3>&​-''​
  
Line 608: Line 606:
  
 I used to have trouble choosing between ''​0&<​3''​ ''​3&>​1''​ ''​3>&​1''​ ''<​nowiki>​->​2</​nowiki>''​ ''​-<&​0''​ ''&​-<​0''​ I used to have trouble choosing between ''​0&<​3''​ ''​3&>​1''​ ''​3>&​1''​ ''<​nowiki>​->​2</​nowiki>''​ ''​-<&​0''​ ''&​-<​0''​
-''​0<&​-''​ etc... (I think probably because the syntax ​represents ​more the result, i.e., the redirection,​ than+''​0<&​-''​ etc... (I think probably because the syntax ​is more representative of the result, i.e., the redirection,​ than
 what is done, i.e., opening, closing, or duplicating file descriptors). what is done, i.e., opening, closing, or duplicating file descriptors).
  
Line 619: Line 617:
  
   * ''​lhs''​ is always a file description,​ i.e., a number:   * ''​lhs''​ is always a file description,​ i.e., a number:
-    * Either ​one we want to open, duplicate, move or one we want to close. If the op is ''<''​ then there is an implicit 0, if it's ''>''​ or ''<​nowiki>>></​nowiki>'',​ there is an implicit 1.+    * Either we want to open, duplicate, move or we want to close. If the op is ''<''​ then there is an implicit 0, if it's ''>''​ or ''<​nowiki>>></​nowiki>'',​ there is an implicit 1.
  
  
-  * ''​op''​ is either ​''<'',​ ''>'',​ ''<​nowiki>>></​nowiki>'',​ ''>​|'',​ or ''<>'':​ +  * ''​op''​ is ''<'',​ ''>'',​ ''<​nowiki>>></​nowiki>'',​ ''>​|'',​ or ''<>'':​ 
-    * ''<''​ if the file decriptor in ''​lhs''​ will be read, ''>''​ if it will be written, ''<​nowiki>>></​nowiki>''​ if data will be appended to the file, ''>​|''​ to overwrite an existing file  or ''<>''​ if it will be both read and written.+    * ''<''​ if the file decriptor in ''​lhs''​ will be read, ''>''​ if it will be written, ''<​nowiki>>></​nowiki>''​ if data is to be appended to the file, ''>​|''​ to overwrite an existing file  or ''<>''​ if it will be both read and written.
  
  
   * ''​rhs''​ is the thing that the file descriptor will describe:   * ''​rhs''​ is the thing that the file descriptor will describe:
-    * It can be either ​the name of a file, the place where another descriptor goes (''&​1''​),​ or the special nowhere, ''&​-'',​ which will close the file descriptor.+    * It can be the name of a file, the place where another descriptor goes (''&​1''​),​ or, ''&​-'',​ which will close the file descriptor.
    
  
Line 633: Line 631:
 inexact, but I think it really helps to easily find that, say ''<​nowiki>&​->​0</​nowiki>''​ is inexact, but I think it really helps to easily find that, say ''<​nowiki>&​->​0</​nowiki>''​ is
 incorrect. incorrect.
 +
 +==== A note on style ====
 +
 +The shell is pretty loose about what it considers a valid redirect. While opinions probably differ, this author has some (strong) recommendations:​
 +
 +  * **Always** keep redirections "​tightly grouped"​ -- that is, **do not** include whitespace anywhere within the redirection syntax except within quotes if required on the RHS (e.g. a filename that contains a space). Since shells fundamentally use whitespace to delimit fields in general, it is visually much clearer for each redirection to be separated by whitespace, but grouped in chunks that contain no unnecessary whitespace.
 +
 +  * **Do** always put a space between each redirection,​ and between the argument list and the first redirect.
 +
 +  * **Always** place redirections together at the very end of a command after all arguments. Never precede a command with a redirect. Never put a redirect in the middle of the arguments.
 +
 +  * **Never** use the Csh ''&>​foo''​ and ''>&​foo''​ shorthand redirects. Use the long form ''>​foo 2>&​1''​. (see: [[obsolete]])
 +
 +<​code>​
 +# Good! This is clearly a simple commmand with two arguments and 4 redirections
 +cmd arg1 arg2 <myFile 3<&1 2>/​dev/​null >&2
 +
 +# Good!
 +{ cmd1 <<<'​my input';​ cmd2; } >​someFile
 +
 +# Bad. Is the "​1"​ a file descriptor or an argument to cmd? (answer: it's the FD). Is the space after the herestring part of the input data? (answer: No).
 +# The redirects are also not delimited in any obvious way.
 +cmd 2>& 1 <<<​ stuff
 +
 +# Hideously Bad. It's difficult to tell where the redirects are and whether they'​re even valid redirects.
 +# This is in fact one command with one argument, an assignment, and three redirects.
 +foo=bar<​baz bork<<<​ blarg>​bleh
 +</​code>​
  
 ====== Conclusion ====== ====== Conclusion ======
  • howto/redirection_tutorial.1287235571.txt
  • Last modified: 2011/11/11 17:47
  • (external edit)