syntax:expansion:proc_subst

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
syntax:expansion:proc_subst [2012/06/23 21:55]
ormaaj "scope"
syntax:expansion:proc_subst [2018/03/02 09:54] (current)
centos-hater [Avoiding subshells] Grammar fix
Line 2: Line 2:
  
 {{keywords>​bash shell scripting expansion substitution text stdin stdout save capture}} {{keywords>​bash shell scripting expansion substitution text stdin stdout save capture}}
 +
 +Process substitution is a form of redirection where the input or output of a process (some sequence of commands) appear as a temporary file.
  
 <​code>​ <​code>​
Line 8: Line 10:
 >( <​LIST>​ ) >( <​LIST>​ )
 </​code>​ </​code>​
- 
-This only works on systems that support named pipes (FIFO - a [[dict:​terms:​special_file | special file]]) or the ''/​dev/​fd/​NNNNN''​-syntax to access open files. 
  
 Process substitution is performed **simultaneously** with [[syntax:pe | parameter expansion]],​ [[syntax:​expansion:​cmdsubst | command substitution]] and [[syntax:​expansion:​arith | arithmetic expansion]]. Process substitution is performed **simultaneously** with [[syntax:pe | parameter expansion]],​ [[syntax:​expansion:​cmdsubst | command substitution]] and [[syntax:​expansion:​arith | arithmetic expansion]].
Line 20: Line 20:
 That, for example, allows to give data to a command that can't be reached by pipelining (that doesn'​t expect its data from ''​stdin''​ but from a file). That, for example, allows to give data to a command that can't be reached by pipelining (that doesn'​t expect its data from ''​stdin''​ but from a file).
  
-If a process substitution is expanded as an argument to a function, or assigned to an environment variable when calling the function, the process substitution will be "held open" until the function returns.+==== Scope ====
  
-===== Example ​=====+If a process substitution is expanded as an argument to a function, expanded to an environment variable during calling of a function, or expanded to any assignment within a function, the process substitution will be "held open" for use by any command within the function or its callees, until the function in which it was set returns. If the same variable is set again within a callee, unless the new variable is local, the previous process substitution is closed and will be unavailable to the caller when the callee returns. 
 + 
 +In essence, process substitutions expanded to variables within functions remain open until the function in which the process substitution occured returns - even when assigned to locals that were set by a function'​s caller. Dynamic scope doesn'​t protect them from closing. 
 + 
 +===== Examples ​=====
  
 This code is useless, but it demonstrates how it works: This code is useless, but it demonstrates how it works:
-<​code>​+<​code ​bash>
 $ echo <(ls) $ echo <(ls)
 /dev/fd/63 /dev/fd/63
 </​code>​ </​code>​
  
-The **output** of the ''​ls''​-program can be accessed by reading the file ''/​dev/​fd/​63''​.+The **output** of the ''​ls''​-program can then be accessed by reading the file ''/​dev/​fd/​63''​.
  
-This is very useful ​in redirection statements+Consider the following:​ 
-**wrong** piece of code to count all files in ''/​etc''​ is: +<code bash> 
-<​code>​+diff <(ls "​$first_directory"​) <(ls "​$second_directory"​) 
 +</​code>​ 
 +This will compare the contents of each directory. In this command, each //​process// ​is //​substituted//​ for a //file//, and diff doesn'​t see <(bla), it sees two files, so the effective command is something like 
 +<code bash> 
 +diff /dev/fd/63 /​dev/​fd/​64 
 +</​code>​ 
 +where those files are written to and destroyed automatically. 
 +==== Avoiding subshells ==== 
 +<WRAP center round info 60%> 
 +See Also: [[http://​mywiki.wooledge.org/​BashFAQ/​024 | BashFAQ/​024]] -- //I set variables in a loop that'​s ​in a pipelineWhy do they disappear after the loop terminates? Or, why can't I pipe data to read?// 
 +</​WRAP>​ 
 + 
 +One of the most common uses for process substitutions is to avoid the final subshell that results from executing a pipeline. The following is a **wrong** piece of code to count all files in ''/​etc''​ is: 
 + 
 +<​code ​bash>
 counter=0 counter=0
  
-find /etc | while read; do +find /etc -print0 ​| while IFS= read -rd ''​ _; do 
-  ((counter++))+    ((counter++))
 done done
  
-echo "​$counter files"+echo "​$counter ​files" # prints "​0 ​files"
 </​code>​ </​code>​
-Why this is wrong? Due to the pipe, the ''​while read; do ... done''​ part is executed in a subshell. That also means ''​counter''​ is incremented in that subshell. When the pipeline finishes, the subshell is terminated, and the ''​counter''​ you access is the ''​counter''​ from the main shell - the one at "​0"​! 
  
-Process substitution helps us here to avoid the pipe (which is the reason for the subshell):​ +Due to the pipe, the ''​while read; do ... done''​ part is executed in a subshell (in Bash, by default), which means ''​counter''​ is only incremented within the subshell. When the pipeline finishes, the subshell is terminated, and the ''​counter''​ visible to ''​echo''​ is still at "​0"​! 
-<​code>​+ 
 +Process substitution helps us avoid the pipe operator ​(the reason for the subshell): 
 + 
 +<​code ​bash>
 counter=0 counter=0
  
-while read; do +while IFS= read -rN1 _; do 
-  ((counter++)) +    ((counter++)) 
-done < <(find /etc)+done < <(find /etc -printf ' ')
  
 echo "​$counter files" echo "​$counter files"
 </​code>​ </​code>​
-This is the normal input file redirection ''<​ FILE'',​ just that ''​FILE''​ is dynamically generated by process substitution:​ + 
-<​code>​ +This is the normal input file redirection ''<​ FILE'',​ just that the ''​FILE'' ​in this case is the result of process substitution. It's important to note that the space is required in order to disambiguate the syntax from [[syntax:redirection#​here_documents | here documents]]. 
-< <​(COMMAND)+ 
 +<​code ​bash
 +< <​(COMMAND) ​# Good. 
 +: <<​(...) # Wrong. Will be parsed as a heredoc. Bash fails when it comes across the unquoted metacharacter ''​(''​ 
 +: ><​(...) # Technically valid but pointless syntax. Bash opens the pipe for writing, while the commands within the process substitution have their stdout connected to the pipe.
 </​code>​ </​code>​
 +
 +==== Process substitution assigned to a parameter ====
  
 This example demonstrates how process substitutions can be made to resemble "​passable"​ objects. This results in converting the output of ''​f'''​s argument to uppercase. This example demonstrates how process substitutions can be made to resemble "​passable"​ objects. This results in converting the output of ''​f'''​s argument to uppercase.
  
-<​code>​+<​code ​bash>
 f() { f() {
     cat "​$1"​ >"​$x"​     cat "​$1"​ >"​$x"​
-};+}
  
 x=>(tr '​[:​lower:​]'​ '​[:​upper:​]'​) f <(echo 'hi there'​) x=>(tr '​[:​lower:​]'​ '​[:​upper:​]'​) f <(echo 'hi there'​)
 </​code>​ </​code>​
 +
 +See the above section on [[#scope]]
 +
 +===== Bugs and Portability Considerations =====
 +
 +  * Process substitution is not specified by POSIX.
 +  * Process substitution is disabled completely in Bash POSIX mode.
 +  * Process substitution is implemented by Bash, Zsh, Ksh{88,93}, but not (yet) pdksh derivatives (mksh). Coprocesses may be used instead.
 +  * Process substitution is supported only on systems that support either named pipes (FIFO - a [[dict:​terms:​special_file | special file]]) or the ''/​dev/​fd/​*''​ method for accessing open files. If the system doesn'​t support ''/​dev/​fd/​*'',​ Bash falls back to creating named pipes. Note that not all shells that support process substitution have that fallback.
 +  * Bash evaluates process substitutions within array indices, but not other arithmetic contexts. Ksh and Zsh do not. (Possible Bug)
 +<code bash>
 +# print "​moo"​
 +dev=fd=1 _[1<​(echo moo >&​2)]=
 +# fork bomb
 +${dev[${dev='​dev[1>​(${dev[dev]})]'​}]}
 +</​code>​
 +  * Issues with wait, race conditions, etc: https://​groups.google.com/​forum/?​fromgroups=#​!topic/​comp.unix.shell/​GqLNzUA4ulA
  
 ===== See also ===== ===== See also =====
  • syntax/expansion/proc_subst.1340488524.txt
  • Last modified: 2012/06/23 21:55
  • by ormaaj