Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
syntax:keywords:coproc [2011/09/28 20:03]
thebonsai [See also]
syntax:keywords:coproc [2013/04/14 12:36] (current)
thebonsai Don't tread version 4 special - it has been around for a long time now
Line 1: Line 1:
 ====== The coproc keyword ====== ====== The coproc keyword ======
-:V4: 
  
 ===== Synopsis ===== ===== Synopsis =====
 +
 <​code>​ <​code>​
  ​coproc [NAME] command [redirections]  ​coproc [NAME] command [redirections]
 </​code>​ </​code>​
- 
  
 ===== Description ===== ===== Description =====
-Bash 4.0 introduced the coprocesses,​ a feature certainly familiar to ksh users. 
  
-''​coproc''​ starts a command ​in the backgound ​setting up pipes so that you can +Bash 4.0 introduced //​coprocesses//,​ a feature certainly familiar to ksh users. The ''​coproc'' ​keyword ​starts a command ​as a background job, setting up pipes connected to both its stdin and stdout ​so that you can interact with it bidirectionally. Optionally, the co-process can have a name ''​NAME''​. If ''​NAME''​ is given, the command that follows **must be a compound command**. If no ''​NAME''​ is given, then the command can be either simple or compound.
-interact with it. Optionally, the co-process can have a name ''​NAME''​.+
  
-If ''​NAME'' ​is given, the following command **must be compound command**. If no ''​NAME'' ​ist giventhe command can be a simple command or a compound ​command.+The process ID of the shell spawned to execute the coprocess is available through the value of the variable named by ''​NAME'' ​followed by a ''​_PID''​ suffix. For example, the variable name used to store the PID of coproc started with no ''​NAME''​ given would be ''​COPROC_PID''​ (because ''​COPROC''​ is the default ''​NAME''​). [[commands:​builtin:​wait]] may be used to wait for the coprocess to terminate. Additionally,​ coprocesses may be manipulated through their ''​jobspec''​. 
 + 
 +==== Return status ==== 
 + 
 +The return status of coprocess is the exit status of its command.
  
 ==== Redirections ==== ==== Redirections ====
  
-The redirections are normal redirections that are set after the pipe +The optional ​redirections are applied ​after the pipes have been set up. Some examples:
-has been set up, some examples:+
  
-<​code>​+<​code ​bash>
 # redirecting stderr in the pipe # redirecting stderr in the pipe
-$ coproc { ls thisfiledoesntexist;​ read ;} 2>&1+$ coproc { ls thisfiledoesntexist;​ read; } 2>&1
 [2] 23084 [2] 23084
-$ read -${COPROC[0]};​printf ​"%s\n" ​"$REPLY"+IFS= read -ru ${COPROC[0]} ​x; printf ​'%s\n' ​"$x"
 ls: cannot access thisfiledoesntexist:​ No such file or directory ls: cannot access thisfiledoesntexist:​ No such file or directory
 </​code>​ </​code>​
  
-<​code>​+<​code ​bash>
 #let the output of the coprocess go to stdout #let the output of the coprocess go to stdout
-$ { coproc mycoproc { awk '​{print "​foo"​ $0;​fflush()}'​ ;} >&3 ;} 3>&1+$ { coproc mycoproc { awk '​{print "​foo"​ $0;​fflush()}';​ } >&3; } 3>&1
 [2] 23092 [2] 23092
 $ echo bar >&​${mycoproc[1]} $ echo bar >&​${mycoproc[1]}
Line 37: Line 37:
 </​code>​ </​code>​
  
-Here we need to save the previous file descriptor of stdout, because by the time +Here we need to save the previous file descriptor of stdout, because by the time we redirect the fds of the coprocessstdout has already ​been redirected to the pipe. 
-we want to redirect the fds of the coprocess stdout has been redirected +
-to the pipe.  +
  
 ==== Pitfalls ==== ==== Pitfalls ====
  
-=== Avoid the command | while read subshell ===+=== Avoid the final pipeline ​subshell ===
  
-The traditional ​KSH workaround to avoid the subshell when doing +The traditional ​Ksh workaround to avoid the subshell when doing ''​command | while read'' ​is to use a coprocess. UnfortunatelyBash'​s ​behavior differs.
-command | while read is to use a coprocess, ​unfortunately,​ it seems  +
-that bash'​s ​behaviour differ from KSH.+
  
-In KSH you would do: +In Ksh you would do: 
-<​code>​ +<​code ​bash> 
-ls |& #start a coprocess +# ksh93 or mksh/pdksh derivatives 
-while read -file;​do ​echo "​$file";​done #read its output+ls |& # start a coprocess 
 +while IFS= read -rp file; do print -r -- "​$file";​ done # read its output
 </​code>​ </​code>​
  
 In bash: In bash:
-<​code>​+<​code ​bash>
 #​DOESN'​T WORK #​DOESN'​T WORK
 $ coproc ls $ coproc ls
 [1] 23232 [1] 23232
-$ while read -${COPROC[0]} line;​do ​echo "​$line";​done+$ while IFS= read -ru ${COPROC[0]} line; do printf '​%s\n' ​"​$line";​ done
 bash: read: line: invalid file descriptor specification bash: read: line: invalid file descriptor specification
 [1]+  Done                    coproc COPROC ls [1]+  Done                    coproc COPROC ls
 </​code>​ </​code>​
  
-By the time we start reading from the output of the coprocess, the +By the time we start reading from the output of the coprocess, the file descriptor has been closed
-file descriptor has been closed.+ 
 +See [[http://​mywiki.wooledge.org/​BashFAQ/​024 | this FAQ entry on Greg's wiki]] for other pipeline subshell workarounds.
  
 === Buffering === === Buffering ===
  
-In the first example, we used ''​fflush()''​ in the ''​awk''​ command, ​this was +In the first example, we GNU awk'​s ​''​fflush()''​ command. As always, when you use pipes the I/O operations are buffered. Let's see what happens with ''​sed'':​
-done on purpose, as always ​when you use pipes the I/O operations are buffered+
-let's see what happens with ''​sed'':​+
  
-<​code>​+<​code ​bash>
 $ coproc sed s/^/foo/ $ coproc sed s/^/foo/
 [1] 22981 [1] 22981
 $ echo bar >&​${COPROC[1]} $ echo bar >&​${COPROC[1]}
-$ read -t 3 -${COPROC[0]};​ (( $? >127 )) && echo "​nothing read"+$ read -t 3 -ru ${COPROC[0]} ​_; (( $? > 127 )) && echo "​nothing read"
 nothing read nothing read
 </​code>​ </​code>​
  
-Even though this example is the same as the first ''​awk''​ example, the +Even though this example is the same as the first ''​awk''​ example, the ''​read'' ​doesn'​t return because the output is waiting in a buffer.
-read doesn'​t return, simply ​because the output is waiting in a buffer.+
  
-See [[http://​wooledge.org:8000/​BashFAQ/​009|this faq entry on Greg's wiki]] for some workarounds.+See [[http://mywiki.wooledge.org/​BashFAQ/​009 | this faq entry on Greg's wiki]] for some workarounds ​and more information on buffering issues.
  
 === background processes === === background processes ===
  
-The file descriptors ​of the coprocesses ​are available ​to the +A coprocess' ​file descriptors are accessible only to the process from which the ''​coproc'' ​was started. They are not inherited ​by subshells.
-shell where you run ''​coproc''​, but they are not inherited. +
-Here a not so meaningful illustration,​ suppose we want something that +
-continuely reads the output of our coprocess and echo the result:+
  
-<​code>​+Here is a not-so-meaningful illustration. Suppose we want to continuously read the output of a coprocess and ''​echo''​ the result: 
 + 
 +<​code ​bash>
 #NOT WORKING #NOT WORKING
 $ coproc awk '​{print "​foo"​ $0;​fflush()}'​ $ coproc awk '​{print "​foo"​ $0;​fflush()}'​
 [2] 23100 [2] 23100
-$ while read -${COPROC[0]};​do ​echo "$REPLY";done &+$ while IFS= read -ru ${COPROC[0]} ​x; do printf '​%s\n' ​"$x"; done &
 [3] 23104 [3] 23104
-$ ./bash: line 243: read: 61: invalid file descriptor: Bad file +bash: line 243: read: 61: invalid file descriptor: Bad file descriptor
-descriptor+
 </​code>​ </​code>​
-it failsbecause the descriptor is not avalaible in the subshell + 
-created by &.+This fails because the file descriptors created by the parent are not available to the subshell created by &.
  
 A possible workaround: A possible workaround:
  
-<​code>​+<​code ​bash>
 #WARNING: for illustration purpose ONLY #WARNING: for illustration purpose ONLY
 # this is not the way to make the coprocess print its output # this is not the way to make the coprocess print its output
Line 116: Line 108:
 [2] 23109 [2] 23109
 $ exec 3<&​${COPROC[0]} $ exec 3<&​${COPROC[0]}
-$ while read -3;do echo "$REPLY";done &+$ while IFS= read -ru x; do printf '​%s\n' ​"$x"; done &
 [3] 23110 [3] 23110
 $ echo bar >&​${COPROC[1]} $ echo bar >&​${COPROC[1]}
Line 122: Line 114:
 </​code>​ </​code>​
  
-Here the fd 3 is inherited+Herefd 3 is inherited.
- +
-=== Only one coprocess at a time === +
- +
-The title says it all, complain to the bug-bash mailing list if you want more.+
  
 ===== Examples ===== ===== Examples =====
  
 ==== Anonymous Coprocess ==== ==== Anonymous Coprocess ====
-__**First let's see an example without ''​NAME'':​**__ 
  
-<​code>​+Unlike ksh, Bash doesn'​t have true anonymous coprocesses. Instead, Bash assigns FDs to a default array named ''​COPROC''​ if no ''​NAME''​ is supplied. Here's an example: 
 + 
 +<​code ​bash>
 $ coproc awk '​{print "​foo"​ $0;​fflush()}'​ $ coproc awk '​{print "​foo"​ $0;​fflush()}'​
 [1] 22978 [1] 22978
 </​code>​ </​code>​
  
-The command starts in the background, coproc returns ​immedately. +This command starts in the background, ​and ''​coproc'' ​returns ​immediatelyTwo new file descriptors are now available via the ''​COPROC'' ​arrayWe can send data to our command:
-new files descriptors are now available via the COPROC array+
-We can send data to our command:+
  
-<​code>​+<​code ​bash>
 $ echo bar >&​${COPROC[1]} $ echo bar >&​${COPROC[1]}
 </​code>​ </​code>​
Line 148: Line 135:
 And then read its output: And then read its output:
  
-<​code>​ +<​code ​bash
-$ read -${COPROC[0]};​printf ​"%s\n" ​"$REPLY"+IFS= read -ru ${COPROC[0]} ​x; printf ​'%s\n' ​"$x"
 foobar foobar
 </​code>​ </​code>​
Line 163: Line 150:
 ==== Named Coprocess ==== ==== Named Coprocess ====
  
-__**Using a named coprocess is as simple, we just need a compound command like when defining a function:**__+Using a named coprocess is simple. We just need a compound command ​(like when defining a function), and the resulting FDs will be assigned to the indexed array ''​NAME''​ we supply instead.
  
-<​code>​+<​code ​bash>
 $ coproc mycoproc { awk '​{print "​foo"​ $0;​fflush()}'​ ;} $ coproc mycoproc { awk '​{print "​foo"​ $0;​fflush()}'​ ;}
 [1] 23058 [1] 23058
 $ echo bar >&​${mycoproc[1]} $ echo bar >&​${mycoproc[1]}
-$ read -${mycoproc[0]};​printf ​"%s\n" ​"$REPLY"+IFS= read -ru ${mycoproc[0]} ​x; printf ​'%s\n' ​"$x"
 foobar foobar
 $ kill $mycoproc_PID $ kill $mycoproc_PID
Line 178: Line 165:
 ==== Redirecting the output of a script to a file and to the screen ==== ==== Redirecting the output of a script to a file and to the screen ====
  
-<​code>​+<​code ​bash>
 #!/bin/bash #!/bin/bash
 # we start tee in the background # we start tee in the background
Line 186: Line 173:
 exec >&​${tee[1]} 2>&1 exec >&​${tee[1]} 2>&1
 </​code>​ </​code>​
- 
  
 ===== Portability considerations ===== ===== Portability considerations =====
  
-  * the ''​coproc''​ keyword is not specified by POSIX(R) +  * The ''​coproc''​ keyword is not specified by POSIX(R) 
-  * other shells might have **different** ways to solve the coprocess problem +  * The ''​coproc''​ keyword appeared in Bash version 4.0-alpha 
-  * the ''​coproc''​ keyword appeared in Bash version 4.0-alpha+  * The ''​-p''​ option to Bash's ''​print''​ loadable is a NOOP and not connected to Bash coprocesses in any way. It is only recognized as an option for ksh compatibility,​ and has no effect. 
 +  * The ''​-p''​ option to Bash's ''​[[commands:​builtin:​read | read]]''​ builtin conflicts with that of all kshes and zsh. The equivalent in those shells is to add a ''​\?​prompt''​ suffix to the first variable name argument to ''​read''​. i.e., if the first variable name given contains a ''?''​ character, the remainder of the argument is used as the prompt string. Since this feature is pointless and redundant, I suggest not using it in either shell. Simply precede the ''​read''​ command with a ''​printf %s prompt >&​2''​. ​
  
 +==== Other shells ====
 +  ​
 +ksh93, mksh, zsh, and Bash all support something called "​coprocesses"​ which all do approximately the same thing. ksh93 and mksh have virtually identical syntax and semantics for coprocs. A //list// operator: ''​|&''​ is added to the language which runs the preceding //​pipeline//​ as a coprocess (This is another reason not to use the special ''​|&''​ pipe operator in Bash -- its syntax is conflicting). The ''​-p''​ option to the ''​read''​ and ''​print''​ builtins can then be used to read and write to the pipe of the coprocess (whose FD isn't yet known). Special redirects are added to move the last spawned coprocess to a different FD: ''<&​p''​ and ''>&​p'',​ at which point it can be accessed at the new FD using ordinary redirection,​ and another coprocess may then be started, again using ''​|&''​.
 +
 +zsh coprocesses are very similar to ksh except in the way they are started. zsh adds the shell reserved word ''​coproc''​ to the pipeline syntax (similar to the way Bash's ''​time''​ keyword works), so that the pipeline that follows is started as a coproc. The coproc'​s input and output FDs can then be accessed and moved using the same ''​read''/''​print''​ ''​-p''​ and redirects used by the ksh shells.
 +
 +It is unfortunate that Bash chose to go against existing practice in their coproc implementation,​ especially considering it was the last of the major shells to incorporate this feature. However, Bash's method accomplishes the same without requiring nearly as much additional syntax. The ''​coproc''​ keyword is easy enough to wrap in a function such that it takes Bash code as an ordinary argument and/or stdin like ''​eval''​. Coprocess functionality in other shells can be similarly wrapped to create a ''​COPROC''​ array automatically.
 +
 +==== Only one coprocess at a time ====
 +
 +The title says it all, complain to the bug-bash mailing list if you want more. See http://​lists.gnu.org/​archive/​html/​bug-bash/​2011-04/​msg00056.html for more details
 +
 +The ability to use multiple coprocesses in Bash is considered "​experimental"​. Bash will throw an error if you attempt to start more than one. This may be overridden at compile-time with the ''​MULTIPLE_COPROCS''​ option. However, at this time there are still issues -- see the above mailing list discussion.
  
 ===== See also ===== ===== See also =====
  
   * [[http://​www.ict.griffith.edu.au/​anthony/​info/​shell/​co-processes.hints|Anthony Thyssen'​s Coprocess Hints]] - excellent summary of everything around the topic   * [[http://​www.ict.griffith.edu.au/​anthony/​info/​shell/​co-processes.hints|Anthony Thyssen'​s Coprocess Hints]] - excellent summary of everything around the topic
- 
  • syntax/keywords/coproc.1317240207.txt
  • Last modified: 2011/09/28 20:03
  • by thebonsai