- Article pages now have a discussion option at the bottom (moderated/captcha, but no registration needed)

Editing files with the ed text editor from scripts

Why ed?

Unlike sed, ed is really a file editor. If you try to change file contents with sed, and the file is open elsewhere and read by some process, you will find out that GNU sed and its -i option of course does not edit in-file. There are circumstances where you may need that, either editing active and open files or not having GNU sed or some other sed with "in-place" option available.

Why ed?

  • maybe your sed doesn't support in-place edit
  • maybe you need to be as portable as possible
  • maybe you need to really edit in-file (and not create a new file like GNU sed)
  • last but not least: standard ed has very good editing and addressing possibilities, compared to standard sed

Don't get me wrong, this is not meant as anti-sed article! It's just meant to show you another way that may do the job.

Commanding ed

Since ed is an interactive text editor, it reads and executes commands that come from stdin. Regarding this, we have several ways to feed our commands to it:

Pipelines

echo '<ED-COMMANDS>' | ed <FILE>

To inject the needed newlines etc. it can be easier to use the printf builtin ("help printf"), here as an example for a complete Bash function to precede text to a filecontent:

# insertHead "$text" "$file"

insertHead() {
  printf '%s\n' H 1i "$1" . w | ed -s "$2"
}

Here-strings

ed <FILE> <<< '<ED-COMMANDS>'

Here-documents

ed <FILE> <<EOF
<ED-COMMANDS>
EOF

Which one you prefer is your own problem ;-) I will use the here-strings, since it looks best here IMHO.

Of course there are some other ways to provide the standard input to ed, for example using process substitution. But these should be enough for daily needs.

Since ed wants commands separated by newlines, I'll use a special Bash quoting method, the C-like strings $'TEXT', as it can interpret a various set of escape sequences and special characters. I'll also usually use the -s option to ed to make it less babbling.

The basic interface

Of course, check the ed manpage for a detailed reference!

ed makes (similar to vi or vim) a difference between a "command mode" and an "interactive mode". For non-interactive use, the command mode will be the most used one.

Commands to ed have a simple and regular structure: zero, one, or two addresses followed by a single-character command, possibly followed by parameters to that command. These addresses specify one or more lines in the text buffer. Every command that requires addresses has default addresses, so that the addresses very often can be omitted.

The line addressing is relative to the so-called current line. If the edit buffer is not empty, the initial value for the current line shall be the last line in the edit buffer, otherwise zero. Generally, the current line is the last line affected by a command. All addresses can only address single lines, not blocks of lines!

Line addresses or commands using regular expressions interpret the POSIX Basic Regular Expressions (BRE). A null BRE is used to reference the most recent used BRE. Since ed addressing is only for single lines, no RE can ever match a newline.

Debugging your ed scripts

By default ed is not very talkative and will simply print a "?" when an error occurs. Interactively you can use the h command to get a short message explaining the last error. You can also turn on a mode that makes ed prints automatically this message with the H command. It is a good idea to always add this command in front of all your ed scripts:

bash > ed -s file <<< $'H\n,df'
?
script, line 2: Invalid command suffix

While working on your script, you might make errors and destroy your file, you might be tempted to first try your script doing something like:

# Works, but there is better

# copy my original file
cp file file.test

# try my script on the file
ed -s file.test <<< $'H\n<ed commands>\nw'

# see the results
cat file.test
There is a much better way though, you can use the ed command p to print the file, now your testing would look like:

ed -s file <<< $'H\n<ed commands>\n,p'
the , (comma) in front of the p command is a shortcut for 1,$ which defines an address range for the first to the last line, ,p thus means print the whole file, after it has been modified. When your script works you only have to replace the ,p by a w.

Of course, even if the file is not modified by the p command, it is always a good idea to keep backups around.

Editing your files

Most of these things can be done with sed, yes. But there are also things that can't be done in sed or can only be done with a very complex code.

Simple word substitutions

Like sed, ed also knows the common s/FROM/TO/ command and it also can take line-addresses. If no substitution is made on the addressed lines, it's considered as an error.

Substitutions through the whole file

ed -s test.txt <<< $',s/Windows(R)-compatible/POSIX-conform/g\nw'

Note: The comma as single address operator is an alias for 1,$ ("all lines").

Substitutions in specific lines

On a line containing fruits, do the substitution:

ed -s test.txt <<< $'/fruits/s/apple/banana/g\nw'

On the 5th line after a line containing fruits, do the substitution:

ed -s test.txt <<< $'/fruits/+5s/apple/banana/g\nw'

Block operations

Delete a block of text

The simple one is a well-known (by position) block of text:

# delete lines number 2 to 4 (2, 3, 4)
ed -s test.txt <<< $'2,5d\nw'

This deletes all lines matching a specific regular expression:

# delete all lines matching foobar
ed -s test.txt <<< $'g/foobar/d\nw'
g/regexp/ applies the command following it to all the lines matching the regexp

Move a block of text

…using the m command: <ADDRESS> m <TARGET-ADDRESS>

This is definitely a thing that can't be done in sed that easy.

# moving lines 5-9 to the end of the file
ed -s test.txt <<< $'5,9m$\nw'

# moving lines 5-9 to line 3
ed -s test.txt <<< $'5,9m3\nw'

Copy a block of text

…using the t command: <ADDRESS> t <TARGET-ADDRESS>

You use the t command just like you use the m (move) command.

# make a copy of lines 5-9 and place it to the end of the file
ed -s test.txt <<< $'5,9t$\nw'

# make a copy of lines 5-9 and place it to line 3
ed -s test.txt <<< $'5,9t3\nw'

Join all lines

…but leave the final newline intact. This is done by an extra command: j (join).

ed -s file <<< $'1,$j\nw'

Compared with two other methods (using tr or sed), you don't have to delete all newlines and manually add one at the end.

File operations

Insert another file

How to insert another file? Just as in sed, it's done by the r command (read). This one inserts another file at the line before the last line (and printd the result to stdout - ,p):

ed -s FILE1 <<< $'$-1 r FILE2\n,p'

To compare, here's a possible sed solution which must use Bash arithmetics and the external program wc:

sed "$(($(wc -l < FILE1)-1))r FILE2" FILE1

# UPDATE here's one which uses GNU sed's "e" parameter for the s-command
#   it executes the commands found in pattern space. I'll take that as a
#   security risk, but well, sometimes GNU > security, you know...
sed '${h;s/.*/cat FILE2/e;G}' FILE1

Another approach, in two invocations of sed, that avoids the use of external commands completely:

sed $'${s/$/\\n-||-/;r FILE2\n}' FILE1 | sed '0,/-||-/{//!h;N;//D};$G'

Pitfalls

ed is not sed

ed and sed might look similar, but the same command(s) might act differently:

/foo/d

In sed /foo/d will delete all lines matching foo, in ed the commands are not repeated on each line so this command will search the next line matching foo and delete it. If you want to delete all lines matching foo, or do a subsitution on all lines matching foo you have to tell ed about it with the g command:

echo $'1\n1\n3' > file

#replace all lines matching 1 by "replacement"
ed -s file <<< $'g/1/s/1/replacement/\n,p' 

#replace the first line matching 1 by "replacement"
#(because it starts searching from the last line)
ed -s file <<< $'s/1/replacement/\n,p'

an error stops the script

You might think that it's not a problem and that the same thing happens with sed and you're right, with the exception that with ed not finding a pattern is an error while sed just continue with the next line. For instance, let's say that you want to change foo by bar on the first line of the file and add something after the next line, ed will stop if it cannot find foo on the first line, sed will continue.

#Gnu sed version
sed -e '1s/foo/bar/' -e '$a\something' file

#First ed version, does nothing if foo is not found on the first line:
ed -s file <<< $'H\n1s/foo/bar/\na\nsomething\n.\nw'

If you want the same behaviour you can use g/foo/ to trick ed. g/foo/ will apply the command on all lines matching foo, thus the substitution will succeed and ed will not goe into error if foo is not found:

#Second version will add the line with "something" even if foo is not found
ed -s file <<< $'H\n1g/foo/s/foo/bar/\na\nsomething\n.\nw'

In fact even a substitution that fails after a g/ / commands does not seem to cause an error ie you can also use a trick like g/./s/foo/bar/ to attempt the substitution on all non blank lines

here documents

shell parameters are expanded

If you don't quote the delimiter, $ has a special meaning. This sounds obvious but is easy to forget this fact when you use address like $-1 or command like $a. Either quote the $ or the delimiter:

#fails
ed -s file << EOF
$a
last line
.
w
EOF 

#ok
ed -s file << EOF
\$a
last line
.
w
EOF 

#ok again
ed -s file << 'EOF'
$a
last line
.
w
EOF 

"." is not a command

The . used to terminate the command "a" must be the only thing on the line. take care of it if you indent the commands:

#ed don't care for the spaces before the commands, but . must be the only thing on the line:
ed -s file << EOF
    a
my content
.
    w
EOF

Simulate other commands

Keep in mind that in all the examples below, the whole file will be read into the memory.

A simple grep

ed -s file <<< 'g/foo/p'

# equivalent
ed -s file <<< 'g/foo/'

Note: Rumors tell that the name of the command grep might be related to the notaion g/RE/p (global ⇒ regular expression ⇒ print).

wc -l

Since the default line for the ed command to print the line number is the last line, a simple = (equal sign) will print this line number and thus the number of lines of the file:

ed -s file <<< '='

cat

Yea, it's a joke…

ed -s file <<< $',p'

…but a similar thing to a cat showing line-endings and escapes can be done with the list command (l):

ed -s file <<< $',l'

FIXME to be continued

Links

Reference:

Misc info / tutorials:

Discussion

noname, 2010/04/12 07:45

wc -l Since the current line (the line addressed when no explicit address is given) is the last line after startup, a = (equal sign) there will print the line number of the last line, i.e. the number of lines of the file: the = (equal sign) doesn't depend where we are now, by default it always prints the number of $ line try ed -s /etc/passwd «< $'1\n='

Jan Schampera, 2010/04/12 16:19

Aaaah you're absolutely right. My ed manpage mentions "default addresses". Thanks!

Enter your comment
 
howto/edit-ed.txt · Last modified: 2012/04/28 07:16 by ormaaj
GNU Free Documentation License 1.3
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0