- Article pages now have a discussion option at the bottom (moderated/captcha, but no registration needed)

Quotes and escaping

Quoting and escaping is really an important way to influence the way, Bash treats your input. There are three recognized types:

  • per-character escaping using a backslash: \$stuff
  • weak quoting with double-quotes: "stuff"
  • strong quoting with single-quotes: 'stuff'

All three forms have the very same purpose: They give you general control over parsing, expansion and expansion's results.

Beside these common basic variants, there are some more special quoting methods (like interpreting ANSI-C escapes in a string) you'll meet below.

:!: ATTENTION :!: These quote characters (", double quote and ', single quote) are a syntax element that influences parsing. It is not related to eventual quote characters that are passed as text to the commandline! The syntax-quotes are removed before the command is called! Look:

### NO NO NO: this passes three strings:
###      (1)  "my
###      (2)  multiword
###      (3)  argument"
MYARG="\"my multiword argument\""
somecommand $MYARG

### THIS IS NOT (!!!!) THE SAME AS ###
command "my multiword argument"

### YOU NEED ###
MYARG="my multiword argument"
command "$MYARG"

Per-character escaping

Per-character escaping is useful in different places, also here, on expansions and substitutions. In general, a character that has a special meaning for Bash, like the dollar-sign ($) to introduce some expansion types, can be masked to not have that special meaning using the backslash:

echo \$HOME is set to \"$HOME\"

  • \$HOME won't expand because it's not variable expansion syntax anymore
  • The quotes are masked with the backslash to be literal - otherwise they would be interpreted by Bash

The sequence \<newline> (an unquoted backslash, followed by a <newline> character) is interpreted as line continuation. It is removed from the input stream and thus effectively ignored. Use it to beautify your code:

# escapestr_sed()
# read a stream from stdin and escape characters in text that could be interpreted as
# special characters by sed
escape_sed() {
 sed \
  -e 's/\//\\\//g' \
  -e 's/\&/\\\&/g'
}

The backslash can be used to mask every character that has a special meaning for bash. Exception: Inside a single-quoted string (see below).

Weak quoting

Inside a weak-quoted string there's no special interpretion of:

  • spaces as word-separators (on inital commandline splitting and on word splitting!)
  • single-quotes to introduce strong-quoting (see below)
  • characters for pattern matching
  • pathname expansion
  • process substitution

Everything else, especially parameter expansion, is performed!

ls -l "*"
Will not be expanded. ls gets the literal * as argument. It will, unless you have a file named *, spit out an error.

echo "Your PATH is: $PATH"
Will work as expected. $PATH is expanded, because it's only double- (weak-) quoted.

If a backslash in double-quotes ("weak quoting") occurs, there are 2 ways to deal with it

  • if the baskslash is followed by a character that would have a special meaning even inside double-quotes, the backslash is removed and the following character looses its special meaning
  • if the backslash is followed by a character without special meaning, the backslash is not removed

In particuar this means that "\$" will become $, but "\x" will become \x.

Strong quoting

Strong quoting is very easy to explain:

Inside a single-quoted string nothing(!!!!) is interpreted, except the single-quote that closes the quoting.

echo 'Your PATH is: $PATH'
That $PATH won't be expanded, it's interpreted as normal ordinary text, because it's surrounded by strong quotes.

In practise that means, to produce a text like Here's my test… as a single-quoted string, you have to leave and re-enter the single-quoting to get the character "'" as literal text:

# WRONG
echo 'Here's my test...'

# RIGHT
echo 'Here'\''s my test...'

# ALTERNATIVE: It's also possible to mix-and-match quotes for readability:
echo "Here's my test"

ANSI C like strings

There's another quoting mechanism, Bash provides: Strings that are scanned for ANSI C like escape sequences. The Syntax is

$'string'
where the following escape sequences are decoded in string:

Code Meaning
\" double-quote
\' single-quote
\\ backslash
\a terminal alert character (bell)
\b backspace
\e escape (ASCII 033)
\E escape (ASCII 033) \E is non-standard
\f form feed
\n newline
\r carriage return
\t horizontal tab
\v vertical tab
\cx a control-x character, for example $'\cZ' to print the control sequence composed by Ctrl-Z (^Z)
\uXXXX Interprets XXXX as hexadecimal number and prints the corresponding character from the character set (4 digits) (Bash 4.2-alpha)
\UXXXXXXXX Interprets XXXX as hexadecimal number and prints the corresponding character from the character set (8 digits) (Bash 4.2-alpha)
\nnn the eight-bit character whose value is the octal value nnn (one to three digits)
\xHH the eight-bit character whose value is the hexadecimal value HH (one or two hex digits)

This is especially useful when you want to give special characters as arguments to some programs, like giving a newline to sed.

The resulting text is treated as if it was single-quoted. No further expansions happen.

The $'...' syntax comes from ksh93, but is portable to most modern shells including pdksh. A specification for it was accepted for SUS issue 7. There are still some stragglers, such as most ash variants including dash, (except busybox built with "bash compatibility" features).

I18N/L10N

A dollar-sign followed by a double-quoted string, for example

echo $"generating database..."
means I18N. If there is a translation available for that string, it is used instead of the given text. If not, or if the locale is C/POSIX, the dollar sign simply is ignored, which results in a normal double-quoted string.

If the string was replaced (translated), the result is double-quoted.

In case you're a C-programmer: The purpose of $"…" is the same as for gettext() or _().

For useful examples to localize your scripts, please see Appendix I of the Advanced Bash Scripting Guide.

Attention: There is a security hole. Please read in the gettext documentation

Common mistakes

String lists in for-loops

The classic for-loop uses a list of words to iterate through. This list can - of course - also be in a variable:

mylist="DOG CAT BIRD HORSE"

WRONG way to iterate through this list:

for animal in "$mylist"; do
    echo $animal
done
Why? Due to the double-quotes, technically, the expansion of $mylist is seen as one word. The for-loop iterates exactly one time, with animal set to the whole list.

RIGHT way to iterate through this list:

for animal in $mylist; do
    echo $animal
done

Working out the test-command

The command test or [ … ] ( the classic test command) is a normal ordinary command, so normal ordinary syntax rules apply. Let's take string comparison as example:

[ WORD = WORD ]

The ] at the end is a convenience; if you type which [ you will see that there is in fact a binary with that name. So if we were writing this as just a test command it would be:

test WORD = WORD

When you compare variables, it's wise to quote them. Let's invent a test string with spaces:

mystring="my string"

And now check that string against the word "testword":

[ $mystring = testword ] # WRONG!!!
This fails! These are too much arguments for the string comparison test. After all expansions performed you really execute:
[ my string = testword ]
test my string = testword
Which is wrong, because my and string are two separate arguments.

So what you really want to do is:

[ "$mystring" = testword ] # RIGHT!!!

test 'my string' = testword

Now the command has three parameters, which makes sense for a binary (two argument) operator.

Hint: Inside the conditional expression ([[ ]]) Bash doesn't perform word splitting, and thus you don't need to quote your variable references - they are always seen as "one word".

See also

Discussion

Szilvi, 2012/09/02 21:56

Thanks for posting this Great article! It really helped my see these magic things of bash more clearly. But I still have a problem that I can't solve. I'll post it here, maybe someone can help me.

So here is it: I have a test folder with some subfolders:

# find .
.
./etc
./etc/a
./a
./a/b
./e
./c
./c/d

I want to list all folders and files except the etc folder and its contents I'll use this command, and i get exactly what i want:

# find . ! -wholename "./etc*"
.
./a
./a/b
./e
./c
./c/d

Till this point it's fine. But inside a bash script I NEED THIS AS STRING because i'm building up the condition based on some internal values. Now look at this:

# cond='! -wholename "./etc*"'

And when i run find again…

# find . $cond
.
./etc
./etc/a
./a
./a/b
./e
./c
./c/d

…it lists the etc folder and its contents which it shouldn't.

I'm sure it's a quotation problem, and i tried all variations i know, but i wasn't able to solve the problem. Where is the mistake?

I appreciate your help Szilvi

Jan Schampera, 2012/09/03 11:27

Yea, this is a quoting problem.

The text you write in the variable ("./etc") is really text. The quotes you give on commandline (find . ! -wholename "./etc*") is syntax. You can't "store syntax in variables". The syntax (quoting) is used to tell Bash what a word is when it can't automatically detect it (and especially here, to not make Bash expand the wildcard itself, but to pass it as text to find).

In general, you should construct an array where every element contains one "word" and the whole array forms the arguments you want to pass to find:

cond=(! -wholename "./etc*")
# use it
find ... "${cond[@]}" ...

The most correct fix would be using the -prune test/action from find, anyways. Please see the find article on Greg's wiki

Aaron, 2012/10/20 02:01

From command line this works

mailx -s "GM $HOST Case Logs for Ticket" aaron.brandt@XXXXX.com </home/ab007652/tmp/GM.$HOST.$$ \r.\r

But from within a script is doesnt, How come?

Doesnt work.. I have a problem with old version of mail so this is the only work around

# for your records.
        printf "\n\E[1;33mSend your trouble shooting to the ticket?\033[m [y/N] "
        read -e TICKET
if [ "$TICKET" == "y" ];
        then
        mailx -s "Ticket #: $TKT" smc@XXXX.com </home/ab007652/tmp/GM.$HOST.$$ \r.\r
else
        mailx -s "GM $HOST Case Logs for Ticket: $TKT" $USR@XXXX.com </home/ab007652/tmp/GM.$HOST.$$ \r.\r
fi

Jan Schampera, 2012/10/20 13:01

What is this \r.\r there? Just give mailx the text file as input and test.

Aaron, 2012/10/20 17:37

Those are carrage returns, If I use mailx -s "Ticket #: $TKT" smc@XXXX.com </home/ab007652/tmp/GM.$HOST.$$ by itself it just hangs until I issue the "."

From a bash prompt the \r.\r works but if I use it in a script it does not. It is as if the \r.\r isnt even there.

If I use mail -s it doesnt include the subject line only mailx -s does.

Jan Schampera, 2012/10/20 18:49

The best way for you would be writing a mail (in mail format, with headers etc.) and deliver it with your MTA's sendmail -oi (or equivalent).

Everything else is too much guessing.

sshaw, 2013/12/31 04:14

set -x 

Is a good way for one to debug and/or better understand bash quoting:

~ >dirs='/etc /tmp'
~ >(set -x; ls "$dirs" > /dev/null)
+ ls '/etc /tmp'
ls: /etc /tmp: No such file or directory
~ >(set -x; ls $dirs > /dev/null)
+ ls /etc /tmp
~ >dirs="/etc/*"
~ >(set -x; ls "$dirs" > /dev/null)
+ ls '/etc/*'
ls: /etc/*: No such file or directory
~ >(set -x; ls $dirs > /dev/null)
+ ls /etc/RemoteManagement.launchd /etc/afpovertcp.cfg /etc/aliases
# ... 

Enter your comment
 
syntax/quoting.txt · Last modified: 2013/07/07 12:34 by ormaaj
GNU Free Documentation License 1.3
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0