Linux Training

Linux training for private, public & voluntary sector.

0800 024 8425

City LinUX Training Courses

Section 9.
Shell programming

"Nobody really knows what the Bourne shell’s grammar is. Even examination of the source code is little help."

Tom Duff.

9. Shell programming.

9.1. Standard in, standard error and standard out.

All shell programming requires that we understand the role of the three fundamental input and output streams as evinced in the C programming language; standard in, standard errorand standard out.

Standard in is the input stream, it’s file descriptor is always 0. Standard in is represented at the file level by /dev/stdin.

When using the shell interactively, standard in is normally connected to the keyboard.

Standard out is the output stream. The file description is 1. We can connect to standard out through the file dev/stdout. When using the shell interactively standard out is normally connected to the terminal or a pseudo-terminal device.

Standard error is an alternative output stream that is typically used for for error messages or diagnostics. The file descriptor is 2. File access is through /dev/stderr. Standard error is also usually connected to the terminal device.

Using pipes we can connect the output stream of one program to the input steam of another.

9.2. Input and Output redirection.

Using redirection we direct the output streams stderr and stdout to the same or to different files.

Some metanotation and terminal special characters.

Image imgs/sa101-7.png

9.3. Exercises.

sa101$ ls foo
ls: cannot access foo: No such file or directory
sa101$ ls foo >bar
ls: cannot access foo: No such file or directory
sa101$ ls bar
sa101$ cat bar
sa101$ ls foo 2>bar
sa101$ cat bar
ls: cannot access foo: No such file or directory
sa101$ ls foo bar >foobar 2>&1
sa101$ cat foobar
ls: cannot access foo: No such file or directory

Redirecting the output stream to a file creates the file if it does not exist. If the file does exist it is truncated to zero bytes and then the new content is added. To append to an existing file use ">>".

sa101$ cat >cats
sa101$ cat >>cats
sa101$ cat cats

9.4. Pipes.

Using the pipe character | we can connect standard out from one process to standard in of another.

The command wc counts the number of words, lines and bytes in a file.

We can find those groups of which root is a member using a regular expression.

sa101$ grep root /etc/group

By piping the output from grep into the input of wc we can count them.

sa101$ grep root /etc/group|wc -l

If we want a list of the group names identified by the group command we could alternatively pipe the output to cut.

sa101$ grep root /etc/group|cut -d: -f1

Here we have set 2 options to the cut command:

-d: set the field delimiter to ":".

-f1 print the first field.

When cut gets the option -d it looks for the next non-space character which it will use as the field delimiter. The space character (ASCII decimal 32) is the token separator in UNIX / Linux shells but cut doesn’t care if the command is recast as

sa101$ cut -d : -f 1

This behaviour is not consistent across all UNIX / Linux commands. You will often find with this kind of flag that it is essential to keep the flag and the required character adjacent to each other. When using awk for instance the field separator is set with the -f flag. If the next character is a space ( ) this will be used as the field separator.

9.5. Pipelines

"A pipeline is a sequence of one or more commands separated by one of the control operators | or |& . The format for a pipeline is

[time [-p]] [ ! ] command [ [| | |& ] command2 ... ]

The standard output of command is connected via a pipe to the standard input of command2. This connection is performed before any redirection specified by the command... If |& is used, the standard error of command is connected to command2’s standard input through the pipe; it is shorthand for 2>&1|. This implicit redirection of the standard error is performed after any redirection specified by the command."
Chet Ramey et al - bash man page 2012

9.6. Lists.

"A list is a sequence of one or more pipelines separated by one of the operators ;, &, &&, or ||, and optionally terminated by one of ;, &, or <newline>.

Of these list operators, && and || have equal precedence, followed by ; and &, which have equal precedence" [with each other].

"A sequence of one or more newlines may appear in a list instead of a semicolon....

If a command is terminated by the control operator &, the shell executes the command in the background in a subshell. The shell does not wait for the command to finish, and the return status is 0. Commands separated by a (;) are executed sequentially; the shell waits for each command to terminate in turn. The return status is the exit status of the last command executed.

AND  and OR  lists are sequences of one of more pipelines separated by the && and || control operators, respectively. AND and OR lists are executed with left associativity. An AND list has the form

command1 && command2

command2 is executed if, and only if, command1 returns an exit status of zero.

An OR list has the form

command1 || command2

command2 is executed if and only if command1 returns a non-zero exit status. The return status of AND and OR lists is the exit status of the last command executed in the list."
Chet Ramey et al - bash man page 2012.

9.7. Exercises.

Investigate the use of the command ps using the man command.

Pipe the output of ps -ef to wc. How many processes are running on the local host?

The first field in the output from ps -ef is the process user id.

Using ps, and grep generate a list of processes being run with the root id. Use the same pipeline again to list those process executing with your user id.

Extend the pipeline with wc and count the number of processes being run with the root id.

When using a regular expression to find root processes, the count is different if the string "root" is anchored to the start of line to that of the non-anchored string. Explain why that is so. Investigate the root processes by inspection.

9.8. Setting variables in bash.

Local variables are are set with a simple equate i.e.


The value set in the variable may be obtained by preceding the variable name with $.

sa101$ colour=blue
sa101$ echo $colour

Local variables

Local variables only exist in the current shell.

sa101$ echo $colour
sa101$ bash
sa101$ echo $colour
sa101$ exit
sa101$ echo $colour

Note that the when we return to the shell where the local variable was set it remains available. Global variables.

Global variables are created by exporting the local variable. As always there is more than one way to do this.

sa101$ COLOUR=blue
sa101$ export COLOUR

NB The use of a variable name in the export command is implicit.

sa101$ export COLOUR2=red
sa101$ echo $COLOUR $COLOUR2
blue red
sa101$ bash
sa101$ echo $COLOUR && echo $COLOUR2 && exit

Note that in the last command we are using the shell AND list to output the variables and return to the previous shell.

I have used all upper case names for the exported variables. This is a very strong tradition in UNIX / Linux shell scripting which should be followed. Unfortunately not every exported variable is in upper case.

A listing of all exported variables set in the current shell can be obtained with the commands env. or printenv.

A listing of all variables (including those exported) is produced by the command set.

9.9. Bourne Shell Variables

Bash uses certain shell variables in the same way as the Bourne shell. In some cases, Bash assigns a default value to the variable.

Image imgs/sa101-8.png

9.10. Bash Variables

These variables are set or used by Bash, but other shells do not normally treat them specially. For a full list see man bash.

9.11. Command Substitution

Command substitution allows the output of a command to replace the command itself. Command substitution occurs when a command is enclosed as follows:




Bash performs the expansion by executing command and replacing the command substitution with the standard output of the command, with any trailing newlines deleted. Embedded newlines are not deleted, but they may be removed during word splitting. The command substitution $(cat file) can be replaced by the equivalent but faster $(< file).

When the old-style back quote form of substitution is used, backslash retains its literal meaning except when followed by ’$’, ’‘’, or ’´. The first back quote not preceded by a backslash terminates the command substitution. When using the $(command) form, all characters between the parentheses make up the command; none are treated specially.

Command substitutions may be nested. To nest when using the back quoted form, escape the inner back quotes with backslashes.

If the substitution appears within double quotes, word splitting and filename expansion are not performed on the results." Chet Ramey et al - bash man page 2012

9.12. Exercises.

The date command allows all users to get the current date and time and the superuser (root) to set the system time and date.

Read the man page to discover the options that are available and the formats in which the date may be output.

Note the options -d and --date=STRING. This is one of those less well known but rather powerful options that seem to sneak under the radar from time to time. The -d option should actually be " -d <STRING> and you need to refer to the section DATE STRING, later in the manual, to get a sense of the available STRINGs. Using a largely free format "STRING" it is possible to obtain a correctly formated output for dates other that of "today" or now. E.g. -d "last month" will give the previous month, a tremdously useful tool for automated scanning and reporting of logged events.

sa101$ date
Mon Sep 23 21:28:33 BST 2013
sa101$ date +%b
sa101$ date +%b -d "last month"

Enter the command to output the current date and time.

Adjust the command to output:

the month as a text string.

the day of the month,

the day of the year,

the number of seconds since the epoch.

Now use command substitution to set a variable called dom to the current day of the month. (Variables are set with a simple equate i.e. <variable>=<value>). Take a listing of all current processes and redirect to a file named with the date in the format YYMMDD:HHmmss.

Use the table below to revise the commands and metanotation you have studied so far.

9.13. Tools and metanotation:

Image imgs/sa101-9.png


The layout and associated style sheets for this page are taken from the World Wide Web Consortium and used here under the W3C software licence.