Physics 210: Unix / Linux

Index

• Introduction and Motivation
• Interaction with the Shell: Keyboard and Mouse Features
  
• Linux Desktop Environments
  
• Files and Directories
  • Absolute and Relative Pathnames, Working Directory
  • Home directories
  • "Dot" and "Dot-Dot"
  • Filenames
• Commands Overview
  • General Structure
  • Executables and Paths
  • Control Characters
  • bash Startup Files
  • Hidden Files
    
  • Shell Aliases
  • Default 210 Startup Files
    
  • Shell Options
• Basic Commands
  • Getting Help or Information
    • man
  • Communicating with Other Machines
    • ssh
    • Mail
  • Exiting bash / Logging out
  • exit
  • logout
  • Changing Your Password
  • passwd
    
  • Creating, Manipulating and Viewing Files
    • Text editors: gedit, vi/vim (gvim) or emacs (xemacs)
    • more
    • lpr
    • cd and pwd
    • ls
    • mkdir
    • cp
    • mv
    • rm
    • chmod
    • scp
• More about bash
  
  • Local Variables
    
  • Environment Variables
  • Using bash Pattern Matching
  • The bash History
    
  • Standard Input, Standard Output and Standard Error
  • Input and Output Redirection
  • Pipes
  • Regular Expressions and grep
    
  • Using Quotes: (' ', " ", and ` `)
    • Forward quotes: ' '
    • Double quotes: " "
    • Backward quotes: ` `
  • Job Control
• Basic Shell Programming

INTRODUCTION AND MOTIVATION

• Portability: All Unix systems support the command-line approach, and by sticking with standard features, what you learn on the Linux machines that you will be using in the course will be applicable on virtually all Unix systems.
• Power: Commands can be extended and combined in a straightforward way. These new commands include not only those supplied by the operating system, but also new commands (programs) that you can create using shell scripting and programming features, or with specific programming languages such as C, Fortran, Java, Python and many others.
  
• Speed: Command-line interfaces minimize the amount of information that needs to be passed from machine to machine when working remotely. If you just want to accomplish a few quick tasks, few things are more annoying than a sluggish GUI. As available network speeds get faster and faster, this becomes less of a concern, but one still encounters situations where the command-line is more effective than a GUI.
  

When you type commands in Unix, you are actually interacting with the OS through a special program called a shell, which provides a more user-friendly command-line interface than that defined by the basic Unix commands themselves. In this course, we will  focus on the use of bash ("bourne again shell") which over the years has become the most commonly used shell in the Linux community. However, you should be aware that there are other shells available for your use, and that you can, in fact, change your default shell using the chsh command. In particular, tcsh is still in widespread use and if you are interested in learning a bit about its features, you can start with a version of these notes that discusses it in some detail. However, particularly if you are new to Linux I recommend that you stick with bash for the duration of the course.

In the notes that follow, commands that you type to the shell, as well as the output from the commands and the shell prompt (usually denoted "% ") will appear in typewriter font. Here's an example

% pwd
/home/choptuik
% date
Thu Aug 22 13:34:01 PDT 2013
%

Interacting with the Shell: Keyboard and Mouse Features

One very useful feature of bash is the ability to recall previously executed commands, and to edit them via the "arrow" keys (as well as "Delete" and "Backspace"). After you have typed a few commands, hit the "up arrow" key a few times and note how you scroll back through the commands you have previously issued. Use the "down arrow" the same number of times to return to the basic prompt.

There is another handy feature of bash and other shells (which should be enabled by default on your lab accounts), which is generically known as completion. The basic idea is that you can type the first few characters of a command name (i.e. the first word typed after the prompt), or a filename (i.e. essentially any word that follows a command name), and then type the TAB key.  If there is a unique command name (filename) that begins with the characters that you have typed, the shell will automatically complete the command name (filename). Especially for long names this can save a considerable amount of typing.  If the initial few characters that you have typed do not uniquely identify a command or filename, then if you type a second TAB (with no other intervening characters), the shell will display a listing of all possible matches, then allow you to continue typing. As you enter additional characters you can again use TAB at any point to attempt a completion (or two consecutive TABS to see all possible completions).

You should also become familiar with the use of the mouse (or equivalent mouse device) to select, cut and paste text within a shell, as well as within many other Unix applications. Historically, Unix systems have tended to use a three button mouse, and the following instructions assume that you are using one: in the current era, mice usually have only two buttons. In this case, third-button-actions can often be emulated by depressing the two buttons simultaneously.  Additionally, for those mice that have a roller, depressing the roller will act as a third-button click. Here, then, are the basic text manipulation actions that can be achieved using the mouse:

• Select (highlight) arbitrary text: Depress and hold down the left mouse button while sweeping over the desired selection. The selected text will be highlighted.
  
• Select a single whitespace-delimited string (word): Double click on the word using the left button. The selected word will be highlighted. (Note: whitespace = spaces/blanks or TAB characters)
  
• Select a single line of text: Triple click with the left button anywhere on the desired line. The selected line will be highlighted.
• Paste selected (highlighted) text:  Position the cursor at the desired insertion point using a single click of the left mouse or via the arrow keys. Notice that this action will clear the highlighting of the selected text, but that in all of the above cases, the selection process has automatically copied the selected text to a global buffer (what you might know as a clipboard). Depress the middle button, and the previously selected text will be inserted before the cursor.  The buffer/clipboard contents remain intact until the next selection action.
• Cut text: Select text as above and then depress either the delete or backspace key.  The selected text will be deleted and stored in the buffer/clipboard so you can, for example, subsequently paste it per the previous instruction.

Linux Desktop Environments

• KDE
• GNOME

The Ubuntu distribution installed on the lab computers uses a customized version of the GNOME desktop called Unity.  Documentation for Unity is available HERE, as well as via the desktop itself, once you login to one of the lab machines.  In our early lab sessions you will become familiar with key features of Unity that will be needed for course work, and it should be a straightforward matter for you to become more expert in its use as the course progresses.

FILES AND DIRECTORIES

• Plain files: that contain specific information such as plain text, MATLAB code, executable code, PDF code, a Maple worksheet etc.
• Directories: special files that serve as containers for other files (including other directories). In the Windows/Mac worlds, directories are known as folders, and although the 'directory' terminology is used exclusively in these notes, I fully expect that many of you will only find it natural to use 'folder'
  

/home/choptuik/junk

/home/choptuik

/home

/

% pwd
/home/choptuik

foo

dir1/dir2/foo

/home/choptuik

/home/choptuik/foo
/home/choptuik/dir1/dir2/foo

Home directories: Each user of a Unix system typically has a single directory called his/her home directory that serves as the base of his/her personal files. The command cd (change [working] directory) with no arguments will always take you to your home directory. On the lab machines you should see something like this

% cd
% pwd
/home/phys210t

/home/choptuik

% cd ~

% cd ~/dir1/dir2

% cd /home/choptuik

% cd /home/choptuik/dir1/dir2

% cd ~phys210t

"Dot" and "Dot-Dot": Unix uses a single period (.) and two periods (..) to refer to the working directory and the parent of the working directory, respectively:

% cd ~phys210t/dir1
% pwd
/home/phys210t/dir1
% cd ..
% pwd
/home/phys210t
% cd .
% pwd
/home/phys210t

% cd .

Filenames: There are relatively few restrictions on filenames in Unix. On most systems (including Linux systems), the length of a filename cannot exceed 255 characters. Any character except slash (/) (for obvious reasons) and "null" may be used. However, you should avoid using characters that are special to the shell (such as ( ) * ? $ !) as well as blanks (spaces). In fact, it is probably a good idea to stick to the set:

a-z A-Z 0-9 _ . -

program.c  (extension .c)
paper.tex  (extension .tex)
the.longextension (extension .longextension)
noextension (no extension)

this_is_a_long_file_name
this-is-another-long-file-name

Unix generally makes it difficult for you to create a filename that starts with a minus. It is also non-trivial to get rid of such a file, so be careful. If you accidentally create a file with a name containing characters special to the shell (such as * or ?), the best thing to do is remove or rename (move) the file immediately by enclosing its name in single quotes to prevent shell evaluation:

% rm -i 'file_name_with_an_embedded_*_asterisk'
% mv 'file_name_with_an_embedded_*_asterisk' sane_name

COMMANDS OVERVIEW

command_name [options] [arguments]

% ls -l
% cp -R ...
% man -k ...

% ls --color=auto -CF

% cp file1 file2  
% grep 'a string' file1

% grep a string file1

Executables and Paths: In Unix, a command such as ls or cp is usually the name of a file that is known to the system to be executable (see the discussion of chmod below). To invoke the command, you must either type the absolute pathname of the executable file or ensure that the file can be found in one of the directories specified by your path. In bash, the current list of directories that constitute your path is maintained in the environment variable, PATH (note that case is significant for bash variables). To display the contents of this variable, type:

% echo $PATH

.:/home/phys210t/bin:/home/phys210/bin:/opt/maple16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

% ls

% helpme
-bash: helpme: command not found

Control Characters: The following control characters typically have the following special meanings or uses within bash. (If they don't, then your keyboard bindings are "non-standard" and you may wish to contact the system administrator about it.) You should familiarize yourself with the action and typical usage of each. I will use a caret (^) to denote the Control (Ctrl) key. Then

% ^Z 

• ^D: End-of-file (EOF). Type ^D to signal end of input when interacting with a program (such as Mail) that is reading input from the terminal. Here's an example using Mail on the lab machines:

  % Mail -s "test message" choptuik@physics.ubc.ca
  This is a one line message.
  ^D
  Cc:
  % 
  

  If you try the above exercise, you will notice that the shell does not "echo" the ^D. This is typical of control characters---you must know when and where to type them and what sort of behaviour to expect. In this case, Mail prompts for an optional list of addresses to which the message is to be carbon-copied, but other commands, such as cat, will not echo anything. In almost all cases, however, you should be presented with a command prompt once you have typed ^D. Also, by default, bash exits when it encounters EOF, so if you type ^D at a shell prompt, you may find that you are logged out from the terminal session. If you don't like this behaviour (I don't), include the following line in your ~/.bashrc (it is included in the course default ~/.bashrc):

  set -o ignoreeof
  

  Note that set is a bash builtin command (i.e. a sub-command of the bash interpreter) that controls many features of the operation of bash, and is discussed in slightly more detail in the shell options section below.
  


• ^C: Interrupt. Type ^C to kill (stop in a non-restartable fashion) commands (processes) that you have started from the command-line. This is particularly useful for commands that are taking much longer to execute or producing much more output to the terminal than you had anticipated.
  
  
• ^Z: Suspend. Type ^Z to suspend (stop in a restartable fashion) commands that you have started from the shell. It is often convenient to temporarily halt execution of a command as will be discussed in job control below.
  
  
• ^V: Escape Special Characters: Type ^V in order to "escape" (protect from shell evaluation) certain other control characters and special keys.

  Example 1: Search for [TAB] characters in file foo using grep:

  % grep '^V[TAB]' foo
  

  Example 2: Removing "Carriage returns" (^M) using vi

  :%s/^V^M//g
  

bash Startup Files: You can customize the environment that results whenever a new bash starts by creating and/or modifying certain startup files that reside in your home directory.  Before  proceeding, however, we must note that bash makes a distinction between login shells and purely interactive shells, and executes a different startup file (assuming that it exists) in each case.  You will start a bash login shell if, for example, you connect to the main physics server, hyper, from a remote machine such as your home computer.  In this case you will have to go through the login procedure of typing your user (account) name and your password.  On the other hand, shells that you start from the Linux GUI on one of the workstations in the computer lab, for example, will be purely interactive. In this instance the computer already "knows" who you are and that you are logged in, and does not ask you for your login name or password.  Given this, the two most important startup files (there are more, but we don't have to discuss them here, and you can get the full details from the bash man page) are as follows

• ~/.profile
  Commands in this file are executed each time a login bash is started.
• ~/.bashrc
  Commands in this file are executed each time a purely interactive bash is started.

1. Do all of your customizations in the ~/.bashrc file,
2. Keep the ~/.profile file as simple as possible
3. As the last command in the ~/.profile file, execute the commands in the ~/.bashrc file, using for example the lines
   

   if [ -f ~/.bashrc ]; then
     source ~/.bashrc
   fi
   

   Note that the first and third lines of the above constitute a test for the existence of the ~/.bashrc file, while the source ~/.bashrc command then executes the contents of the ~/.bashrc file, provided that it exists (i.e. the source command tells the shell to execute the commands in the file that is supplied as an argument to it). When sourcing or otherwise manipulating files in startup files, you should always perform this type of existence check.  Otherwise an error message is apt to be generated, and this can sometimes cause problems with the overall startup process.
   

1. ALWAYS MAKE A BACKUP COPY of  the startup file, using for example
   

   % cp ~/.bashrc ~/.bashrc.O
   

2. During the process of modifying one of the startup files, always keep at least one terminal window open to the machine until you have tested (via ssh to the machine, for example, see below for information on ssh) that you can still login.  The reason that this is so vital is that it is possible that your modifications will introduce one or more bugs into the startup files which can make it impossible for you to login.  However, as long as the terminal window to the machine remains open, you can kill the failed login process (e.g. the ssh command) as necessary using ^C, try to correct the bugs, and repeat the test login procedure.  Finally, if you can't get your desired modifications to work, you can then restore the original contents of the startup file(s) from the saved copy using, e.g.
   

   % cp ~/.bashrc.O ~/.bashrc
   

% cd; ls -a

% ls -d .??*

Shell Aliases: As you will discover, the syntax of many Unix commands is quite complicated and furthermore, the "bare-bones" version of some commands is less than ideal for interactive use, particularly by novices. bash provides a simple mechanism called aliasing that allows you to easily remedy these deficiencies in many cases. The basic syntax for aliasing is

% alias name=string

% alias ls='ls -FC'

The following lines define aliases for rm, cp and mv (see below) that will not clobber (destroy/overwrite) files without first asking you for explicit confirmation. They are highly recommended for novices and experts alike.

% alias rm='rm -i'
% alias cp='cp -i'
% alias mv='mv -i'

% alias RM='/bin/rm'
% alias CP='/bin/cp'
% alias MV='/bin/mv'

% alias

if [ -f ~/.aliases ]; then
   source ~/.aliases
fi

• .bashrc
• .profile
• .aliases

% echo $SHELLOPTS
braceexpand:emacs:hashall:histexpand:history:ignoreeof:interactive-comments:monitor:noclobber

BASIC COMMANDS

Getting Help or Information:

man

Use man (short for manual) to print information about a specific Unix command, or to print a list of commands that have something to do with a specified topic (-k option, for keyword). Although these days it is often easy to get info about a command via an online search (Google e.g.) it is still worth becoming familiar with man. Although the level of intelligibility for commands (especially for novices) varies widely, most basic commands are thoroughly described in their man pages, with usage examples in many cases. It helps to develop an ability to scan quickly through text looking for specific information you feel will be of use. Examples of man invocations include:

% man man

% man cp

% man -k compiler

Output from man will typically look like

% man man

NAME
       man - an interface to the on-line reference manuals

SYNOPSIS
       man  [-C  file]  [-d]  [-D]  [--warnings[=warnings]]  [-R encoding] [-L
       locale] [-m system[,...]] [-M path] [-S list]  [-e  extension]  [-i|-I]
       [--regex|--wildcard]  [--names-only]  [-a]  [-u] [-P pager] [-r prompt]
       [-7] [-E encoding] [--no-hyphenation]  [-p  string]  [-t]  [-T[device]]
       [-H[browser]] [-X[dpi]] [-Z] [[section] page ...] ...
       man -k [apropos options] regexp ...
       man -f [whatis options] page ...
       man  -l  [-C  file] [-d] [-D] [--warnings[=warnings]] [-R encoding] [-L
       locale] [-P pager] [-r prompt] [-7]  [-E  encoding]  [-p  string]  [-t]
       [-T[device]] [-H[browser]] [-X[dpi]] [-Z] file ...
       man -w|-W [-C file] [-d] [-D] page ...
       man -c [-C file] [-d] [-D] page ...
       man [-hV]

DESCRIPTION
       man  is  the  system’s manual pager. Each page argument given to man is
       normally the name of a program, utility or function.  The  manual  page
       associated  with each of these arguments is then found and displayed. A
       section, if provided, will direct man to look only in that  section  of
       the  manual.   The  default action is to search in all of the available
       sections, following a pre-defined order and to show only the first page
       found, even if page exists in several sections.
                                 .
                                 .
                                 .

% man -k language

ajc (1)              - compiler and bytecode weaver for the AspectJ and Java languages
asy (1)              - Asymptote: a script-based vector graphics language
awk (1)              - pattern scanning and text processing language
bc (1)               - An arbitrary precision calculator language
                               .
                               .
                               .
vdmsish (3perl)       - Perl pragma to control VMS-specific language features 
xasy (1x)            - script-based vector graphics language
XtSetLanguageProc (3) - set the language procedure
YAML (3pm)           - YAML Ain't Markup Language (tm)
yorick (1)           - interpreted language for numerical analysis and postprocessing                                                                                      

% man awk

ssh

Use ssh to establish a secure (i.e. encrypted) connection from one Unix machine to another. This is the basic mechanism that can be used to (1) start a Unix shell on a remote host and (2) execute one or more Unix commands on such a machine. During this course, you will probably find this command most useful if you are using a ssh client such as putty on one of your personal machines to login to the main physics server, hyper.phas.ubc.ca.  However, ssh is extensively used in the "real world" for logging in from one machine to another, where each machine is typically running some flavour of Unix.

Note that you cannot ssh into any of the lab machines per se, but this is irrelevant since your home directory on hyper is the same as it is on the lab machines (so that all of your directories/files are accessible on hyper), and all of the software used in the course (xmaple and octave in particular) is available on hyper.

Typical usage of ssh is

bh0% ssh hyper.phas.ubc.ca -l choptuik

choptuik@hyper.phas.ubc.ca's password:

% ssh choptuik@hyper.phas.ubc.ca
% slogin hyper.phas.ubc.ca -l choptuik
% slogin choptuik@hyper.phas.ubc.ca

The first of the above alternate forms is generally the most convenient to type, and is the one that I use.

If additional arguments are supplied to ssh, they are interpreted as commands to be executed remotely. In this case, control immediately returns to the invoking shell after completion (successful, or otherwise) of the command(s), as seen in the following examples, where the password prompts have been suppressed:

hyper% ssh matt@bh0.phas.ubc.ca date
Thu Aug 22 14:27:49 PDT 2013

hyper% ssh matt@bh0.phas.ubc.ca 'pwd; date'
/home/matt
Thu Aug 22 14:28:00 PDT 2013

hyper%

lab-machine% ssh -X matt@bh0.phas.ubc.ca

bh0% gedit

bh0% gedit
gedit: cannot connect to X server

Gory Details of ssh: In contrast to many of the other commands described here, the behaviour of ssh depends crucially on the current context for the command, which, by convention, ssh stores as a number of files in the directory ~/.ssh (i.e. as a number of files in a directory named .ssh, located in your home directory). If ~/.ssh does not exist (which nominally means that you have yet to issue the ssh command from that specific account), it will automatically be created, and certain files within ~/.ssh will be created and/or modified.

For example, assume that as choptuik@hyper.phas.ubc.ca, I have never used the ssh command. However, I can and do login into hyper.phas.ubc.ca (as choptuik) using one of the workstations in the computer lab, and start up a command shell. I can now establish a secure connection to my account on bh0.phas.ubc.ca via ssh as follows:

hyper% ssh matt@bh0.phas.ubc.ca

The authenticity of host 'bh0.phas.ubc.ca (142.103.234.164)' can't be established.
RSA key fingerprint is b1:62:66:2f:39:98:e8:ec:b3:91:cd:b4:1f:b1:d5:48.
Are you sure you want to continue connecting (yes/no)?

Warning: Permanently added 'bh0.phas.ubc.ca,142.103.234.164' (RSA) to the list of known hosts.
matt@bh0.phas.ubc.ca's password:    

When I'm done my work on bh0, I can use the logout (or exit) command

bh0% logout
Connection to bh0.phas.ubc.ca closed.
hyper%

Assuming I've done the above, I now see that the directory ~/.ssh has been created, and contains the file known_hosts:

hyper% cd ~/.ssh
hyper% ls
known_hosts

hyper% ssh matt@bh0.phas.ubc.ca
matt@bh0.phas.ubc.ca's password:

Mail

All of you are undoubtedly already expert in the sending and receiving of e-mail, using one or more of your favourite mail clients, and I will use the "Connect/Blackboard" system to communicate electronically with the class as a whole. Note that this mechanism does not allow me to directly see your individual e-mail addresses, so if you want to get personalized e-mail from myself or the TAs, you will need to first send us a message to which we can reply.

In this course, you will not be required to use a mail client on your physics account.

However, in the spirit of mastering command-line Unix, we consider a brief illustration of the use of a very old, and, especially for your generation, a very primitive mail client known as Mail (we've already encountered this program in our discussion of control characters):

Again, here's a basic example showing how to use Mail to send a message:

% Mail -s "this is the subject" choptuik@phas.ubc.ca
This is a one line test message.
^D
Cc:
%

% Mail -s "sending a file as a message" matt@laplace.phas.ubc.ca < message

If you are interested, you can consult the man page for Mail for additional information on its use. Most importantly, you should note that although it may seem like obsolete technology, the type of usage illustrated above can be quite useful.  For example, you might write a script that takes a long time to accomplish some task. It can then be convenient to have the script send you a message when it has completed. This is cumbersome, if not impossible, to accomplish using GUI-based mail clients, but is essentially trivial with Mail.

Exiting bash / Logging out:

exit

Type exit to leave both login and purely interactive shells.

If there are suspended jobs (see job control below), you will get a warning message, and you will not be logged out.

% exit
There are stopped jobs

logout

logout has the same effect as exit, but can only be used in login shells.  If you enter logout in a purely interactive shell, you will receive the message

% logout
bash: logout: not login shell: use `exit'

% ssh choptuik@hyper.physics.ubc.ca
choptuik@hyper's password:
Last login: Mon Aug 27 11:51:30 2012 from bh0.physics.ubc.ca

hyper% passwd

Old password: 
Password: 
Checking, please wait ...
Reenter Password: 
Password okay.  Changing password ...
LDAP - Setting LDAP password for choptuik.
LDAP2 - Setting LDAP password for choptuik.
IPA02 - Setting LDAP password for choptuik.
Setting password in Windows domain.
Exited with code: 0
Propagating to other servers..alpha..beta..delta---------------------------------------
      AUTHORIZED LOGINS ONLY
---------------------------------------

This host is only for running alpine.

   Do not run other programs!!

---------------------------------------
..mail..omega..done. 

hyper%

Creating, Manipulating and Viewing Files (including Directories):

Text editors: gedit, vi (vim / gvim) or emacs (xemacs)

Although you may find it somewhat painful (especially if you've developed a serious relationship with Microsoft Word), I consider it an absolutely key goal for everyone in this course to become reasonably proficient in at least one of the text editors: gedit, vi, gvim or emacs (or some other text editor installed on the lab machines that is instructor-approved). Note that a text editor, although similar in spirit to a word processor, really has a different fundamental purpose. As the name suggests, this is to create and manipulate files that contain plain text (i.e. files for which many of the features of modern word processors, such as the ability to create documents that use different fonts, font-sizes, styles, colours ... and/or that include tables, figures etc. etc. are completely irrelevant).  Plain text files are central to the use of most programming languages and programming environments in Unix, as well as to the configuration and customization of the operating system itself. During the course, many of the homework assignments, as well as your term projects, will require the creation of this type of file.

For many, if not most of you, gedit is probably the most straightforward choice, and is known simply as "Text Editor" in Unity.  Again, in an early lab session you will learn how to start up this editor and use it to create and save simple text files. Of the three editors listed above, gedit provides the most intuitive user interface, namely a GUI which, though not as snazzy looking, closely resembles that of word processors with which you will no doubt be familiar. I expect that most of you will be able to readily master gedit, with little if any help,  If you do need assistance, the application provides extensive help facilities. Note that gedit does not fully implement the mouse-text-manipulation features discussed in the introductory section: various pieces of text can be selected using the sweeping, double and triple clicking actions described above, but then you must use the GUIs cut/copy/paste operations (via the icons, menu entries or hot keys). Also observe that gedit may not be available on some non-Linux Unix systems that you may need to work on, but given the introductory nature of this course, I don't consider that sufficient reason to dissuade you from its use.

vi and emacs are the two major "traditional" text-editors that are found on most Unix implementations (certainly vi should be!) Both are themselves text-based; that is, they do not provide a GUI, and for the most part, do not allow for manipulation of the text being edited with the mouse.  Moreover, vi developed a reputation for being suitable mainly for "hardcore" users, who didn't mind dealing with its rather unique, simplistic, and not entirely intuitive (to put it mildly!) user interface. emacs, on the other hand, was viewed as a much more elegant, powerful and full-featured editor, to the point that with a suitable configuration, you could get emacs to do just about everything but make coffee.  Personally, I use vi, since that's the editor I first encountered on Unix, and although it is a good idea for any Unix user to know a bit about vi (if only because some of its syntax appears in many other standard Unix commands), I would strongly recommend that any of you who know neither vi or emacs, and who wish to learn one of them, seriously consider learning emacs, at least to start.  In addition, if you intend to become a serious Unix user, then you really should learn how to use one of these text-based editors, since you will almost certainly encounter situations where you need to edit files using a basic terminal session that will not support the use of a GUI.

Over the years, vi on Linux systems has evolved to become vim (for Vi IMproved), so that, for example, if you execute vi on the lab machines, it is actually vim that starts up.  This is a minor point, but something to keep in mind should you be looking online for information concerning vi (i.e. you should probably search for information on vim).

The good news is that there is now a GUI-based version of vi / vim called, gvim, and the version of emacs installed on the lab machines uses a GUI by default. You are more than welcome to use these rather than their text-based antecedents for course work. Again note, however, that these GUIs are not as user friendly as the word processing software that you are probably accustomed to (or gedit for that matter) but they will, for example, allow you to use the mouse to position the cursor as well as to highlight text and cut. Again, it is up to you which text editor you choose to use, but we really want you to learn to use at least one that isn't a Microsoft or Mac/Apple product!

Not least because all of these text editors are complex pieces of software, it is impossible to give any sort of an overview of their use in these introductory notes.  You are therefore directed to the help facilities supplied by the editors themselves, as well as to the online resources listed in the course Online Resources page, for links to web sites and specific documents that should aid in you in your task of mastering your editor of choice.  This in one job that we expect you to do largely on your own, but the TAs and I, will of course, be more than happy to help out as we can, and, I hope, some of your classmates will be able to be of assistance as well.

Finally, if you know of, or find another text editor on the lab machines that you would prefer to use, please talk to me or send me an e-mail message and I will let you know whether I approve of its use, or whether I would rather you make another choice.

more

Use more to view the contents of one or more files, one page at a time. For example:

% more /usr/share/dict/words

A
A's
AA's
AB's
ABM's
AC's
ACTH's
AI's
AIDS's
AM's
AOL
AOL's
ASCII's
ASL's
--More--(0%) 

--More--(0%)

/misspell

...skipping
missive's
missives
misspell
misspelled
misspelling
misspelling's
misspellings
misspells
misspelt
misspend
misspending
--More--(62%)    

lpr

Although many of the applications that you will be using in the computer lab will have their own printing facilities which will be of the type that is likely familiar to you, is also possible to print files from the command line with the lpr command. If no options are passed to lpr, files are sent to the system-default printer, or to the printer specified by your PRINTER environment variable, if it is set. Typical usage is

% lpr file_to_be_printed

% lpr -o sides=one-sided file_to_be_printed

cd and pwd

Use cd and pwd to change (set) and print, respectively, your working directory. We have already seen examples of these commands above. Here's a summary of typical usages (again note the use of semi-colons to separate distinct Unix commands issued on the same line):

% cd
% pwd
/home/choptuik

% cd ~; pwd
/home/choptuik

% cd /tmp; pwd
/tmp

% cd ~phys210; pwd
/home/phys210

% cd ..; pwd
/home

% cd phys210; pwd
/home/phys210

% cd ..

ls

Use ls to list the contents of one or more directories. On Linux systems, I advocate the use of the alias

% alias ls='ls --color=auto -FC'

1. Append special characters (notably * for executables, / for directories and @ for symbolic links) to the names of certain files (the -F option),
2. List in columns (the -C option).
3. Color-code the output, again according to the type of the file.
   

% cd ~phys210t
% ls
cmd*  dir1/  dir2/
%

% cd ~phys210t; ls -a 
./   .aliases       .bashrc  dir1/  .profile  .Xauthority
../  .bash_history  cmd*     dir2/  .viminfo

% cd ~phys210t; ls -l
-rwxr-xr-x 1 phys210t public   39 Aug 23  2012 cmd*
drwxr-xr-x 4 phys210t public 4096 Aug 23  2012 dir1/
drwxr-xr-x 2 phys210t public 4096 Aug 23  2012 dir2/

• first character: file type: - for regular file, d for directory.
• next nine characters: 3 groups of 3 characters each specifying read (r), write (w), and execute (x) permissions for the user (owner of the file), user's in the owner's group and all other users. A - in a permission field indicates that the particular permission is denied.

If any of the arguments to ls is a directory, then the contents of that directory are listed. Finally, note that the -R option will recursively list sub-directories:

% cd ~phys210t; pwd
/home/phys210t

% ls -R
.:
cmd*  dir1/  dir2/

./dir1:
file_1  subdir1/  subdir2/

./dir1/subdir1:
file_s

./dir1/subdir2:
file_3

./dir2:
file_4

% cd /
% ls -R

mkdir

Use mkdir to make (create) one or more directories. Sample usage:

% cd ~
% mkdir tempdir
% cd tempdir; pwd
/home/choptuik/tempdir

% cd ~
% mkdir -p a/long/way/down
% cd a/long/way/down; pwd
/home/choptuik/a/long/way/down

/home/choptuik/a   /home/choptuik/a/long    /home/choptuik/a/long/way

/home/choptuik/a/long/way/down

cp

Use cp to (1) make a copy of a file, (2) copy one or more files to a directory, or (3) duplicate an entire directory structure. The simplest usage is the first, as in:

% cp foo bar

% cp foo bar /tmp

/tmp/foo   /tmp/bar

% cd ~phys210t; ls -a
./   .aliases       .bashrc  dir1/  .profile  .Xauthority
../  .bash_history  cmd*     dir2/  .viminfo

% mkdir -p /tmp/choptuik

% cp -r ~phys210t /tmp/choptuik
cp: cannot open `phys210t/.bash_history' for reading: Permission denied
cp: cannot open `phys210t/.viminfo' for reading: Permission denied
cp: cannot open `phys210t/.Xauthority' for reading: Permission denied

% cd /tmp/choptuik/phys210t; ls -a
./  ../  .aliases  .bashrc  cmd*  dir1/  dir2/  .profile

% cp -r ~phys210t /tmp

mv

Use mv to rename files, or to move files from one directory to another. Again, I assume that mv is aliased to mv -i so that you will be prompted if an existing file will be clobbered by the command. Here's a "rename" example

% ls
thisfile

% mv thisfile thatfile
% ls 
thatfile

% pwd 
/tmp/lev1
% ls
lev2/

% cd lev2
% ls
file1 file2 file3 file4

% mv file1 file2 file3 ..
% ls
file4

% cd ..
% ls
file1 file2 file3 lev2/

rm

Use rm to remove (delete) files or directory hierarchies. The use of the alias rm -i for cautious removal is highly recommended. Once you've removed a file in Unix there is essentially nothing you can do to (easily) recover it other than restoring a copy from backup media  (assuming the system is regularly backed up), and if the file was created since the last backup, you're really out of luck! Examples include:

% rm thisfile

% rm file1 file2 file3 

% rm -r thisdir

% rm thisdir

chmod

Use chmod to change the permissions on a file. See the discussion of ls above for a brief introduction to file-permissions and check the man pages for ls and chmod for additional information. Basically, file permissions control who can do what with your files. Who includes yourself (the user u), users in your group (g) and the rest of the world (the others o). What includes reading (r), writing (w, which itself includes removing/renaming) and executing (x, and recall that execution permission on a directory is required in order to cd to it).  When you create a new file, the system sets the permissions (or mode) of a file to default values that you can modify using the umask command. (See the discussion of umask in the man page for bash for more information).

On the lab machines your defaults should be such that you can do anything you want to a file you've created, while the rest of the world (including fellow group members) normally has read and, where appropriate, execute permission. As the man page will tell you, you can either specify permissions in numeric (octal) form or symbolically. I prefer the latter. Some examples that should be useful to you include:

% chmod go-rwx file_or_directory_to_hide

% chmod a+x executable_file

% chmod a-w file_to_write_protect

% chmod a=r file_for_all_to_read

scp

Use scp (whose syntax is an extension of cp) to copy files or hierarchies from one Unix system to another. scp is part of the ssh distribution, so will prompt you for a password for access to the remote account.

For example, assume I am logged into hyper and that I want to copy my ~/.bashrc file to a file named ~/.bashrc-hyper in my home directory on matt@bh0.phas.ubc.ca.  The following will do the trick

hyper% scp ~/.bashrc matt@bh0.phas.ubc.ca:~/.bashrc-hyper
matt@bh0.phas.ubc.ca's password:
.bashrc                                       100% 1813     1.8KB/s   00:00

hyper% scp -q ~/.bashrc matt@bh0.phas.ubc.ca:~/.bashrc-hyper

hyper% scp -q matt@bh0.phas.ubc.ca:~/.aliases ~/.aliases-bh0

WARNING!! Be very careful using scp, particularly since there is no -i (cautious) option to warn you if existing files will be overwritten (there actually is a -i option, but it serves a completely different purpose!). Also note that there is a -r option for remote-copying entire hierarchies.

MORE ABOUT bash

Local Variables: bash allows for the definition of variables, which are used to store various pieces of information in the form of text strings. Indeed this basic notion of "variable" should be familiar to you if you have experience programming in a language such as C, Maple, Mathematica, python, MATLAB etc. Further, bash distinguishes between two types of variables: local variables, whose values are available only in the current shell, and environment or global variables, whose values are inherited (accessible) by processes (including other shells) that are started within the current shell.

The syntax for defining a new local variable (or for changing its value) is simple:

% varname=value

% var1=val1
% echo $var1
val1

% var1='val1'
% echo $var1
val1

% var2='val1 val2 val3'
% echo $var2
val1 val2 val3

Environment Variables: As mentioned above, bash uses another type of variable---called an environment variable---which is often used for communication between the shell and other processes (possibly another shell, which does not necessarily have to be bash).  To see a list of all currently defined environment variables, use the env command:

% env
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=dd7960f4980905b6f1adf1bb4a82e093-1345246608.891336-1731802610
SSH_CLIENT=142.103.234.164 51727 22
SSH_TTY=/dev/pts/0
HISTFILESIZE=1000
USER=phys210t
MAIL=/var/mail/phys210t
PATH=.:/home/phys210t/bin:/home/phys210/bin:/etc:/usr/etc:/usr/ucb:/bin:/usr/bin:/usr/local/bin:
         . 
         .
         .

% printenv
TERM=xterm
SHELL=/bin/bash
XDG_SESSION_COOKIE=dd7960f4980905b6f1adf1bb4a82e093-1345246608.891336-1731802610
SSH_CLIENT=142.103.234.164 51727 22
SSH_TTY=/dev/pts/0
HISTFILESIZE=1000
USER=phys210t
MAIL=/var/mail/phys210t
PATH=.:/home/phys210t/bin:/home/phys210/bin:/opt/maple16/bin:/usr/local/sbin ...
         . 
         .
         .

% printenv PWD
/home/phys210t

% printenv HOME PATH
/home/phys210t
.:/home/phys210t/bin:/home/phys210/bin:/opt/maple16/bin:/usr/local/sbin ...

% echo $LOGNAME
phys210t

% echo $SHELL $USER
/bin/bash phys210t

% export MYENV='value'
% printenv MYENV
value

% MYENV='newvalue'; printenv MYENV 
newvalue

Note that a key aspect of the global nature of environment variables is that they (and their values) are "inherited" by any shell that is executed within a running bash (which includes the shells that are always started whenever a bash script is executed). Thus, if after having executed the assignments above, I now start a new shell, the environment variable MYENV will be defined with its expected value:

% bash
A new shell starts ...
% printenv MYENV
newvalue

Following is a list of a few of the environment variables that are generally predefined and/or redefined as necessary by bash:

• PATH: Stores the current list of directories that are searched for commands (executables).
  
• HOME: Stores the user's home directory;

  % cd $HOME/dir1
  

  is equivalent to

  % cd ~/dir1
  

• PWD: Stores the current working directory.
• USER: Stores the name of the user account (login name)
• LOGNAME: Same as USER, but more generically available, so preferred in scripts.

Using bash Pattern Matching: bash provides facilities that allow you to concisely refer to one or more files whose names match a given pattern. The process of translating patterns to actual filenames is known as filename expansion or globbing. Patterns are constructed using plain text strings and the following constructs, known as wildcards

 
?       Matches any single character

*       Matches any string of characters (including no 
        characters)

[a-z]   (Example) Matches any single character contained 
        in the specified range (the match set)---unfortunately 
        (from the viewpoint of a Unix neophyte), the precise characters 
        that comprise the match set can vary from system to system, 
        due to the inherent ambiguity in how to order upper 
        and lower case letters (see note below).

[^a-z]  (Example) Matches any single character 
        not contained in the specified range

[02468]

% ls ??

% cp a* /tmp

% mv *.f ../newdir

% mv *.f *.for 

% mv *.f 

1. a, b, .... y, z, A, B, ... Y, Z
2. A, B, ..., Y, Z, a, b, ..., y, z
3. a, A, b, B, ..., y, Y, z, Z
4. A, a, B, b, ..., Y, y, Z, z

[a-c]

[aBbCc]

[a-c]

[abc]

lab% ls [A-c]*

office% ls [A-Ca-c]*

bash maintains a numbered history of previously entered command lines.  Type

% history

1. Type an exclamation mark, followed by the number of the command in the history list.
   
2. Type an exclamation mark, followed by a string of characters to reexecute the most recently typed command that begins with that string.

Examples:

% !20

reexecutes the 20th command in the history.

% !c

reexecutes the most recently issued command that starts with a 'c'.

Standard Input, Standard Output and Standard Error: A typical Unix command (process, program, job, task, application) reads some input, performs some operations on, or depending on, the input, then produces some output. It proves to be extremely powerful to be able to write programs (including bash scripts) that read and write their input and output from "standard" locations. Thus, Unix defines the notions of

• standard input: default source of input
• standard output: default destination of output
• standard error: default destination for error messages and diagnostics

% cat 
foo
foo
bar
bar
^D

Input and Output Redirection: The power and flexibility of the stdin/stdout mechanism becomes apparent when we consider the operations of input and output redirection that are implemented in bash (and many other shells). As the name suggests, redirection means that stdin and/or stdout are associated with some source/sink other than the terminal.

Input Redirection is accomplished using the < (less-than) character, followed by the name of a file from which the input is to be extracted. Thus the command-line

% cat < input_to_cat

% cat input_to_cat

Output Redirection is accomplished using the > (greater than) character, again followed by the name of a file into which the (standard) output of the command is to be directed. Thus

% cat > output_from_cat

set -o noclobber

% cat > output_from_cat
output_from_cat: File exists

The standard output from a command can also be appended to a file using the two-character sequence >> (no intervening spaces). Thus

% cat >> existing_file

From time to time it is convenient to be able to "throw away" the standard output of a command. Unix systems have a special file called /dev/null that is ideally suited for this purpose. Output redirection to this file, as in:

verbose_command > /dev/null

Pipes: Part of the "Unix programming philosophy" is to keep input and output to and from commands in "machine-readable" form: this usually means keeping the input and output simple, structured and devoid of extraneous information which, while informative to humans, is likely to be a nuisance for other programs. Thus, rather than writing a command that produces output such as:

% pgm_wrong
Time = 0.0 seconds  Force = 6.0 Newtons
Time = 1.0 seconds  Force = 6.1 Newtons
Time = 2.0 seconds  Force = 6.2 Newtons

% pgm_right
0.0   6.0
1.0   6.1
2.0   6.2

% ls -1 | wc
10     10     82

Pipelines can be made as long as desired, and once you know a few Unix commands, you can easily accomplish some fairly sophisticated tasks by interactively building up multi-stage pipelines. Note that it is often useful to build up a pipeline stage by stage, and to do this the command recall mechanism provided via the up-arrow key is helpful, as are the history commands discussed above.

Regular Expressions and grep: Regular expressions may be formally defined as those character strings that are recognized (accepted) by finite state automata. If you haven't studied automata theory, this definition won't be of much use, so for our purposes we will define regular expressions as specifications for rather general patterns that we will wish to detect, usually in the contents of files. Although there are similarities in the Unix specification of regular expressions to bash wildcards (see above), there are important differences as well, so be careful. We begin with regular expressions that match a single character:

a         (Example) Matches 'a', any character other than 
          the special characters: . * [ ] \ ^ or $ may be 
          used as is

\*        (Example) Matches the single character '*'.  
          Note that `\' is the "backslash" character.  A 
          backslash may be used to "escape" any of the 
          special characters listed above
          (including backslash itself)

.         (Period/dot) Matches ANY single character.

[abc]     (Example) Matches any one of 'a', 'b' or 'c'.

[^abc]    (Example) Matches any character that ISN'T an 
          'a', 'b' or 'c'.

[a-z]     (Example) Matches any character in the inclusive 
          range 'a' through 'z'.

[^a-z]    (Example) Matches any character NOT in the 
          inclusive range 'a' through 'z'.

^         Matches the beginning of a line.

$         Matches the end of a line.

ahgfh     (Example) Matches the string 'ahgfh'.  Any string 
          of specific characters (including escaped special 
          characters) may be specified in this fashion.

a*        (Example) Matches zero or more occurrences of the 
          character 'a'.  Any single character expression 
          (except start and end of line) followed by a '*' will 
          match zero or more occurrences of that particular 
          sequence.

.*        Matches an arbitrary string of characters.

Note that, in contrast to the bash globbing mechanism described above, ranges involving lower case and upper case characters, respectively, are always distinct. Thus

[a-z]

matches a single lower case letter only, while

[A-Z]

similarly matches a single upper case letter only (in particular, then, the meaning of a character range in the context of regular expressions does not depend on the collating sequence defined by the locale).

All of this is may be a bit confusing, so it is best to consider the use of regular expressions in the context of the Unix grep command.

grep

grep (which loosely stands for (g)lobal search for (r)egular (e)xpression with (p)rint) has the following general syntax:

   grep [options] regular_expression [file1 file2 ...]

% grep the

% grep the *

Some of the options to grep are worth mentioning here. The first is -i which tells grep to ignore case when pattern-matching. Thus

% grep -i the text

% grep -v the text

% grep -in the text

Here are a few slightly more complicated examples. Note that when supplying a regular expression that contains characters such as '*', '?', '[', '!' ..., that are special to the shell, the regular expression should be surrounded by single quotes to prevent shell interpretation of the shell characters. In fact, you won't go wrong by always enclosing the regular expression in single quotes.

% grep '^.....$' file1

% grep 'a' file1 | grep 'b'

% grep -v '^#' input > output

Pattern matching (searching for strings) using regular expressions is a powerful concept, but one that can be made even more useful with certain extensions. Many of these extensions are implemented in a relative of grep known as egrep. See the man page for egrep if you are interested.

Using Quotes (' ', " ", and ` `): Most shells, including bash, use the three different types of quotes found on a standard keyboard

    ' ' ->  Known as forward quotes, single quotes, quotes 
    " " ->  Known as double quotes
    ` ` ->  Known as backward quotes, back-quotes

Forward quotes: ' ' We have already encountered several examples of the use of forward quotes that inhibit shell evaluation of any and all special characters and/or constructs. Here's an example:

% a=100
% echo $a
100

% b=$a
% echo $b
100

% b='$a'
% echo $b
$a

Double quotes: " " Double quotes function in much the same way as forward quotes, except that the shell "looks inside" them and evaluates (1) any references to the values of shell variables, and (2) anything within back-quotes (see below). Example:

% a=100
% echo $a
100

% string="The value of a is $a"
% echo $string
The value of a is 100

Backward quotes: ` ` The shell uses back-quotes to provide a powerful mechanism for capturing the standard output of a Unix command (or, more generally, a sequence of Unix commands) as a string that can then be assigned to a shell variable or used as an argument to another command. Specifically, when the shell encounters a string enclosed in back-quotes, it attempts to evaluate the string as a Unix command, precisely as if the string had been entered at a shell prompt, and returns the standard output of the command as a string. In effect, the output of the command is substituted for the string and the enclosing back-quotes. Here are a few simple examples:

% date
Thu Aug 22 15:03:49 PDT 2013

% thedate=`date`
% echo $thedate
Thu Aug 22 15:04:00 PDT 2013

% which true
/bin/true

% file `which true`
/bin/true: ELF 64-bit LSB executable, x86-64, ...

% file `which true` `which false`
/bin/true:  ELF 64-bit LSB executable, x86-64, ...
/bin/false: ELF 64-bit LSB executable, x86-64, ...

Finally, here's an example illustrating that back-quote substitution is enabled for strings within double quotes, but disabled for strings within single quotes:

% var1="The current date is `date`"
% echo $var1
The current date is Thu Aug 22 15:04:43 PDT 2013

% var2='The current date is `date`'
% echo $var2
The current date is `date`

Job Control: Unix is a multi-tasking operating system: at any given time, the system is effectively running many distinct processes (commands) simultaneously (of course, if the machine only has one CPU (core), only one process can run at a specific time, so this simultaneity is often somewhat of an illusion). Even within a single shell, it is possible to run several different commands at the same time. Job control refers to the shell facilities for managing how these different processes are run. It should be noted that job control is arguably less important in the current age of windowing systems than it used to be, since one can now simply use multiple shell windows to manage several concurrently running tasks.

Commands issued from the command-line normally run in the foreground. This generally means that the command "takes over" standard input and standard output (the terminal), and, in particular, the command must complete before you can type additional commands to the shell. If, however, the command line is terminated with an ampersand: &, the job is run in the background and you can immediately type new commands while the command executes. Example:

% grep the huge_file > grep_output &
[1] 1299

[1]+    Done   grep the huge_file > grep_output

% grep the huge_file > grep_output 
^Z  
[1}+ Stopped    grep the huge_file > grep_output 
% bg
[1]+ grep the huge_file > grep_output &

% jobs
[1] + Stopped    grep the huge_file > grep_output
[2]   Running    other_command

% fg %1

% kill %1
[1]  Terminated    grep the huge_file > grep_output

On many Unix systems, including Linux, there is a killall command, which allows you to kill processes by name. Finally, the shell will complain if you try to logout or exit the shell when one or more jobs are stopped. Either explicitly kill the jobs (or let them finish up if that's appropriate) or type logout or exit again to ignore the warning, kill all stopped jobs, and exit.

Another useful, though Linux-specific, command is pstree, which shows processes currently running on the host machine in the form of a tree. If you want to limit the output to your own processes (and not, for example, root's), use

% pstree -u your_userid

BASIC SHELL PROGRAMMING

1. Scripts can be used to customize or extend Unix commands in a more powerful and robust fashion than the aliasing mechanism discussed above.
2. Scripts can be used to automate sequences of Unix commands, with the possibility of changing one or more of the arguments to one or more of the commands. If you find yourself frequently typing the same sequence of commands, it takes very little time to create a script to accomplish the task, after which the execution of a single command does the trick. This has the added bonus that the script per se provides documentation for the job you are doing.
3. Many tasks that are cumbersome to perform in the context of a general purpose programming language, such as Java, C or Fortran, are easy to accomplish using a script. This particularly applies to issues involving file and directory manipulation, or the processing of output from a number of programs.

Time constraints preclude anything but a basic overview of bash programming; if you wish to become a wizard of this particular craft, you might want to consult the classic text, The UNIX Programming Environment, by Kernighan and Pike, cited in the following as reference [1]. In addition, there is plenty of information to be found about the subject online (see the representative links at the end of this document, for example. which are also available via the Course Resources page). Finally, should you find yourself in need of complex scripts, you may wish to consider learning/using perl, which is an extremely powerful scripting language that has become very popular in the Unix community over the past couple of decades (the python language is another good choice in this regard).

We start with a very simple example. Consider the problem of "swapping" the names of two files, which arises more often in practice than one might expect, and which cannot be accomplished with a standard Unix command. Assuming that no file t exists in the working directory, the command sequence

% mv a t
% mv b a
% mv t b

#! /bin/bash

# Bare-bones script to swap names of two files

# Usage: swap file1 file2

mv $1  t
mv $2 $1
mv t  $2

#! /bin/bash

Continuing with our dissection of the script, lines that begin with a hash ("number sign") # (excluding the magic first line) such as

# Bare-bones script to swap names of two files

# Usage: swap file1 file2

The final three lines of the script

mv $1  t
mv $2 $1
mv t  $2

Having created a file called swap containing the above lines, I set execute permission on the file with the chmod command

% chmod a+x swap
% ls -l swap
-rwxr-xr-x 1 phys210 public 116 2009-09-07 16:32 swap*

% ls 
f1  f2  swap*
% cat f1
This is the first file.
% cat f2
This is the second file.
% swap f1 f2
% cat f1
This is the second file.
% cat f2
This is the first file.

When developing and debugging a shell program, it is often very useful to enable "tracing" of the script. This is done by adding the -x option to the header line:

#! /bin/bash -x

% swap f1 f2
+ mv f1 t
+ mv f2 f1
+ mv t f2

Although swap as coded above is reasonably functional, it is not very robust and can potentially generate undesired "side-effects" if used incorrectly. Observe, for example, what happens when the script is invoked without any arguments (tracing has now been disabled by removing the -x option in the header)

% swap
mv: missing file argument
Try `mv --help' for more information.
mv: missing file arguments
Try `mv --help' for more information.
mv: missing file argument
Try `mv --help' for more information.

% swap f1
mv: missing file argument
Try `mv --help' for more information.
mv: missing file argument
Try `mv --help' for more information.
% ls
f2  swap*  t

#! /bin/bash

# Improved version of script to swap names of two files

# Set shell variable 'P' to name of script
P=`basename $0`

# Set shell variable 't' to name of temporary file
t=.swap.tempfile.3141

# Usage function
usage () {
cat << END
usage: $P file1 file2
 
       Swaps filenames of file1 and file2
END
exit 1
}

# Function that is invoked if temporary file already exists
t_exists () {
cat << END
$P: Temporary file '$t' exists.  
$P: Remove it explicitly before executing this script.

/bin/rm -f $t
END
exit 1
}

# Function that checks that its (first) argument is an
# existing file
check_file () {
if [ ! -f $1 ]; then
   echo "$P: File '$1' does not exist"
   error="yes"
fi
}

# Argument parsing---script requires exactly 2 arguments
case $# in
2) file1=$1; file2=$2 ;;
*) usage;;
esac

# Check that the arguments refer to existing files
check_file $1
check_file $2

# Bail out if either or both arguments are invalid
test "X${error}" = X || exit 1

# Ensure that temporary file doesn't already exist
test -f $t && t_exists

# Do the swap
mv $file1 $t
mv $file2 $file1
mv $t     $file2

# Normal exit, return 'success' exit status
exit 0

Let us examine this new version of swap in detail.

As the comment indicates, the command

# Set shell variable 'P' to name of script
P=`basename $0`

We use basename here so that if someone invokes our script using its full path name, perhaps

% /home/phys210/shellpgm/ex2/swap f1 f2

The next assignment sets the shell variable t to the name of a temporary file that, under normal circumstances, should never exist in the directory in which swap is executed. This isn't the most bullet-proof of strategies, but it's better than using t itself for the name for the temporary file!

# Set shell variable 't' to name of temporary file
t=.swap.tempfile.3141

The next section of code defines a shell function, called usage, which can be invoked from anywhere in the script. When called, the function will print a message to standard output informing the user of the proper usage of the command, and then exit (stop execution of the script).

# Usage function
usage () {
cat << END
usage: $P file1 file2

       Swaps filenames of file1 and file2
END
exit 1
}

routinename () {
   command
   command
   ...
}

Within the usage routine appears the construct

cat << END
   ...
END

An interesting and useful feature of here-documents is that they are partially interpreted by the shell before being fed into their destination command. In particular, shell-variable-evaluations

$var

`command [arguments]`

usage: swap file1 file2

       Swaps filenames of file1 and file2

exit 1

All Unix commands return such codes (scripts that terminate without an explicit exit, implicitly return success to the invoking shell) and they can be used in the context of shell-control structures such as if, while and until statements.

The function t_exists is very similar in construction to usage, and is used in the unlikely event that a file named .swap.tempfile.3141 does exist in the directory in which the script is invoked.

# Function that is invoked if temporary file already exists
t_exists () {
cat << END
$P: Temporary file '$t' exists.
$P: Remove it explicitly before executing this script.

/bin/rm -f $t
END
exit 1
}

Function check_file illustrates the use of function arguments, as well as the shell if statement.

# Function that checks that its (first) argument is an
# existing file
check_file () {
if [ ! -f $1 ]; then
   echo "$P: File '$1' does not exist"
   error="yes"
fi
}

As with arguments to the script itself, function arguments are accessed positionally, via $1, $2, ... . Note, then, that the evaluation of $1, for example, depends crucially on context (or scope): within a function, $1 evaluates to the first argument to the routine, while outside of any function it evaluates to the first argument to the script.

For our purposes, a suitably general form of the shell if statement is

if command a; then
   commands 1
elif command b; then 
   commands 2
elif command c; then
   commands 3
   ...
else
   commands n
fi

The if statement in our check_file routine

if [ ! -f $1 ]; then
   echo "$P: File '$1' does not exist"
   error="yes"
fi

if test ! -f $1; then
   echo "$P: File '$1' does not exist"
   error="yes"
fi

In the current case, the -f $1 option returns true if the first argument to the routine is an existing regular file (i.e. not a directory or other type of special file). The ! is the negation operator, so the overall test command returns success (true) if the first argument is not an existing regular file.

The next section of code introduces the shell case statement:

# Argument parsing---script requires exactly 2 arguments
case $# in
2) file1=$1; file2=$2 ;;
*) usage;;
esac

A general case statement looks like

case word in 
pattern) commands ;;
pattern) commands ;;
...
esac

In our current example, we match on the built-in shell variable $#, which evaluates to the number of arguments that were supplied to the shell. The first set of actions

2) file1=$1; file2=$2 ;;

*) usage ;;

Using the check_file function, the script then ensures that each argument names an existing regular file:

# Check that the arguments refer to existing files
check_file $file1
check_file $file2

# Bail out if either or both arguments are invalid
test "X${error}" = X || exit 1

"X${error}" = X

command 1 || command 2

if [ ! command 1 ]; then
   command 2
fi

Similarly, the next piece of the script

# Ensure that temporary file doesn't already exist
test -f $t && t_exists

command 1 && command 2

if [ command 1 ]; then
   command 2
fi

Finally, if we've made it past all of the error-checking, it's time to actually swap the filenames, and have the script return a "success" exit status to the invoking environment:

# Do the swap
mv $file1 $t
mv $file2 $file1
mv $t     $file2

# Normal exit, return 'success' code
exit 0

We can now test our improved version of swap, exercising in particular all of the error-checking features that have been incorporated. Here again is a contents-listing of the directory containing the script:

% ls
f1  f2  swap*
% more f1 f2
::::::::::::::
f1
::::::::::::::
This is the first file.
::::::::::::::
f2
::::::::::::::
This is the second file.

We start with a no-argument invocation:

% swap
usage: swap file1 file2
 
       Swaps filenames of file1 and file2

% swap f1
usage: swap file1 file2
 
       Swaps filenames of file1 and file2

We now invoke swap in a "normal" fashion, and verify that it is working properly:

% swap f1 f2
% more f1 f2
::::::::::::::
f1
::::::::::::::
This is the second file.
::::::::::::::
f2
::::::::::::::
This is the first file.

Supplying swap with two arguments that are not names of files in the working directory results in appropriate error messages:

% swap a1 a2
swap: File 'a1' does not exist
swap: File 'a2' does not exist

% swap a1 f2
swap: File 'a1' does not exist

Finally, after (perversely) creating .swap.tempfile.3141 using the touch command (touch filename creates the (empty) file filename if it does not exist, and changes its time of last modification to the current time if it does),

% touch .swap.tempfile.3141

swap: Temporary file '.swap.tempfile.3141' exists.  
swap: Remove it explicitly before executing this script.

/bin/rm -f .swap.tempfile.3141

% /bin/rm -f .swap.tempfile.3141
% swap f1 f2

We will conclude our whirlwind tour of shell programming with a description of a few more control structures, some additional niceties concerning shell variable evaluation, and a glimpse at a command useful for writing scripts that "interact" with the user.

The shell provides three structures for looping. The first is a for loop:

for var in word list; do
   commands  
done

% cat for-example
#! /bin/bash

# Illustrates shell 'for' loop

for i in foo bar 'foo bar '; do
        echo "i -> $i"
done

% for-example
i -> foo
i -> bar
i -> foo bar

In many instances, a for loop in a script will loop over all of the arguments supplied to the script. The built-in shell variable $* evaluates to the argument list, so we can write

for i in $*; do
   commands
done

for i; do
   commands
done

In addition to for iterations, there are also while loops:

while command; do
   commands
done

until command; do
   commands
done

The following table summarizes some built-in shell variables that are particularly useful for script writing [1]:

Variable	Evaluates to
$#	number of arguments
$*	all arguments
$?	return value of last command
$$	process-id of the script

Also observe that, as intimated above in the discussion of environment variables, a bash script inherits all of the environment variables (such as $HOME, $PATH, ...) that have been set in the invoking shell.

As shown in the next table [1], we can also use some tricks in the evaluation of shell variables to make writing scripts a little easier at times:

Expression	Evaluates to
$var	value of var, nothing if var undefined
${var}	same as above; useful if alphanumerics follow variable name
${var-thing}	value of var if defined; otherwise thing; $var unchanged.
${var=thing}	value of var if defined; otherwise thing; if undefined $var set to thing
${var+thing}	thing if var defined; otherwise nothing
${var?message}	if defined, $var; otherwise print message and exit shell.

Finally, the read command can be used to interactively provide input to a script. Here's an example

% cat read-example
#! /bin/bash

echo "Hello there!  Please type in your name:"
read name
echo "Pleased to meet you, $name"

% read-example
Hello there!  Please type in your name
Matthew Choptuik
Pleased to meet you, Matthew Choptuik

References / Additional Information

1. Brian W. Kernighan and Rob Pike, The UNIX Programming Environment, Prentice Hall, 1984
2. Bash Guide for Beginners (Includes sections on writing scripts.)
3. Bash Reference Manual
4. Bash Scripting Tutorial
5. Linux Shell Scripting Tutorial: A Beginner's handbook
6. Advanced Bash-Scripting Guide (An extensive reference, with lots of examples, and which, despite the name, does not assume prior knowledge of scripting or programming.