------------------------------------------------------------------------- PHYS210 UNIX 2, 3 and 4 LABS SUMMARY o Homework directories, editing demos, startup Files, aliases o ssh, shell variables, bash pattern matching, history o Standard input and standard output, input/output redirection o Pipes, grep and regular expressions o All those quotes ------------------------------------------------------------------------- NOTE: As we work through these notes you will be typing various commands to the shell. You can expect the output that you see at times to vary somewhat from that in the notes (i.e. some of the output depends on which user executes the command, when it was executed etc.) However, you should always see output that is at least close/similar to that recorded below. In particular, if you mistype something you can expect to receive an error message---for example, if I mistype 'ls' I will see % lls bash: lls: command not found Above all, ask for help from the TA or instructor if anything doesn't appear to be working as advertised! ------------------------------------------------------------------------- 1) THE /phys210 DIRECTORY AND YOUR SUB-DIRECTORY WITHIN IT Login and locate your personal directory within the /phys210 directory % cd /phys210/your-login-name or % cd /phys210/$LOGNAME The latter command will work for everyone. NOTE! In the following, whenever you see something that you are to type that includes $LOGNAME, you can replace $LOGNAME with your login (account) name The instructor will be doing this demo as user 'phys210d' % pwd /phys210/phys210d Get a long listing of the directory % ls -l total 20 drwx------ 2 phys210d ugrad 4096 Aug 29 12:09 hw1/ drwx------ 2 phys210d ugrad 4096 Aug 29 2012 hw2/ drwx------ 2 phys210d ugrad 4096 Aug 29 2012 hw3/ drwx------ 2 phys210d ugrad 4096 Aug 29 2012 hw4/ drwxr-xr-x 2 phys210d ugrad 4096 Nov 8 2012 public_html/ (The output you see may be slightly different) Permissions on hw[1-4] have been set so that there can be NO access except by user (i.e. you). Please keep them that way. (We will discuss permissions in detail in a future lecture/lab). Change to the public_html directory, When *you* type pwd following execution of the cd command, you should see /phys210/$LOGNAME/public_html % cd public_html % pwd /phys210/phys210d/public_html Get a directory listing, should be a single file 'index.html' % ls index.html View the contents of the file: in your case you should see the text $LOGNAME % more index.html phys210d You will be replacing/modifying index.html in Problem 2 of Homework 1. Access /phys210/$LOGNAME/public_html/index.html using a browser as follows. Point the browser to http://laplace.physics.ubc.ca/Students/$LOGNAME e.g., in my case http://laplace.physics.ubc.ca/Students/phys210d (TYPING $LOGNAME LITERALLY WILL *NOT* WORK) Again, you should see the text $LOGNAME in the browser window. ------------------------------------------------------------------------- DEMOS i) Using gedit to create/edit file called 'example' in your homework directory - Will create/modify /phys210/$LOGNAME/example in two ways 1) By invoking gedit from the command line, when the working directory is /phys210/$LOGNAME 2) By invoking gedit from the file browser (Home Folder on Launcher) - Note gedit automatically creates a backup copy of the file when it is re-edited, with name example~ ii) Using composer (from seamonkey) to edit file in public_html - Ensure seamonkey app is on Launcher, or find from Dash - Open Composer session by clicking on third icon from left on bottom bar - Use Open, then locate and open /phys210/$LOGNAME/public_html/index.html - Modify text to read, e.g. ... My login is $LOGNAME ... save it. - Verify that changes have been made using browser: P210 Home Page -> Student Pages (in Course Links) -> Your Name - NOTE: Composer does NOT automatically make a backup, good practice to make one (using the cp command e.g.) before editing - Add another line ... The home page for this course is HERE ... and make the text 'HERE' link to http://laplace.physics.ubc.ca/210/ - Verify that the link works - ASK FOR HELP AS NECESSARY!! - Examine source using browser - Seamonkey (from app top bar): View -> Page Source - Firefox (from Unity top bar): Tools -> Web Developer -> Page Source - Chrome (from Unity top bar): Tools -> View Source - or, for all three of these browsers: Ctrl-U (^U) ------------------------------------------------------------------------- 2) INSTALLING THE DEFAULT 210 BASH STARTUP FILES Change to your home directory % cd Look for the following files .bash_profile .bash_login .profile .bashrc .aliases using % ls .bash_profile % ls .bash_login % ls .profile % ls .bashrc % ls .aliases BE CAREFUL TO TYPE THESE AND ALL OTHER COMMANDS CAREFULLY/CORRECTLY, ENSURING, IN PARTICULAR, THAT THE NAMES OF ALL FILES ARE SPELLED CORRECTLY For any that exist, and unless you're a Linux expert and you know what these files do, and you're happy with their contents, then make backup copies of them as follows, and as necessary, using one or more of the following cp commands % cp -i .bash_profile .bash_profile.O % cp -i .bash_login .bash_login.O % cp -i .profile .profile.O % cp -i .bashrc .bashrc.O % cp -i .aliases .aliases.O In the highly unlikely case that any of the .O files already exist, then back *them* up using, for example % mv -i .bashrc.O .bashrc.OO, and re-execute % cp -i .bashrc .bashrc.O Now we will copy the PHYS 210 default startup files from the phys210 account to your home directory. Don't confuse phys210's home, referred to as ~phys210 or /home/phys210 with the /phys210 directory discussed in 1) and in which you will do your homework! Again, ensure that you are in your home directory. When *you* type pwd following execution of cd, you should of course see /home/$LOGNAME % cd % pwd /home/phys210d Do the copying, AND BE CAUTIOUS; i.e. use the -i option to cp. If cp prompts you to confirm overwrite then you have NOT mv'ed the original file(s) per the above instructions, so go back to *** above and try mv'ing the file(s) again. % cp -i ~phys210/.bashrc . % cp -i ~phys210/.profile . % cp -i ~phys210/.aliases . Now start up a NEW TERMINAL WINDOW and display the aliases that are defined % alias # You should see output like the following alias CP='/bin/cp' alias LN='/bin/ln' alias MV='/bin/mv' alias RM='/bin/rm' alias a='alias' alias cp='cp -i' alias hi='history' alias ln='ln -i' alias ls='ls --color=auto -CF' alias mv='mv -i' alias rm='rm -i' Display your path: /home/$LOGNAME/path should appear in it, as well as '.' (the current directory) which means the shell will always look in the current directory for commands to execute % echo $PATH .:/home/phys210d/bin:/home/phys210/bin:/opt/maple16/bin: ... An alternative way to display the value of the PATH environment variable % printenv PATH .:/home/phys210d/bin:/home/phys210/bin:/opt/maple16/bin: ... ------------------------------------------------------------------------- 3) DEFINING ALIASES Add a new alias to your ~/.aliases file It's a bit inconvenient to have to type % cd /phys210/$LOGNAME/hw1 to get to your Homework 1 directory, so let's define an alias that will take you there with less typing First, cd to your home directory, and make a back up copy of your ~/.aliases file (making back ups before you start modifying important files is an extremely good habit to develop!) % cd % cp .aliases .aliases.O And verify that you now *do* have a backup, as well as the original % ls .aliases .aliases.O .aliases .aliases.O THIS IS A GOOD CHANCE TO START USING FOR FILE COMPLETION, WILL SAVE AMOUNT OF TYPING, TYPING ERRORS Now, open ~/.aliases (/home/$LOGNAME/.aliases) with your text editor of choice, and add the following four lines at the end of the file (if you're having trouble opening /home/$LOGNAME/.aliases, ask for help!) #----------------------------------------------------------------------- # Define an alias to cd to Homework 1 directory #----------------------------------------------------------------------- alias cdhw1="cd /phys210/$LOGNAME/hw1" Once you have (carefully!) entered these 4 lines, save the file---your editor may prompt you to confirm overwrite of the existing file /home/$LOGNAME/.aliases, but since you have a back ups, it's safe to do the overwriting. Note that due to the use of double quotes in the alias definition (we'll discuss this in a future lecture), you can use either $LOGNAME literally, or use your actual login (account) name. Now source ~/.aliases to "activate" the new alias ... % source ~/.aliases ... verify that 'cdhw1' *has* been defined as an alias ... % alias cdhw1 alias cdhw1='cd /phys210/phys210d/hw1' ... and try it out % cdhw1 % pwd /phys210/phys210d/hw1 IF YOUR NEW ALIAS DOESN'T WORK, TRY TO DEBUG WHAT YOU'VE DONE USING THE ABOVE INSTRUCTIONS, AND/OR ASK FOR HELP ------------------------------------------------------------------------- 4) EXERCISE! Add additional alias definitions to ~/.aliases to change to your 2nd through 4th homework directories, as well as to cd to /phys210/$LOGNAME/public_html. I leave it to you to come up with names for the 4 new aliases. When you're done, be sure to source ~/.aliases, or start a new terminal window, and check that your new definitions have been made and work. IMPORTANT!! DON'T FORGET TO MAKE A BACK UP OF ~/.aliases BEFORE YOU START MODIFYING IT. This may mean overwriting the old back up, but if everything so far has worked as advertised, that should be OK. ------------------------------------------------------------------------- 5) HOMEWORK 1 In principle, at this point you should be able to complete Problems 1 and 2, although you may have to do a little supplementary study via web resources, using the help facility or other documentation for your text editor of choice etc. ------------------------------------------------------------------------- 6) SSH Open a terminal window and type the following % ssh $LOGNAME@hyper.physics.ubc.ca where you can either use $LOGNAME literally, or replace it with your account name. If you have never connected to hyper via ssh (most of you won't have), you should see something like the following: The authenticity of host 'hyper.physics.ubc.ca (142.103.236.100)' can't be established. RSA key fingerprint is c2:eb:96:0f:17:b1:73:b5:86:3c:61:f8:b9:92:f0:85. Are you sure you want to continue connecting (yes/no)? This is a standard error message that ssh will emit whenever you try connecting to a new host--it's a safeguard whose details we will ignore for the time being, so answer yes to the question (you must type 'yes' in full). You will then be asked for your password---enter the same one that you use to access the lab machines. phys210d@hyper.physics.ubc.ca's password: Provided you input the correct password you should be presented with a shell prompt on hyper. Execute the following command to verify that you are indeed logged into hyper: % hostname hyper Once you have done this, type % exit This will close the connection to hyper, and control will be returned to your shell on your lab machine On hyper, you have the same home directory as on the lab machines, and have access to all of the files therein, as well as /phys210/$LOGNAME and all of the files there as well. As discussed in the class notes, ssh-ing to hyper will be most useful should you want to connect from, e.g., a shell running on your home machine. Also note that if you don't specify the login name in a ssh command, it is assumed that you want to login using $LOGNAME. The first usage is *necessary* should you wish to access an account on a remote machine that does not have the same name as the account from which you are issuing the ssh command. Finally, as again discussed in the notes, ssh can be used to execute one or more commands on the remote host, and then return immediately return control to the invoking shell. Try the following % ssh hyper 'hostname; pwd; date' The authenticity of host 'hyper (142.103.236.100)' can't be established. RSA key fingerprint is c2:eb:96:0f:17:b1:73:b5:86:3c:61:f8:b9:92:f0:85. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'hyper' (RSA) to the list of known hosts. phys210d@hyper's password: hyper /home/phys210d Thu Aug 29 12:21:13 PDT 2013 % Note that in the above I've used the short version, hyper, of the remote host name, rather than the "fully qualified" form hyper.physics.ubc.ca The is only possible when the two machines belong to the same domain---in this case physics.ubc.ca. Also note that you will again be presented with a warning message to which you must answer 'yes' in order to connect. We see the message the second time since we have used distinct names for the machine---hyper.physics.ubc.ca.ca and hyper, respectively---in the two invocations of ssh. ------------------------------------------------------------------------- 7) SHELL VARIABLES As the notes discuss, there are two type of variables in bash i) Local variables ii) Environment (global) variables ------------------------------------------------------------------------- 7a) Local variables ------------------------------------------------------------------------- Define a few local variables % a=100 % b=foo % c='ls -l' % d='This is a complete sentence.' and then display their values - remember to use the '$' prefix evaluation mechanism---the echo command takes an arbitrary number of arguments, evaluates them if necessary (as in the case of $ constructs) and then "echos" the values to the terminal % echo $a 100 % echo $b foo % echo $c ls -l % echo $d This is a complete sentence. % echo $a $b $c $d 100 foo ls -l This is a complete sentence. We can use the value (contents) of c as a command (ls -l) % $c drwxr-xr-x 2 phys210d public 4096 Aug 22 09:56 bin -rw-r--r-- 1 phys210d public 283302 Dec 1 2012 C:\nppdf32Log\debuglog.txt drwxr-xr-x 2 phys210d public 4096 Nov 13 2012 Desktop drwxr-xr-x 3 phys210d public 4096 Jul 26 12:34 Documents . . . drwxr-xr-x 2 phys210d public 4096 Aug 16 2012 Pictures drwxr-xr-x 2 phys210d public 4096 Aug 16 2012 Public drwxr-xr-x 2 phys210d public 4096 Sep 13 2012 redirect drwxr-xr-x 2 phys210d public 4096 Aug 16 2012 Templates drwxr-xr-x 2 phys210d public 4096 Aug 16 2012 Videos drwxr-xr-x 5 phys210d public 4096 Aug 22 09:52 work One particularly interesting/useful local variable, PS1 (P-S-one), defines the bash prompt - in these notes I'm pretending that my standard prompt is '% ', but in reality it's like yours phys210d@cord ~]$ Display the value of PS1 phys210d@cord ~]$ echo $PS1 [\u@\h \W]$ Unless you're a bash expert, this output will not be intelligible to you. A basic explanation is contained in ~/.bashrc: #----------------------------------------------------------------------- # Set the command prompt to include the hostname (\h), the username (\u), # and the last component (i.e. the basename) of the current working # directory (\W). See 'man bash' and search for 'PROMPTING' for more # information on configuring the prompt. # # If you want to restore the lab-default prompt which includes the *full* # path name, comment-out or delete the following line. #----------------------------------------------------------------------- PS1="[\u@\h \W]$ " For full details see the appropriate section of the online bash manual at http://www.gnu.org/software/bash/manual/bashref.html#Printing-a-Prompt Let's set the prompt to '% ' phys210d@cord ~]$ PS1='% ' % If you want to get it back to the default, source your ~/.bashrc % source ~/.bashrc [phys210d@cord ~]$ ------------------------------------------------------------------------- 2b) Environment (Global) variables ------------------------------------------------------------------------- This type of variable is a little more interesting. First, you can get a listing of all currently defined environment (hereafter abbreviated 'env') variables (hereafter using the bash 'env' command % env SHELL=/bin/bash TERM=xterm XDG_SESSION_COOKIE=8b161181e519273b51954b6c51c4857b-1377803075.566678-1799778937 SSH_CLIENT=142.103.234.164 34583 22 OLDPWD=/phys210/phys210d/hw1 SSH_TTY=/dev/pts/0 USER=phys210d HISTFILESIZE=1000 PATH=.:/home/phys210d/bin:/home/phys210/bin:/opt/maple16/bin: ... MAIL=/var/mail/phys210d PWD=/home/phys210d LANG=en_CA.UTF-8 HISTCONTROL=ignoredups HOME=/home/phys210d SHLVL=2 LANGUAGE=en_CA:en LOGNAME=phys210d SSH_CONNECTION=142.103.234.164 34583 142.103.243.203 22 DISPLAY=localhost:10.0 XDG_RUNTIME_DIR=/run/user/phys210d _=/usr/bin/env Note that, BY CONVENTION, env vars have all-upper-case names. Also note the env vars that we have already mentioned/used in the course, including PATH, HOME, USER and LOGNAME Display the value of one or more specific env vars in one of two ways % echo $HOME /home/phys210d % printenv HOME /home/phys210d Note how in the first case we use '$HOME' to retrieve the value stored in HOME, while in the second we simply specify the name of the variable. printenv can take multiple arguments % printenv HOME USER LOGNAME /home/phys210d phys210d phys210d Define some new env vars of our own % FOO='foo' % export FOO 'export' tells bash that FOO is to be an env var, and we can combine the two commands % export BAR='bar' Display the values of FOO and BAR % echo $FOO $BAR foo bar % printenv FOO BAR foo bar Note: The printenv form is preferable, since if we forget to export a var, or mess up the export command, echo will still evaluate the var, but printenv won't find it in the list of environment variables ... % LOL='lol' % export lol ... where the wrong variable has been exported. % echo $FOO $LOL foo lol % printenv FOO LOL foo Now comes the interesting part. First, take a look at the value of the 'shell level' env var % printenv SHLVL 1 indicating that this is a "1st-level" shell. Now start up a NEW bash in the original bash % bash % It's hard to tell that anything happened. But check the value of SHLVL % printenv SHLVL 2 so this is a "2nd-level" shell, which makes sense. Now, when we started the new, 2nd-level shell, it inherited all of the env vars (and their values) that were defined in the old shell. So check the values of FOO and BAR % printenv FOO BAR foo bar But the local vars that we defined were NOT inherited, so the following echo command echoes "nothing", i.e. an empty line % echo $a $b $c $d % Return to the old shell % exit exit % Again, hard to tell anything happened except that 'exit' is echoed, and SHLVL is back to 1 % printenv SHLVL 1 Should you want to learn how to write bash scripts and funcations (optional self-study) topic, you will become more familiar with the use of botht types of shell variables. ------------------------------------------------------------------------- 8) BASH PATTERN MATCHING Recall that pattern matching allows us to select filenames that match a specified pattern (naturally) Let's do our work in a directory that contains a LOT of files (2350 to be exact) cd ~phys210/words You can expect the following ls command to take a bit of time! It will also generate so much output that it will overflow the terminal's buffer, and you won't be able to scroll back [using the mouse roller button] to see the first part of the output) % ls A goldeneye prosiness abbas gonadotropic prostatodynia abettal gonococcus protectant ablation goosegirl protestation abortiveness gormaw protococcal . . . Glossotherium promenade zoomantist glume pronaos zoospore glycogenize propagator zygal gnathobasic propiolate zymologic goanna proprietress Godforsaken prorogate Note how ls lists the files in columns according to the collating sequence A, a, B, b, ... Y, y, Z, z as discussed in the notes. Now, let's select various sets of filenames using pattern matching Get a listing of all files whose names ... i) ... begin with z % ls z* zabtie zenithward zoa zoomantist zygal zarabanda zimentwater zonoplacental zoospore zymologic ii) ... have a z in them % ls *z* Aladdinize epitomize nonsympathizer torporize automatize Erethizontidae Nowroze tzaritza azotate femalize ozonoscope uncriticizing bacterize frizziness paronymize unminimized bedazzlingly glycogenize perfectionizement unmunicipalized . . . disequalize monzonitic semihydrobenzoinic zoomantist dizenment muscularize skeletonization zoospore electrocauterization Nazerini squeezing zygal electrolyzer nomadize subzygomatic zymologic emblazer nonanalyzable sucivilized iii) ... end with z % ls *z berkowitz bruzz untz iv) ... are three characters long % ls ??? ask bat icy jib Ker neo nix old Ona Ova ree she vex zoa v) ... begin and end with a vowel % ls [aAeEiIoOuU]*[aAeEiIoOuU] academite asthenia impartance oxyphile aceanthrene athlete impendence ozonoscope acetylsalicylate attune improve Uintatheriidae . . . arthriticine Idiogastra overturnable uranite aschaffite Ilpirra Ovula urubu Asperugo Imerina oxane usucaptable assassinate immotile oxyaphia vi) ... begin and end with a consonant, note consonant = not a vowel, and the caret/hat (^) reverses the sense of the match set % ls [^aAeEiIoOuU]*[^aAeEiIoOuU] babblishly Hippotigris radiator bacchanalianly histiocytic radiology backfall histotrophy raffishly . . . hexadecanoic quinquagenarian zoomantist hidalgoism quintillion zygal hierophantes rabbinistical zymologic Hippeastrum rachiometer vii) ... begin with an a end with a y and have 6 characters % ls a????y archly assify autecy EXERCISE: Try some of your own! Be prepared to show me or one of the TAs your handywork. Bash pattern matching can be used in conjunction with any command. For example, let's create a temporary directory, and copy all of the files that have the character sequence 'and' in their names to that directory Execute the following (again, be very careful with your typing) % mkdir -p /tmp/$LOGNAME/mywords % cp *and* /tmp/$LOGNAME/mywords The first command creates the new directory /tmp/$LOGNAME/mywords (note the use of the -p option to mkdir to ensure that /tmp/$LOGNAME first gets created if it doesn't already exist), the second does the copying. Provided that you have issued these commands correctly, you can cd to the temporary directory, and verify that the copying succeeded as follows % cd /tmp/$LOGNAME/words % ls androgenesis distilland Jutlandish ribbandry toyland Bandhor expand landdrost sandless triandrian Bulanda grandisonant Maorilander sleeveband wanderluster Candlemas handclap misunderstanding stylomandibular zarabanda cropland heterandry northland substandard Note that *and* will expand to many words (filenames) so that the last argument to the cp command MUST be a directory: it it weren't, as in the following, we would get an error message First return to the original words directory ... % cd ~phys210/words ... and issue a faulty 'cp' command ... % cp *and* /tmp/$LOGNAME/words not-a-dir cp: target 'not-a-dir' is not a directory EXERCISE: Create a subdirectory 'morewords' within /tmp/$LOGNAME/mywords and copy all files with names beginning with a vowel and ending with 'ing' to that directory. Ask for assistance as necessary. ------------------------------------------------------------------------- 9) HISTORY/EVENTS As mentioned in class, the key thing to remember here is that you can type history to see a numbered list of previously typed commands % history should generate quite a lot of output for you by this time, and we'll use that fact to advantage in our work with pipes below. ------------------------------------------------------------------------- 10) STANDARD INPUT and STANDARD OUTPUT Recall that the standard input (stdin) is the default source of input for many Unix/Linux commands, and that standard output (stdout) is the default destination of output for many commands. Both are normally "tied" to the terminal, as best illustrated by executing the cat command with no arguments: since cat reads lines from stdin and writes them to stdout, the net effect is that the command "echoes" every line you type until you terminate the command using ^D (control-D) or ^C (control-C) % cat isn't isn't this this fun fun ? ? ^D Note that I/you only typed four of the lines in the above. Much of the power and utility of the Unix/Linux command line derives from the stdin/stdout concept, plus the mechanisms of stdin/stdout redirection and piping (connecting stdout of one command to stdin of another). ------------------------------------------------------------------------- 11) OUTPUT REDIRECTION and OUTPUT APPEND REDIRECTION This is an important topic, so let's work through a few examples of the two types of output redirection OUTPUT REDIRECTION Go to your home directory, and make a directory called redirect, change to it and verify that you're there % cd % mkdir redirect % cd redirect % pwd /home/phys210d/redirect Now copy all of the (non-hidden) files from ~phys210/redirect to the current directory (i.e. ~/redirect), and get a listing to see what was copied. Note that the second argument to the cp command is '.' (dot). Again, this is shell shorthand for the current directory % cp ~phys210/redirect/* . % ls fireice night pools road snowy The contents of these files are all relatively short poems by Robert Frost. Look at snowy, for example, using the more command IMPORTANT! to control 'more', type - space to go forward a page - 'b' to go back a page - 'q' to quit as necessary. So long as all of the file will fit on the terminal window, such control will not be needed (i.e. there will be no 'paging' of the output). % more snowy Whose woods these are I think I know. His house is in the village, though; . . . The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, And miles to go before I sleep. Now generate a long listing of the directory, but redirect the output of the ls command to a file ls-output % ls -l > ls-output Check the contents of ls-output, using cat (for short files, cat and more are essentially equivalent) % cat ls-output total 20 -rw-r--r-- 1 phys210d public 250 Aug 29 12:43 fireice -rw-r--r-- 1 phys210d public 0 Aug 29 12:43 ls-output -rw-r--r-- 1 phys210d public 614 Aug 29 12:43 night -rw-r--r-- 1 phys210d public 539 Aug 29 12:43 pools -rw-r--r-- 1 phys210d public 752 Aug 29 12:43 road -rw-r--r-- 1 phys210d public 555 Aug 29 12:43 snowy Note how ls-output appears in the listing, but has 0 size at the time the listing is generated---i.e. bash creates (or overwrites if noclobber is NOT set) the file to which output is redirected BEFORE it executes the command that generates the output OUTPUT APPEND REDIRECTION Assemble the poems into a single file using cat and the output append redirection operator >> (no intervening whitespace between the two <'s) % cat fireice >> poems % cat night >> poems % cat pools >> poems % cat road >> poems % cat snowy >> poems View the collection of poems % more poems Some say the world will end in fire, Some say in ice. From what I've tasted of desire I hold with those who favor fire. . . . The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, And miles to go before I sleep. The above example is a bit contrived as the same end could be achieved as follows (note that we first remove poems, and recall that RM is an alias for the bare-bones Unix rm command that doesn't ask for explicit confirmation before removing the file. (forever!!) % RM poems % cat fireice night pools road snowy > poems % more poems Some say the world will end in fire, Some say in ice. From what I've tasted of desire I hold with those who favor fire. . . . The woods are lovely, dark and deep, But I have promises to keep, And miles to go before I sleep, And miles to go before I sleep. ------------------------------------------------------------------------- 12) INPUT REDIRECTION & PIPES Change to your home directory, create the directory numbers, change to numbers, and verify that you are there % cd; mkdir numbers; cd numbers; pwd /home/phys210d/numbers (Note that I have combined several commands on a single input line by separating them with semi-colons. You can type them one at at time, should you wish, and the effect will be precisely the same). Copy the file ~phys210/numbers/random100 to the working directory (~/numbers) and view the contents of your copy of random100. % cp ~phys210/numbers/random100 . % more random100 15 13 6 2 12 4 6 5 17 8 5 2 18 --More--(12%) random100 contains a list of 100 randomly generated numbers each in the range 1 to 20, with one number per line. Verify that there are indeed 100 numbers (lines) in the file by using the wc (word count) command with the -l (list number of lines only) option % wc -l random100 100 random100 As mentioned briefly in the Unix notes, sort is a powerful command that can be used to (surprise, surprise) sort lines from files or standard input according to various criteria. By default (i.e. if executed with NO arguments), sort will i) Read lines from standard input until an end-of-file (EOF) is encountered. Recall that you can generate an EOF at the terminal by typing ^D (control-D) ii) Sort the lines in alphabetical order, but using the collating sequence, A, a, B, b, C, c...Y, y, Z, z as discussed in the notes iii) Output the sorted lines to standard output Recall that any command that by default reads and writes to stdin and stdout respectively is known as a FILTER. In this exercise we first want to sort the numbers in random100 in ascending numerical order. To do this, we use the -n option (for numeric sort), and redirect the standard input from random100. By default sort outputs to standard output, so we will save the sorted values using output redirection. So let's do the sorting twice, the first with the output going to stdout, the second with output going to the file random100-sorted % sort -n < random100 1 2 2 2 2 . . . 19 20 20 20 20 20 Invoked in this fashion, there's too much information to be seen on the terminal screen. This presents an ideal opportunity to use a pipe, in this case connecting the output of the sort command to the input of more (which also reads from standard input if executed without a filename argument) % sort -n < random100 | more 1 2 2 2 2 2 2 2 3 3 3 3 4 4 4 --More-- So now let's execute the same sort command, but this time redirect the (standard) output to a file random100-sorted % sort -n < random100 > random100-sorted Sure enough random100-sorted contains the numbers in ascending order as was the case when we executed sort -n < random100 | more % more random100-sorted 1 2 2 2 2 2 2 2 3 3 3 3 4 4 4 --More--(11%) QUESTION: Why does more show the percentage of the random100-sorted file that has been displayed when it fills the screen, but does not show a percentage when 'sort -n < random100 | more' is executed? There's another Unix/Linux command, uniq, which like sort, is a filter and which collapses any number of consecutive lines in its input which are identical to a single instance of the line. Thus, we can use sort, uniq and a pipe to generate an ordered list of which integers are contained in random100: % sort -n < random100 | uniq 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 A quick scan indicates that random100 contains at least one instance of each integer between 1 and 20 inclusive, but we can make an even longer pipeline just to make sure! % sort -n < random100 | uniq | wc -l 20 We can get sort to sort in descending order using the -r option % sort -n -r < random100 | uniq 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 And, finally, note that the output from a PIPELINE (not just a single command) can be redirected to a file % sort -n < random100 | uniq | wc -l > nuniq % cat nuniq 20 IMPORTANT: if you haven't yet thought of this, note that in building up pipelines and other similar contructs (sequences that use input/output redirection), you can save a lot of typing by using the history mechanism (up/down arrows) to retrieve then modify previously issued commandlines. ------------------------------------------------------------------------- 13) GREP AND REGULAR EXPRESSIONS As mentioned in class, constructing non-trivial regular expressions to use in conjunction with grep constitutes a type of programming in its own right. You should thus expect to have to practice and experiment a bit to get the hang of it. The 3rd and 4th problems of the first homework should definitely provide some practice (and possibly some aggravation, so don't be afraid to ask for hints if you're having trouble!!) Let's first use grep with the file 'poems' that we generated by concatenating the 5 Robert Frost poems in ~/redirect. Just to ensure that everyone is working with the same file, execute the following % cd % mkdir grep % cd grep % cp ~phys210/grep/poems . When you've done this, you should be in the directory ~/grep and should see the following (with phys210d replaced with your login) when you generate a long listing for the the directory % ls -l -rw-r--r-- 1 phys210d public 2710 Aug 29 12:51 poems Now let's use grep to display all the lines in poems that ... i) ... contain the word 'the' (case sensitive) % grep the poems Some say the world will end in fire, I have been one acquainted with the night. I have outwalked the furthest city light. I have looked down the saddest city lane. . . . Whose woods these are I think I know. His house is in the village, though; Between the woods and frozen lake The darkest evening of the year. To ask if there's some mistake. The only other sound's the sweep How many lines are there that contain 'the'? % grep the poems | wc -l 30 And how many lines are there that contain 'the' (case insensitive), so includes The (and conceivably ThE, thE, etc.)? Here we use the -i option, for 'ignore case' % grep -i the poems | wc -l 33 ii) ... start with 'I ' (so that 'I' is in the first column of the line and is followed by at least one space). Here we use the ^ (caret) to "anchor" the search pattern to the beginning of the line, and since the search pattern includes whitespace, we'll need to enclose the regular expression (regexp) in forward quotes. In fact, just to be safe, we'll quote ALL the regexps in the subsequent examples (this is a good habit to develop) % grep '^I ' poems I hold with those who favor fire. I think I know enough of hate I have been one acquainted with the night. I have walked out in rain-and back in rain. I have outwalked the furthest city light. I have looked down the saddest city lane. I have passed by the watchman on his beat I have stood still and stopped the sound of feet I have been one acquainted with the night. I doubted if I should ever come back. I shall be telling this with a sigh I took the one less traveled by, iii) ... start with either 'I' or 'F' (whitespace not necessary after either letter in this case) % grep '^[IF]' poems From what I've tasted of desire I hold with those who favor fire. I think I know enough of hate Is also great I have been one acquainted with the night. I have walked out in rain-and back in rain. I have outwalked the furthest city light. I have looked down the saddest city lane. I have passed by the watchman on his beat I have stood still and stopped the sound of feet I have been one acquainted with the night. From snow that melted only yesterday. In leaves no step had trodden black. I doubted if I should ever come back. I shall be telling this with a sigh I took the one less traveled by, iv) ... start with any upper case letter in the inclusive range T through Z % grep '^[T-Z]' poems To know that for destruction ice When far away an interrupted cry These pools that, though in forests, still reflect The total sky almost without defect, Will like the flowers beside them, soon be gone, The trees that have it in their pent-up buds To darken nature and be summer woods To blot out and drink up and sweep away These flowery waters and these watery flowers Two roads diverged in a yellow wood, To where it bent in the undergrowth; Then took the other, as just as fair, Though as for that the passing there Yet knowing how way leads on to way, Two roads diverged in a wood, and I Whose woods these are I think I know. To watch his woods fill up with snow. To stop without a farmhouse near The darkest evening of the year. To ask if there's some mistake. The only other sound's the sweep The woods are lovely, dark and deep, v) ... do not start with an upper case letter or a space. This one is a bit tricky since we use ^ for two distinct purposes, one to anchor the search to the beginning of the line, the other to negate the sense of the range A-Z plus space, i.e. [^A-Z ] means 'not in the inclusive range A through Z nor a space. % grep '^[^A-Z ]' poems ... and there's no output, so true to form for this type of poetry, the first word of each line is capitalized (assuming every non-blank line starts in the first column). vi) ... contain only one or more spaces. This one is also a bit tricky. To ensure that there is at least one space the regular expression needs to have an explicit space. We then follow that with a single character range [ ] that can occur ZERO or more times, i.e. [ ]*. Finally, we need to anchor the search to both the beginning of the line using ^ and the end of the line using $ Also note that I deliberately added spaces to empty lines in ~phys210/grep/poems to make this example generate some matches. % grep '^ [ ]*$' poems You should see a lot of blank lines on the screen at this point. How many? Again, pipe the output into wc -l % grep '^ [ ]*$' poems | wc -l 24 OK, two more examples that may help you with the homework vii) ... contain both 'and' and 'the' (case insensitive). We can do this easily by piping two greps together % grep -i 'and' poems | grep -i 'the' I have stood still and stopped the sound of feet And further still at an unearthly height And like the flowers beside them, chill and shiver, These flowery waters and these watery flowers And having perhaps the better claim, And that has made all the difference. Between the woods and frozen lake The woods are lovely, dark and deep, Note carefully that the first grep in the pipeline is supplied with poems as a file argument, but that the second grep is reading from the standard output of the first one, and hence does not need, and should not be given, a file argument viii) ... that are exactly 30 characters long. Here you need to be careful to type precisely 30 periods (.) between the ^ and $ that match the beginning and end of a line, respectively. Recall that '.' matches ANY single character % grep '^..............................$' poems But if it had to perish twice, Somewhere ages and ages hence: = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Now let's use grep to look for things in our history and list of environment vars. Here, grep will be used in "filter mode", since history or env will supply the input to grep via a pipe. The output, will, of course, be user-dependent Look in the history for grep commands themselves % history | grep grep . . . 1132 mv grep numbers redirect words thedate Save 1227 cd; mkdir grep; cd grep; cp ~phys210/grep/poems . 1229 grep the poems 1230 grep the poems | wc -l 1231 grep -i the poems | wc -l 1232 grep '^I ' poems 1233 grep '^[IF]' poems 1234 grep '^[T-Z]' poems 1235 grep '^[^A-Z ]' poems 1236 grep '^ [ ]*$' poems 1237 grep '^ [ ]*$' poems | wc -l 1238 grep -i 'and' poems | grep -i 'the' 1239 grep '^..............................$' poems 1240 history | grep grep Look in the history for command-lines that begin with cd. Note the construction to match the command-line numbers, as well as the whitespace around the numbers % history | grep '^[ ]*[0-9]*[ ]*cd' . . . 1076 cd 1079 cd; mkdir redirect; cd redirect; pwd 1092 cd ../grep 1129 cd .. 1145 cd 1185 cd ~phys210/words 1197 cd; mkdir redirect; cd redirect; pwd 1213 cd; mkdir numbers; cd numbers; pwd 1227 cd; mkdir grep; cd grep; cp ~phys210/grep/poems . Look for environment vars whose names begin with 'SSH' % env | grep '^SSH' SSH_AGENT_PID=22855 SSH_AUTH_SOCK=/run/user/phys210d/keyring-1dD4BO/ssh Look for environment vars or values that contain $LOGNAME (note the in this example the shell evaluates $LOGNAME to your literal account name BEFORE executing the grep command) % env | grep $LOGNAME GPG_AGENT_INFO=/run/user/phys210d/keyring-1dD4BO/gpg:0:1 OLDPWD=/home/phys210d GNOME_KEYRING_CONTROL=/run/user/phys210d/keyring-1dD4BO USER=phys210d SSH_AUTH_SOCK=/run/user/phys210d/keyring-1dD4BO/ssh PATH=.:/home/phys210d/bin:/home/phys210/bin:/opt/maple16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games PWD=/home/phys210d/grep HOME=/home/phys210d LOGNAME=phys210d XDG_RUNTIME_DIR=/run/user/phys210d XAUTHORITY=/home/phys210d/.Xauthority ------------------------------------------------------------------------- 14) ALL THOSE QUOTES We'll keep this section brief since the various quoting mechanisms available, and the distinctions among them, are most relevant in the context of shell programming, which we will not be studying at this time. Thus, we'll simply work through the examples from the notes. FORWARD QUOTES Recall that these quotes inhibit shell evaluation of any and all special characters and/or constructs: % a=100 % echo $a 100 % b=$a % echo $b 100 % b='$a' % echo $b $a Note how the forward quotes stop the shell from replacing $a with its value prior to the assignment to b. DOUBLE QUOTES Recall that these work like single quotes, except that the shell "looks inside" them and evaluates i) any references to the values of shell variables ii) anything within back-quotes (see below) % a=100 % echo $a 100 % string="The value of a is $a" % echo $string The value of a is 100 You should now understand how/why if you defined the alias cdhw1 using % alias cdhw1="cd /phys210/$LOGNAME/hw1" that you see $LOGNAME replaced by your login name when you subsequently ask to see the definition of cdhw1. % alias cdhw1 alias cdhw1='cd /phys210/phys210d/hw1' (If you used your actual account name rather than $LOGNAME in the definition of cdhw1, cdhw2, etc., you might want to try modifying the definitions to use $LOGNAME). BACKWARD QUOTES Recall that these quotes (backticks) provide the useful and powerful mechanism that allows you to capture the output from a command (or, in general, a pipeline) as a string, which can then, for example, be assigned to a shell variable, or supplied as an argument to another command % date Thu Aug 29 12:57:18 PDT 2013 % thedate=`date` % echo $thedate Thu Aug 29 12:57:29 PDT 2013 % which true /bin/true % file `which true` /bin/true: ELF 64-bit LSB executable, x86-64 ... % file `which true` `which false` /bin/true: ELF 64-bit LSB executable, x86-64 .. /bin/false: ELF 64-bit LSB executable, x86-64 ... ----------------------------------------------------------------------------- We've covered a LOT of ground in the Unix/Linux notes and the lab activities, and if you can master the material in them you'll be well on your way to becoming a Unix/Linux command-line guru! -----------------------------------------------------------------------------