Shell Scripting Hamish Whittal Shell Scripting by Hamish Whittal Published 2005-01-25 22:36:19 Copyright © 2004 The Shuttleworth Foundation Unless otherwise expressly stated, all original material of whatever nature created by the contributors of the Learn Linux community, is licensed under the Creative Commons [http://creativecommons. org/] license Attribution-ShareAlike 2. 0 [http://creativecommons. org/licenses/by-sa/2. 0/] [http://creativecommons. org/licenses/by-sa/2. 0/]. What follows is a copy of the “human-readable summary” of this document. The Legal Code (full license) may be read here [http://creativecommons. org/licenses/by-sa/2. /legalcode/]. You are free: • • • to copy, distribute, display, and perform the work to make derivative works to make commercial use of the work Under the following conditions: Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. • • For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above.
This is a human-readable summary of the Legal Code (the full license) [http://creativecommons. org/licenses/by-sa/2. 0/legalcode/]. Table of Contents 1. Tour de Shell Scripting ………………………………………………………………… 1 Introduction to the Course structure ……………………………………………. 1 Adage 1. 0: …………………………………………………………………….. 1 What you will need ………………………………………………………….. 1 What is Shell Scripting …………………………………………………….. Introduction to info, man and the whatis database ………………………….. 4 info pages ……………………………………………………………………… 4 Man Pages …………………………………………………………………….. 9 The whatis Database ………………………………………………………. 12 Revising some Basic Commands ……………………………………………… 13 The who command ………………………………………………………… 14 The w Command …………………………………………………………… 6 The “date” command ……………………………………………………… 17 The ‘echo’ command ………………………………………………………. 20 File Commands …………………………………………………………….. 23 System Commands ………………………………………………………… 29 stdin, stdout, stderr ……………………………………………………………….. 33 stdin …………………………………………………………………………… 34 stdout ………………………………………………………………………….. 4 Using stdin and stdout simultaneously ………………………………… 37 Appending to a file ………………………………………………………… 37 stderr ………………………………………………………………………….. 38 stdout, stderr and using the ampersand (&) ………………………….. 40 Exercises: …………………………………………………………………….. 41 Unamed Pipes ………………………………………………………………. 41 2. The Shell ………………………………………………………………………………… 5 Introduction ………………………………………………………………………… 45 What is the login shell? ………………………………………………………….. 46 Exercises ……………………………………………………………………… 48 The job of the shell ……………………………………………………………….. 50 Command Interpreter ……………………………………………………… 50 Allows for variables to be set ……………………………………………. 1 I/O redirection ………………………………………………………………. 51 Pipelines ………………………………………………………………….. …. 51 Customising your environment …………………………………………. 51 Conclusion: ………………………………………………………………….. 51 3. Regular Expressions ………………………………………………………………….. 53 Introduction ………………………………………………………………………… 53 What are regular expressions? ………………………………………………… 53 The fullstop ………………………………………………………………….. 54 Let’s explore “sed” syntax ……………………………………………….. 55 Square brackets ( [ ] ), the caret ( ^ ) and the dollar ( $ ) …………. 56 vi Shell Scripting Using sed and pipes ……………………………………………………….. 59 The splat (asterisk) ( * ) ………………………………………………….. 61 The plus operator ( + ) ……………………………………………………. 3 Returning from detour to our discussion on curly braces … …….. 66 RE’s are greedy for matching patterns ………………………………… 68 Placeholders and word boundaries …………………………………….. 70 Word boundaries ( < and > ) – a formal explanation ………………. 72 The tr command …………………………………………………………………… 73 The cut command …………………………………………………………………. 75 First Example in stages: ………………………………………………….. 7 Second Example in stages: ………………………………………………. 78 Third example in stages ………………………………………………….. 79 Exercises: …………………………………………………………………….. 81 The paste command ………………………………………………………………. 81 The uniq command ……………………………………………………………….. 83 The Sort command ……………………………………………………………….. 84 The grep command ……………………………………………………………….. 8 grep, egrep and fgrep ……………………………………………………… 92 Exercises: …………………………………………………………………….. 93 Challenge sequence: ………………………………………………………. 93 4. Practically Shell Scripting ………………………………………………………….. 95 Section Techniques to use when writing, saving and executing Shell Scripts ……………………………………………………………………………….. 95 Detour: File Extension labels ……………………………………………. 6 Comments in scripts ………………………………………………………. 97 Variables ……………………………………………………………………… 98 Shebang or hashpling #! ………………………………………………… 100 Exit ………………………………………………………………………….. 102 Null and unset variables ………………………………………………… 102 Variable Expansion ………………………………………………………. 105 Environmental vs shell variables ……………………………………… 06 Arithmetic in the shell …………………………………………………… 108 Examples …………………………………………………………………… 110 Exercises: …………………………………………………………………… 112 5. Using Quotation marks in the Shell …………………………………………….. 115 Introduction ………………………………………………………………………. 115 Single Quotes or “ticks” ……………………………………………………….. 15 Exercises: …………………………………………………………………… 118 Double Quotes …………………………………………………………………… 118 Exercises ……………………………………………………………………. 121 Backticks ………………………………………………………………………….. 121 Exercises: …………………………………………………………………… 124 Shell Arithmetic’s with expr and back quotes …………………………….. 25 Another tip when using quotation marks ………………………………….. 126 6. So, you want an Argument? ………………………………………………………. 129 Introduction ………………………………………………………………………. 129 Positional Parameters 0 and 1 through 9 …………………………………… 130 vii Exercises: …………………………………………………………………… 132 Challenge sequence: …………………………………………………….. 132 Other arguments used with positional parameters ……………………….. 33 $# How many positional arguments have we got ? ………………. 133 $* – display all positional parameters ……………………………….. 133 Using the “shift” command – for more than 9 positional parameters …………………………………………………………………………………. 134 Exit status of the previous command ………………………………… 136 7. Where to From Here? ………………………………………………………………. 139 Making Decisions ……………………………………………………………….. 39 Testing for a true or false condition ………………………………….. 139 Different types of tests ………………………………………………….. 140 Logical Operators ……………………………………………………………….. 146 Exercises: …………………………………………………………………… 150 Conditions in the shell …………………………………………………………. 151 Using the “if” statement ………………………………………………… 151 The “if” “then” “else” statement ………………………………………. 52 The “elif” statement ……………………………………………………… 153 The “case” statement …………………………………………………….. 154 Exercises ……………………………………………………………………. 156 Challenge sequence: …………………………………………………….. 157 Debugging your scripts ………………………………………………………… 157 The NULL command ……………………………………………………. 158 The || and && commands ………………………………………………. 59 Exercises: …………………………………………………………………… 160 8. Loops …………………………………………………………………………………… 163 Introduction ………………………………………………………………….. ….. 163 The “for” loop …………………………………………………………….. 163 while and until loops …………………………………………………….. 170 getopts Using arguments and parameters ………………………………….. 78 Exercises: …………………………………………………………………… 181 9. User input to a script ……………………………………………………………….. 183 Introduction ………………………………………………………………………. 183 The read command ……………………………………………………………… 183 Presenting the output …………………………………………………………… 188 The echo command ………………………………………………………. 88 The printf command …………………………………………………….. 191 10. Additional Information …………………………………………………………… 197 The shell environmental variables pertaining to scripting …………….. 197 The Source command ………………………………………………………….. 199 Exercises: …………………………………………………………………… 202 the exec command ………………………………………………………………. 202 Other methods of executing a script or a series of commands …. 04 11. Positional parameters & variables re-visited ………………………………… 209 Introduction ………………………………………………………………………. 209 PARAM:-value ………………………………………………………………….. 210 PARAM:=value ………………………………………………………………….. 211 viii Shell Scripting ${param:+value} ………………………………………………………………… 212 ? ${variable%pattern} …………………………………………………………… 14 MAGIC%%r*a …………………………………………………………………… 215 variable#pattern ………………………………………………………………….. 216 variable:OFFSET:LENGTH ………………………………………………….. 217 #variable …………………………………………………………………………… 219 Re-assigning parameters with set ……………………………………………. 220 Explaining the default field separator field – IFS ………………………… 21 Setting variables as “readonly” ………………………………………………. 222 Exercises: …………………………………………………………………… 222 Challenge sequences: ……………………………………………………. 223 12. Bits and pieces – tying up the loose ends …………………………………….. 225 The eval command ……………………………………………………………… 225 Running commands in the background using & …………………………. 227 Traps and signals ………………………………………………………………… 29 Signals ………………………………………………………………………. 229 Traps ………………………………………………………………………… 232 Exercises: …………………………………………………………………… 236 File descriptors re-visited ……………………………………………………… 236 Exercises ……………………………………………………………………. 241 Here documents ………………………………………………………………….. 42 Exercises ……………………………………………………………………. 246 Functions ………………………………………………………………………….. 246 Exercises: …………………………………………………………………… 249 Challenge sequence ……………………………………………………… 249 A. Writing html pages …………………………………………………………………. 251 B. An introduction to dialog …………………………………………………………. 253 C.
A Comparisson of bash, tsch and ksh ………………………………………….. 255 Index ………………………………………………………………………………………. 257 List of Figures 1. 1. The structure of the info pages ……………………………………………………. 5 2. 1. Parent- and sub-shells ……………………………………………………………… 47 6. 1. Using Shift Command to access parameters ……………………………….. 134 10. 1. Parent- and sub-shells ………………………………………………………….. 200 x
This page intentionally left blank List of Tables 1. 1. Wildcards …………………………………………………………………………….. 23 1. 2. Standard Input, Standard Output and Standard Error ………………………. 34 xii This page intentionally left blank Chapter 1. Tour de Shell Scripting Introduction to the Course structure This course is structured with the following students in mind: • A “newbie” to shell scripting, OR • A user who wants a refresher on a particular aspect of shell scripting. For Category 1 users I suggest you work through each section, do all the relevant exercises and Labs.
If you fall into Category 2 above, then just look up the section you need a refresher on, do the examples below each section to drive the point home and move on. The Lab (project) is designed to get you into the groove of writing shell scripts and is, in essence, an example of the practical application of the shell script. You will learn an immense amount from doing the exercises and Labs, and you should have a great deal of fun too. In the Labs, we begin by building a really simple script and progress to a script that you will want to show your friends… but don’t.
Unless they’re Linux nuts too, they’ll think you’re just weird!!!! Finally, for those geeks out there, (or those of you who think you’re too clever for this sort of stuff), there are additional challenge sequences. For example, where the “wannabe-geeks” build a simple menu system, you must build a menu system with the “ncurses” library. This may mean reading up on the dialog package, figuring out how it works and then implementing it. I have included in the appendices explanations of the challenge sequences. Look out for the challenge sequences and really pull out all the stops! Adage 1. 0:
Oh, a final word of warning. If you haven’t noticed already, Unix and Linux people have a pretty wacky sense of humor. I’m no exception. If you find some of my jokes and quips in this course offensive, you’re definitely taking this whole Linux thing WAY TOO SERIOUSLY. Take a chill pill and re-read it and relax! What you will need 2 Tour de Shell Scripting • Pencil and Paper Yes. I know these two concepts are foreign to some of you, but hey, give an old codger like me a break. • To have logged onto your favorite Linux distribution as a user (with your username). We don’t at this stage need to be logged in as root.
At some time during this course you will need to log in as root. If you get to that point and are not the system administrator for the machine you are working on, then you may need to build your very own Linux machine. Any system administrator in their right mind would NEVER give you the root password. Of course, if you are the system administrator, you already have the root password! What is Shell Scripting Adage 1. 1: In order to learn to be a great system administrator, and “shell script-er”, you MUST be. . . . . LAZY. Yes, that’s right, LAZY. Say it, and again. Once more. Good!!! Why?
Because, if you are anything like me, you want to spend your time doing things you love, not the mundane boring stuff that will leave you feeling like you’ve really had to work hard! If you’re lazy, you will think of simpler, better, faster and more efficient ways of getting through your day’s tasks, look efficient, get many pats on the old’ back, or taps on the old’ head and leave work feeling like you’re just the smartest person around. Adage 1. 2 Next, if you REALLY want to learn to script, NEVER do manually, what you can do by using a script. Script everything!!!! So, let’s get scripting. But first, what is a shell? The shell, in UNIX and Linux is the equivalent of a command interpreter in Windows. Its job is to accept commands, as typed by the user on the command line, and interpret each of those commands, acting on it however necessary. The shell is a little like DOS operating system in many ways; the only difference being that it’s like DOS on steroids. I hope that over the remainder of this course you will you will understand this sentiment. For example typing: ls -l on the command line produces some output. How does UNIX know to call the ls command? How does it know to interpret the -l as a switch? What about the output?
How does the command output know to come to the screen? By chance? Nope. Nothing in Linux happens by chance! The shell does these things! What about a shell script? A shell script is in essence, a whole bunch of shell commands entered together in a file. A little like the DOS batch file, where many shell commands are grouped together to perform some function(s). What if we wanted to run two commands over and over again? Say, free and df -h One way of doing it would be to type the commands in over and over again. More work!!! Of course it is. We are looking at means of sticking to adage 1. 1, not so?
So, we could get clever and type both commands on a single line, separated by a semi-colon free;df -h 4 Tour de Shell Scripting We’ve reduced our finger-work, but not by much. Again the better way of doing this is to put both these commands into a file. For our example we will call this file mycmds. sh: riaan@debian:/tmp> vi mycmds. sh riaan@debian:/tmp> chmod +x mycmds. sh riaan@debian:/tmp> . /mycmds. sh total used free shared buffers Mem: 321628 317836 3792 0 14644 -/+ buffers/cache: 214656 106972 Swap: 506480 1060 505420 file system Size Used Avail Use% Mounted on /dev/hda1 5. 5G 3. 5G 2. G 63% / tmpfs 158M 4. 0K 158M 1% /dev/shm riaan@debian:/tmp> Then all we have to do it execute it, and voila , we have “created a new command”, aka mycmds. sh. Now each time, we merely need to run the script and the commands are executed together. Introduction to info, man and the whatis database This is a shell scripting course, but we’re going to start off by looking at the info pages, the man pages and the whatis database before we start scripting. This is a good idea because at least we know that we’re all on the same page. So, what is this man page, and info page, and that other stuff you mentioned?
Man pages is a term used as a short-hand for manual pages – or the manual. Info pages, are like manual pages (man), but are a newer format that the movers and shakers are trying to adopt. Which to use? Well both actually. In some cases, man does not contain all the necessary information, and then one needs to refer to the info pages. Sometimes it may be far easier to locate that which you are looking for by firing up the manual page. First we’ll tackle info pages. info pages The diagram below illustrates the structure of the info pages. Refer to it while reading this section cached 88536 5 Figure 1. 1.
The structure of the info pages Info pages are like man pages in many ways, except they provide a lot more information than man pages (well mostly anyway). Info pages are available on all the Linux distributions currently available, and they are similar to the man pages, although in some instances give you more information. If we look at a typical way of invoking info, it would be the word info, followed by a space, followed by the name of the command that you want to invoke info on. For example, for the command ls: info ls —^- 6 Tour de Shell Scripting Type the commands just as you see them here.
I have placed minus signs under the command and it’s arguments, and a caret (^) under the space. This is to illustrate that the command should be typed EXACTLY as is. This should give us an information page on the ls command. We could invoke info on a whole range of utilities by typing: info coreutils —^—— where coreutils is just another group of info pages. Coreutils is a grouping of info pages, containing commands such as ls, cd, cp, mv or list directory contents (ls), change directory (cd), copy files and directories (cp) and move or rename files (mv). One could, for instance, type: info mv
The way info is structured, is that when you first start it, it will display the node at which you are beginning your search. In our coreutils example, on the top of the page (right at the top) the first line looks as follows: File: coreutils. info, Node: Top, Next: Introduction, Up: (dir) Starting on the left-hand side, we see the file that we’re “info’ing” the coreutils. info file. The filename that contains information about the ls, cp, mv and cd commands amongst others is coreutils. info. The current Node is Top, which indicates that we’re at the top of the coreutils info page and cannot go higher within this group of pages.
From the top level of the info page on coreutils we can now do a couple of things: We can go to the next node (by typing ‘n’), which is the next topic, called Introduction. You will notice that there is no link to a Previous topic, since we are at the top of this group of pages. 7 We could scroll down to the menu and select a particular topic (node) we want displayed. File: info. info, Node: Top, Next: Getting Started, Up: (dir) Info: An Introduction ********************* The GNU Project distributes most of its on-line manuals in the “Info format”, which you read using an “Info reader”.
You are probably using an Info reader to read this now. If you are new to the Info reader and want to learn how to use it, type the command `h’ now. It brings you to a programmed instruction sequence. To read about expert-level Info commands, type `n’ twice. This brings you to `Info for Experts’, skipping over the `Getting Started’ chapter. * Menu: * * * * Getting Started:: Expert Info:: Creating an Info File:: Index:: Getting started using an Info reader. Info commands for experts. How to make your own Info file. An index of topics, commands, and variabl –zz-Info: (info. info. z)Top, 24 lines –All—————————– If you were to scroll down to the Directory listing line, you’ll see that on the left-hand side there’s an asterisk, followed by the topic, followed by a double colon and what is inside the info group: * Directory listing:: ls dir vdir d v dircolors 8 Tour de Shell Scripting These are the topics covered in this particular node. If you hit enter at this stage. You should see that the node has changed. The top line of your page will look as follows: File: coreutils. info, Node: Directory listing, Next: Basic operations,Prev: Ope This is similar to the top of the coreutils. nfo page as described above, but this example includes a previous node, which is “Operating on characters”, with the next node being “Basic operations”. Once I’ve scrolled down (using my arrow keys) to * Directory listing, I may want to go and look at information about the ls command to see what I can do with ls. Again you use the up or down arrow key and scroll to “ls invocation” and hit Enter Once there you can read the ls info page to see what it tells you about the ls command. How do you go back to the Directory listing info page? Type u for UP, which should take you to the previous listing.
How do you go from “Directory listing” to “Basic operations”, when you’re currently at the “Directory listing” node? n will take you to the NEXT node (taking you from the “Directory listing” to “Basic operations”). If you want to go right to the top, in other words, back to the coreutils group, press t for TOP. You can do searches within info by using the forward slash (/) and then include a pattern, which might be something like /Directory This tells info that you want to look for the pattern Directory. Bingo! We find Directory listing, as it is the first entry that matches the pattern.
If you want to use the same pattern to search again, press forward slash followed by enter: / This will allow you to search for the pattern “Directory” once again. How do you Man Pages 9 quit info? q will quit info. If you want to go one page up at a time, then your backspace key will take you one page up at a time. Finally, to obtain help within info, type ‘? ‘. This will get you into the help page for info. To leave help, press CTRL-x-0. That is essentially how info works. Part of the reason for moving to info rather than man pages is to put everything into texinfo format rather than gz-man format.
In the future, much more software will include manual pages in texinfo format, so it’s a good idea to learn how the info pages work. Exercises: 1. Run info on find. 2. press u. To which node of the info page does this take you? 3. Search for the find command. 4. Select the find command. 5. If I were to try to find a file using it’s inode number, how would I do this. (Hint: search for inum) 6. What node does this (inum) form part of? 7. Go to the “Finding Files” node and select the Actions node. 8. How do you run a command on a single file once the file has been found. Man Pages
Having covered the info pages, we need to look at man pages since man is the standard on most UNIX and Linux systems. ‘man’ is short for manual. This is not a sexist operating system. There are no woman pages but we can find out how to make some a little later (to keep man company). } Manual pages are available on every operating system. (If your system administrator hasn’t installed them, ask him politely to do so, as no Linux system should be running without man pages. ). The man command is actually run through a program called less, which is like more except it offers more than the more command does. 0 Tour de Shell Scripting Mark Nudelman, the developer of less, couldn’t call it more, since there was already a more command, so it became less. Linux people do have a sense of humor. To invoke man pages type: man For example, the ls command that we info’ed above, $ man ls | less Looking at our example above, the manual page on the ls command is run through the less command. What can you do from within man? Well, pretty much the things you can do with info. Instead of a menu system, and nodes, we’re looking at a single page detailing all the options and switches of the ls command.
If we want to search for a particular pattern we would use forward slash (/) just like we did in info. For example, we could type /SEE ALSO This pattern search would take us to the end of the man page to look at the SEE ALSO section. We could type question mark with a pattern, which does a reverse search for the specified pattern. Remember forward slash does a forward search and question mark does a reverse search. ?NAME 11 This pattern search will reverse search up the man page to look for the pattern NAME. You will notice that I’m not saying look for the string NAME, rather I’m saying look for the pattern NAME.
This is because pattern matching is a critically important part of UNIX and Linux and shell scripting. We’ll learn more about patterns as we go through the course. If we want to scroll down one page at a time within the man page (i. e. we’ve looked at page 1 and we’ve read and understood it, and we want to go to page 2), then the space bar takes us forward by a page. Similarly if we want to reverse up the man page, we press b for back, which will scroll backwards through the man page. How do we get back to our prompt? The ‘q’ key comes in handy again. ‘q’ for quit. man pages are generally broken down into a host of different sections.
There’s a SYNOPSIS section, a DESCRIPTION section, and a SEE ALSO section. Read through the man page and you will understand the different sections. If you need help on moving around through the man page, type ‘h’ for help, which will give you a listing of all the help commands. You will see that it has displayed the help commands NOT for man but for less. Why? Because the pager for man, (pager, the tool that gives you one page at a time instead of just scrolling the man page past you too fast to read), is the less command We will cover the less command a bit later but you can look it up with the info pages as follows: nfo less So ‘h’ within the man page will show you help on the ‘less’ command at the same time as displaying the requested manual page. Sometimes you need to read a man page three or four times before you completely understand it, and of course sometimes you may never understand it! Don’t be deterred. That’s what separates the kanga’s from the roo’s. Exercises for man: 1. do a man on the nl command 12 Tour de Shell Scripting 2. What is the function of this command? 3. How would one right justify the number ensuring it has leading zeros? 4. And also number non-blank lines? 5.
Who wrote this program? 6. What else should we view to get a complete picture of the nl command? 7. What version of the nl command do you have installed on your system? The whatis Database The whatis database is usually rebuilt on a Linux system at night. The job of the whatis database is to search through every manual page on your system looking for the NAME section within these man pages, classifying them and placing them into a database, to facilitate searching in the future. The whatis database is useful in that it gives us the ability to quickly look up what a particular command does.
So if I asked you to tell me what the nl command does, you could look it up in the man pages or you could look it up in the whatis database. man nl or whatis nl The latter method should return with the NAME section from the man page, showing you what the commands job is on the system. It should tell you that nl numbers the lines. Similarly wc counts words, lines and characters for you. The whatis database is very useful because it allows you to quickly find out, what a particular command on the system does. If the whatis database is not up-to-date, it is quite simple to update it.
Generally though, the updating of the whatis database is a simple automated process. Once a night, the operating system should go about updating the whatis database. Even if the system administrator has installed a whole host of new software on the system, by virtue of the fact that the man pages for that software would be installed at the Addage 1. 3 13 same time as your new application is installed, your whatis database should pick up those pages and add them to its database each night. As a side note, updating the whatis database manually is simply a matter of $ makewhatis -u -w nd the whatis database will be updated. Adage 1. 3 – the notion of being good because you are lazy. The idea behind being lazy is that you want to take a system and get it to do stuff automatically for you, so that you can spend more time surf-skiing or walking on the mountain, or doing things you enjoy doing. Now people say to me “Why must I be lazy? ” Because it means that you need to think of better, quicker ways of doing things and shell scripting is a way to achieve that. If you haven’t thought of a better way of doing it, you’re not applying your mind.
If you apply your mind you will find that there are many different ways to skin a cat in Linux. Shell scripting is one of the many ways you can speed up mundane tasks. So the idea behind shell scripting is to automate this process of getting jobs to be done on your behalf. To achieve this using scripts, you could take a series of system administration tasks, put them together in a single script, run them unattended and they should produce output that would (hopefully) match what you require. Finally, this brings me to another adage. Addage 1. 3 There is never only one way of solving a problem in Linux.
The way I solve a problem may be completely different to the way you solve it. But does this matter? Absolutely not! Is my solution better than yours? Perhaps, but I have been doing this for a long time, so take note of how I do it. Revising some Basic Commands 14 Tour de Shell Scripting There are some basic commands that we are going to look at. The idea is to get you into the process of understanding how commands are structured and build an understanding of what the commands do. From hereon out, I’m going to assume that you can find out lots of things about commands primarily by looking at info and man pages.
Almost every Linux command can be run from the command line using various switches (or arguments / options) which allow one to change the output of this command in a number of different ways. The who command The who command is designed to tell you who’s logged on to the system. If we run the who command without any switches, the left hand column shows the user id. This the user currently logged on to the system. In your case, you might be logged on as root, or perhaps as your user. The second column indicates where you are logged in. riaan@debian:~> who riaan :0 riaan pts/0 riaan pts/3 iaan pts/4 riaan pts/5 riaan pts/6 riaan pts/7 riaan@debian:~> Apr Apr Apr Apr Apr Apr Apr 30 30 30 30 30 30 30 11:13 (console) 11:13 11:14 11:30 13:19 12:07 12:09 So if you look at the who command output, my user riaan is logged in from :0 which is the X console. He’s also logged on to pts/0 and pts/1 These are pseudo terminals, indicating he’s also logged into two pseudo terminals. The final, third column indicates what time the user logged on. The who command tells us about users using our system. That’s great! What are the other switches that we can use with who. 15 who –help
This will show you the various switches that we can use with the who; command. So if we use a: who -H it prints a heading line for us. The output should look as follows: $ who -H NAME heidi mwest heidi LINE ttyp1 ttyp2 ttyp4 TIME FROM Nov 27 17:29 (168. 210. 56. 177:S) Nov 10 15:04 (apotheosis) Nov 11 13:18 (168. 210. 56. 177:S) To view a short listing which is the default listing for the who command: who -s Using the -u switch: who -u will show the users and their process id”s. In scripts, one can use the same commands as on the command line, including all the switches those commands use.
One can run any command and produce standard text output, which one can use. We’ll talk about how you can use the output later. Run the command $ who -u root tty2 riaan :0 Aug Aug 4 10:41 4 07:53 . old 2839 2836 (console) 16 Tour de Shell Scripting to identify which users are logged into your system and from which processes they are logged on. This will show how long a terminal has been idle. It will show not only which users are logged on and what process ids they are but also how long that user has been idle. Idle users might have gone out for lunch or they might have left for the day.
In default mode, most of these systems don’t log you out if you’re idle for longer than 10 or 15 minutes. In the old days, most systems were configured to automatically log you out after 15 minutes. On Debian, the -i switch does not add any extra output, it simply prints a message suggesting that you not use -i as it will be removed in future releases. Use -u as above. However the -i switch may work with other brands of Linux. Okay, so that’s the who command. We’re going to use these commands later to build a system to monitor our system automatically, because we want to be spending our time doing things we enjoy. ho command exercises: 1. How many users are currently logged onto your system? 2. Is any user logged onto your terminals tty1 -> tty6? 3. Who (which user and or terminal) has been idle for the longest time? 4. Ensure that all output is displayed with a heading line. 5. What run-level are you currently running in and what was your previous run-level? What other command will show you this information? 6. What would you imagine a users message status would be? (Take a guess or read the man page for write) The w Command What does the w command do? You could run: riaan@debian:~> whatis w w (1) Show who is logged on and what they are doing. riaan@debian:~> The “date” command 17 riaan@linux:~> w 21:40:17 up 11:03, 6 users, load average: 0. 30, 0. 34, 0. 30 USER TTY LOGIN@ IDLE JCPU PCPU WHAT root tty2 21:40 8. 00s 0. 06s 0. 06s -bash riaan :0 10:38 ?xdm? 58:31 0. 04s -:0 riaan pts/0 10:38 11:01m 0. 00s 2. 08s kdeinit: kwrited riaan pts/3 11:18 10:22m 14:37 2. 63s /usr/lib/java/bin/java -m riaan pts/4 11:28 1:07m 0. 21s 0. 21s /bin/bash riaan pts/5 11:28 0. 00s 0. 17s 0. 03s w which should print some information about the w command. The w command not only tells us who are logged in, but also what they’re doing.
Are these users running applications? What actual processes are they running at this time? Perhaps someone’s running an application like OpenOffice. w will tell us this. If you look at the output of this command, it’s got a list of headings that are fairly similar to the format of the who command. Later we’ll have a look at modifying the report columns, to get the output into a different format that may be more useful. The “date” command One of the reasons for taking you through these commands is that we’re going to start writing our first shell scripts using these commands, so it is as well that we understand them now.
The date command is a useful command that can do all sorts of nifty things for us (apart from printing out the date). It can convert between Unix time, (which is the number of seconds since 1/1/1970 commonly known as the epoch) and a human readable (normal) date and time. Conversely, it can also convert back from date time today to the number of seconds that have elapsed since the 1/1/1970. It can format output in a whole variety of ways. Let’s look at some examples of the date command. For that I’m going to do: info date If you scroll down, you will see a section with examples.
Looking at the example: 18 Tour de Shell Scripting date +” ” We may now include a string describing the format inside these quotation marks. In the shell there’s a big distinction between double quotes, single quotes (which is another lesson altogether, see Chapter 5 [115]), and back quotes – let’s not get them confused for now. Within this double quoted string we can include any number of arguments. What arguments can we include? Each argument starts with a percentage sign. To display the time, we could use: %H – -will give us the hours in 24 hour format (0-23). M – -will give us the minutes (0-59) of the day If we had the following string: date +”%H:%M” we will end up with hours and minutes of the day on our system. The result of the above command should be similar to: 15:04 But let’s say that we want the hours in 12-hour format rather than 24-hour format. We could then replace the %H with a %l. The result would then be: 3:04 There’s a host of other things that we could do. For example if we are in 12-hour format, 3:04 doesn’t indicate whether it is morning or afternoon. Thus we could include %p: date +”%l:%M %p” 19 This would show us that the time is actually: :04 PM rather than 3:04 AM. That’s for time, but what about for the date? What happens if we want to show the date, which is: 24-Nov-2003 then, we should in theory be able to create a date string to reflect this format. A way we can do is this is using the following: date +”%d-%b-%Y” where %b is a short format for month to produce Nov instead of November. If we want to combine the date and time: date +”%d-%b-%Y %l:%M %p” This would give us the full date and time report: 24-Nov-2003 3:04 PM There are a lot of other parameters that you can use within the date command.
You can view these by looking at the relevant info page with : info date 20 Tour de Shell Scripting We’re going to use this command in our script, because in almost every script that you will write you are going to want to know what time the script started, what time the script ended, when it did a particular job within the script, etc. date Exercises 1. Using the info command for assistance, format the output of date, so that it resembles the following: Today’s date is Tuesday, 27 January 2004. The time is currently 11h 32m and 49s. 2. Show the date in Julian format. 3. What day of the year is today? . Include in the output of the date, the time zone and the AM/PM indicator 5. Given the number 1075200287, determine what date, day and time this represents. The ‘echo’ command The final command I want to describe is a command used to send output to the screen: echo. We’ve seen so far that we were able to run commands but, as yet, we don’t know how to simply output some text to the screen. We may want to print a string to the screen, prior to printing the date. Something such as: Today’s date is: 24-Nov-2003 3:04 PM We would need some way of echoing that to the screen, wouldn’t we?
In order to do this, there is the echo command. echo can be a bit of a nasty gremlin because there are at least two echo commands on your system. One is a shell built-in, and one is an external command and therefore it can be a bit tricky. We’re going to start off with a vanilla case. Later on we will see how we can choose which echo command we want to use. So by way of an example, we’ll use it to format the date command. 21 echo “Today’s date is: ” date +”%d-%b-%Y %l:%M %p” This would be a good time to show you how to create your first shell script. We’re going to edit a file and for this you can use any editor1
Open your editor (whatever editor you prefer) and put the following commands in the first lines of the file: echo “Today’s date is: ” date +”%d-%b-%Y %l:%M %p” Save that file as myfirstscript and exit your editor. You’ve just created your first shell script. Great! How easy was that? How do you run it to make it actually do its job? Running a Shell Script Linux has three sets of permissions that set the permission mode of the file . One for the owner, one for the group and one for everyone else (i. e. Everyone that is not the owner, and is not part of the group to which the file belongs).
You would have covered this in an earlier course (Linux Fundamentals). The mode of the file will also determine whether the file is executable or not. Thus, to make the file executable, the mode of the file must have the execute (x) permissions set. Note that this differs from Microsoft Windows which looks at the extension to decide the type of the file. Windows assumes that . com and . exe are executable files. Notice too, that myfirstscript does not have an extension. Is that valid? Sure it is. This is Linux after all!! In sum, to make our script executable we must change the mode of the file.
How? Using chmod (change mode) command as follows: 1 If you haven’t been taught vi, an excellent place to start is using vimtutor. Type vimtutor on the command line to begin the lessons. It only takes 45 minutes, but is well worth the effort. vi is the best editor in the world, in my humble opinion Almost everyone in the UNIX/Linux world has some relationship with it – a love-hate relationship: some people love it, some people hate it. But if you’re going to learn any editor, learn vi. Since it is the de-facto editor on almost any UNIX/Linux variant in the market today, earning the basics at least should stand you in good stead in you years as a system administrator. 22 Tour de Shell Scripting chmod +x myfirstscript This script is now executable. Does this mean that we’ve executed it? No, not yet. We execute it with: ./myfirstscript and that should output: Today’s date is: 24-Nov-2003 3:04 PM Finally, you will notice that in order to execute this command, I preceded it with a “. /”. Try running the script without the “. /”. What happens? What the “. /” does is that the shell FIRST looks in they current directory (. ) for the script before trying to look in the PATH.
We will cover this is a little more detail later. Of course, you could add the script to a new directory ~/bin (in my case /home/hamish/bin). Since this directory is in my PATH environment, the script will be “found” and will execute even without using “. /” Creating Scripts Exercises 1. Write a simple script to print “Hello ” on the screen. 2. Modify your scripts to additionally output the directory you are currently in. 3. Write a simple menu system, with the menu options as follows(note: this menu does not have to respond to the user’s input at this stage): • 0. Exit • 1. Output a text version • 2. Output an HTML version
Challenge sequence: 23 • 3. Print help • 4. Provide a shell prompt Select an option [1-4] or 0 to exit: 4. Include the current date and time in the top right hand corner of your menu 5. Ensure that your menu system (I’m assuming you called it menu. sh) can execute. Execute it. Challenge sequence: Consult the appendixAppendix B [253] and teach yourself the dialog package. There are many examples in /usr/share/doc/dialog-xxx. yyy. Once you understand this package, modify your menu system in 3 above to use the ncurses library (dialog) to spruce it up. File Commands This section is going to cover file commands.
File commands are commands such as ls (list). Notice again, how the laziness of the Unix people comes to the fore. They could have had a command called list, but that would have required two more characters (and two more carpals – fingers! ) and clearly that was a lot more effort, so we just have the ls command. The ls command shows us a listing of files in a particular directory. This is an appropriate place to take a detour on our tour de scripting and have a look at file matching and wildcard matching. It may be something that you’re familiar with, but let’s have a look at it and come back to ls in a moment.
Wildcards Wildcard matching allows us to match a number of files using a combination of characters. What are those characters? Table 1. 1. Wildcards Symbol Name Purpose 24 Tour de Shell Scripting * Splat ? [] question mark square brackets ! bang matches 0 or more of any character matches 0 or 1 character matches one from a range of characters invert the pattern Wildcard characters with the asterisk (*) So if we typed ls * we end up listing 0 or more of any characters. So it would match any filename of any length because any filename would have a minimum of a single character.
The splat matches 0 or more characters following each other. the question mark ( ? ) wildcard character The question mark will match a single instance of any character. Later, when we study regular expressions, the full stop ( . ) matches any single character. Given the expression: ls -la ab? a this would match the files: abba ab1a ab_a ab9a abca … The square brackets ( [] ) What range of characters do [] include? Well we may say something like: [abc] 25 which would match any (single character) within the following range: a or b or c but it does not match abc since this is 3 characters long. Similarly [ab]cde ould match acde or bcde but it would not match abcde. Square brackets indicate a range of alternative characters. How would we match both: ham01 and ham10 Simple, with: ham[0-9][0-9] as [0-9] implies that any character in the range 0 to 9 will match. Would it also match ham9? No, for the reason that our pattern search is expecting 2 numeric characters after the pattern ham. (two sets of square brackets are displayed) We could also combine letters and numbers, for example: ls [abc123] which would match the files: a b 26 Tour de Shell Scripting c 1 2 3 (Yes Linux and UNIX can have files with all of the above names! Notice when we used the square brackets [0-9], we use the range specifier, which was the dash character in the middle. This dash has nothing to do with the minus sign and means match anything in that range from 0 through to 9. Thus, typing: [a-z] matches any character in the range from a to z. the bang ( ! ) The final pattern matching wildcard is the bang command. The exclamation mark is the inverse of what you’re trying to match. If we were to look at our pattern matching, we could say:ls [! a-c]* which, would match everything NOT starting with an “a” or a “b” or a “c”, followed by anything else.
So would it match abc ? No, because it starts with an a. Would it match bcde? No, because it starts with a b. Finally would it match erst? Yes, because q is not in the range a to c. So it would match q followed by any set of zero or more characters. Let’s do some more examples using the pattern matching 27 wildcards. Prior to doing the following command ensure that you create a new temporary directory, and change directory to this new directory [ by doing so, it will be easier to clean up after the exercise is over. ] I’d like you run the following command (I’m not going to explain it now, it will be covered later). ouch {planes,trains,boats,bikes}_{10,1,101,1001}. {bak,bat,zip,tar} This command creates a number of files for us in one go. Typing: ls p* will show us all files that start with a ‘p’, followed by 0 or more characters (planes_ in our case). Similarly ls b* will show us the files boats_ and bikes_ since only these files begin with a ‘b’. If we typed: ls bik* it will produce all the bikes files, while ls b*_??. * will indicate all the files that start with a ‘b’, followed by 0 or more characters, followed by an underscore followed by exactly 2 characters, followed by a full stop, followed by 0 or more characters.
Thus only boats_10. bak boats_10. bat boats_10. zip 28 Tour de Shell Scripting boats_10. tar bikes_10. bak bikes_10. bat bikes_10. zip bikes_10. tar I’ve given you some additional examples that you can try out in your own time. It would be worth your time running through these examples and ensuring that you understand everything. Exercises: Run the following command in a new subdirectory touch {fred,mary,joe,frank,penny}_{williams,wells,ferreira,gammon}. {1,2,3,4,5} Look at the following tu understand the different fields touch {fred,mary,joe,frank,penny}_{williams,wells,ferreira,gammon}. 1,2,3,4,5} ——Name——————-Surname——-Category- 1. list all the people who fall into category 5 2. List all the people whose surnames begin with a w. 3. List only people in category 2, whom’s surnames begin with a w, and whom’s first name begins with an “f” 4. Move all “fred’s” into a directory on their own, but exclude the “gammons” 5. Ensure that joe and penny wells in category 3 and 5 are deleted. 6. List only those people whose names have 5 characters in them. Returning to file commands Returning from our detour, there’s a whole bunch of other file commands that we can look at.
We’ve touched on the ls command, which gives us a listing of files and from the previous set of examples on pattern matching and wildcarding, you will have gotten an idea of how ls works. System Commands 29 ls , like every other Linux command can take switches. Below is a quick summary to some of the switch options. -l show a long listing (include file name, file size, date last modified, the permissions, the mode of the file, the owner, the group of the file) -a shows all files including {hidden} files (. and .. ) Two special hidden files are: . .. this file is our current directory his file is our previous or parent directory. Often in your home directory, you will have full control over your . directory (i. e. your current directory) but you will most probably have absolutely no control of your .. directory (i. e. your parent directory). We have other commands like: cp copy files or directories mv move files or directories wc shows the number of lines, number of words and number of characters in a file. wc -l show us the number of lines in our file. nl numbers the lines in our file. System Commands There are many system commands that we can use.
We’re going to start using these in our shell scripts. Remember, a shell script is nothing more than a group of Linux commands working together to produce a new command. In order to build our system that is going to manage our Linux machine, we’re going to need to know a little bit about system commands. System commands such as: df shows the disk free space du shows the disk usage fdisk shows the partitioning on our disk 30 Tour de Shell Scripting iostat shows the input output status vmstat shows the virtual memory status free shows the amount of free memory
We will use these commands, but they are a subset of the commands available to us for monitoring our system. The df and du commands For example, mostly we want to run a command to check whether our file systems are filling up. It makes no sense to have a full filesystem! For that we might use the df command df would produce a listing of our partitions on our Linux machine and will show us which partitions are 100% full, which partitions are almost empty, which partitions are mounted, etc.. What we’re going to be doing, is working our way towards building a system that will automatically show us when a partition becomes 100% full.
Or perhaps we want to build a system that shows us when it’s greater than a particular threshold. So we might set a threshold of 95% full, it’s no good telling us when the system has crashed that it’s crashed; we want to know before the system crashes. Several switches can be used with df such as: df -h This produces output in human readable format. If you run this command now, you should see at least one partition and that’s the root partition. It should show you how much space is being used, how much is available, what the size of the partition is and what particular filesystem it’s mounted on.
The df command is what we’ll start working on for our practical, because it most certainly is one of the most useful commands that you’re going to need to run to make sure your system is not becoming overfull. Alternately, you could use the du (disk usage) command to show you which files are consuming the largest volume of disk space: du – s 31 will show a summary of our current filesystems’ capacity. Again, how do you get information about these commands? Using the info or man command will inform you about switches pertinent to these commands. For example a further switch for the df command might be: df – hT hich will tell us what type of filesystem we’re running. It might be an extended 2, 3, or a vfat filesystem. The “du” command, like the “df” command, has a complete set of switches unique to it and shortly we’ll start looking at those in a bit more detail. the fdisk command The fdisk command is used primarily to show us what partitions we have on a particular disk. BlackBeauty:/install # fdisk -l Disk /dev/hda: 10. 0 GB, 10056130560 bytes 240 heads, 63 sectors/track, 1299 cylinders Units = cylinders of 15120 * 512 = 7741440 bytes Device /dev/hda1 /dev/hda2 /dev/hda5 /dev/hda6 Boot * Start 1 761 761 828
