Shell Scripting Recipes
A Problem-Solution Approach

Chris F.A. Johnson

Contents
About the Author xv
About the Technical Reviewer xvii
Acknowledgments xix
Introduction xxi

CHAPTER 1:
The POSIX Shell and Command-Line Utilities

Shell Commands 1
Parameters and Variables 7
    Positional Parameters 7
  Special Parameters 7
standard-vars — A Collection of Useful Variables 9
Patterns 9
  Filename expansion 10
  Regular Expressions 10
Parameter Expansion 12
  The Bourne Shell Expansions 12
  POSIX Parameter Expansions 14
  Shell-Specific Expansions, bash2, and ksh93 15
Shell Arithmetic 16
Aliases 16
Sourcing a File 17
Functions 17
  Functions Are Fast 17
  Command Substitution Is Slow 17
  Using the Functions in This Book 18
standard-funcs: A Collection of Useful Commands 19
  1.1 get_key — Get a Single Keystroke from the User 19
  1.2 getline — Prompt User to Enter a Line 20
  1.3 press_any_key — Prompt for a Single Keypress 21
  1.4 menu1 — Print a Menu and Execute a Selected Command 22
  1.5 arg — Prompt for Required Argument If None Supplied 23
  1.6 die — Print Error Message and Exit with Error Status 24
  1.7 show_date — Display Date in D[D] MMM YYYY Format 25
  1.8 date_vars — Set Date and Time Variables 26
  1.9 is_num — Is This a Positive Integer? 28
  1.10 abbrev_num — Abbreviate Large Numbers 29
  1.11 commas — Add Thousands Separators to a Number 30
  1.12 pr1 — Print Arguments, One to a Line 32
  1.13 checkdirs — Check for Directories; Create If Necessary 33
  1.14 checkfiles — Check That a Directory Contains Certain Files 33
  1.15 zpad — Pad a Number with Leading Zeroes 34
  1.16 cleanup — Remove Temporary Files and Reset Terminal on Exit 35
The Unix Utilities 36
Summary 40

CHAPTER 2
Playing with Files: Viewing, Manipulating, and Editing Text Files

End of the Line for OS Differences: Converting Text Files 42
  2.1 dos2unix — Convert Windows Text Files to Unix 42
  2.2 unix2dos — Convert a Unix File to Windows 43
  2.3 mac2unix — Convert Macintosh Files to Unix 44
  2.4 unix2mac — Convert Unix Files to Mac Format 44
  2.5 dos2mac — Convert Windows Files to Macintosh 45
  2.6 mac2dos — Convert Macintosh Files to Windows 45
Displaying Files 46
  2.7 prn — Print File with Line Numbers 46
  2.8 prw — Print One Word per Line 49
  2.9 wbl — Sort Words by Length 51
Formatting File Facts 52
  2.10 finfo — File Information Display 52
  2.11 wfreq — Word Frequency 56
  2.12 lfreq — Letter Frequency 56
  2.13 fed — A Simple Batch File Editor 58
Summary 60

CHAPTER 3
String Briefs

Character Actions: The char-funcs Library 61
  3.1 chr — Convert a Decimal Number to an ASCII Character 62
  3.2 asc — Convert a Character to Its Decimal Equivalent 63
  3.3 nxt — Get the Next Character in ASCII Sequence 64
  3.4 upr — Convert Character(s) to Uppercase 65
  3.5 lwr — Convert Character(s) to Lowercase 67
String Cleaning: The string-funcs Library 69
  3.6 sub — Replace First Occurrence of a Pattern 69
  3.7 gsub — Globally Replace a Pattern in a String 70
  3.8 repeat — Build a String of a Specified Length 72
  3.9 index, rindex — Find Position of One String Within Another 74
  3.10 substr — Extract a Portion of a String 76
  3.11 insert_str — Place One String Inside Another 78
Summary 79

CHAPTER 4
What's in a Word?

Finding and Massaging Word Lists 82
wf-funcs: WordFinder Function Library 82
  4.1 write_config — Write User's Information to the Configuration File 82
  4.2 do_config — Check For and Source Default Configuration File 83
  4.3 set_sysdict — Select the Dictionary Directory 84
  4.4 mkwsig — Sort Letters in a Word 85
  4.5 wf-clean — Remove Carriage Returns and Accents 86
  4.6 wf-compounds — Squish Compound Words and Save with Lengths 87
  4.7 wf-setup — Prepare Word and Anagram Lists 89
Playing with Matches 91
  4.8 wf — Find Words That Match a Pattern 91
  4.9 wfb — Find Words That Begin with a Given Pattern 93
  4.10 wfe — Find Words That End with a Given Pattern 94
  4.11 wfc — Find Words That Contain a Given Pattern 95
  4.12 wfit — Find Words That Fit Together in a Grid 97
  4.13 anagram — Find Words That Are Anagrams of a Given Word 100
  4.14 aplus — Find Anagrams of a Word with a Letter Added 101
  4.15 aminus — Remove Each Letter in Turn and Anagram What's Left 103
Summary 104

CHAPTER 5
Scripting by Numbers

The math-funcs Library 105
  5.1 fpmul — Multiply Decimal Fractions 106
  5.2 int — Return the Integer Portion of a Decimal Fraction 110
  5.3 round — Round the Argument to the Nearest Integer 110
  5.4 pow — Raise a Number to Any Given Power 111
  5.5 square — Raise a Number to the Second Power 113
  5.6 cube — Raise a Number to the Third Power 113
  5.7 calc — A Simple Command-Line Calculator 114
Adding and Averaging 115
  5.8 total — Add a List of Numbers 115
  5.9 mean — Find the Arithmetic Mean of a List of Numbers 116
  5.10 median — Find the Median of a List of Numbers 118
  5.11 mode — Find the Number That Appears Most in a List 119
  5.12 range — Find the Range of a Set of Numbers 120
  5.13 stdev — Finding the Standard Deviation 122
Converting Between Unit Systems 123
  5.14 conversion-funcs — Converting Metric Units 124
  5.15 conversion — A Menu System for Metric Conversion 131
Summary 135

CHAPTER 6
Loose Names Sink Scripts: Bringing Sanity to Filenames

What's in a Name? 137
  POSIX Portable Filenames 138
  OK Filenames 138
Functioning Filenames: The filename-funcs Library 139
  6.1 basename — Extract the Last Element of a Pathname 139
  6.2 dirname — Return All but the Last Element of a Pathname 141
  6.3 is_pfname — Check for POSIX Portable Filename 142
  6.4 is_OKfname — Check Whether a Filename Is Acceptable 143
  6.5 pfname — Convert Nonportable Characters in Filename 144
  6.6 OKfname — Make a Filename Acceptable 145
  6.7 is_whitespc — Does the Filename Contain Whitespace Characters? 146
  6.8 whitespc — Fix Filenames Containing Whitespace Characters 146
  6.9 is_dir — Is This a Directory I See Before Me? 147
  6.10 nodoublechar — Remove Duplicate Characters from a String 148
  6.11 new_filename — Change Filename to Desired Character Set 149
  6.12 fix_pwd — Fix All the Filenames in the Current Directory 150
  6.13 fixfname — Convert Filenames to Sensible Names 152
Summary 154

CHAPTER 7
Treading a Righteous PATH

The path-funcs Library 157
  7.1 path — Display a User's PATH 157
  7.2 unslash — Remove Extraneous Slashes 158
  7.3 checkpath — Clean Up the PATH Variable 159
  7.4 addpath — Add Directories to the PATH Variable 161
  7.5 rmpath — Remove One or More Directories from $PATH 163
Summary 164

CHAPTER 8
The Dating Game (Sample chapter: PDF or HTML )

The date-funcs Library 166
  8.1 split_date — Divide a Date into Day, Month, and Year 166
  8.2 is_leap_year — Is There an Extra Day This Year? 167
  8.3 days_in_month — How Many Days Hath September? 169
  8.4 date2julian — Calculate the Julian Day Number 170
  8.5 julian2date — Convert Julian Back to Year, Month, and Day 170
  8.6 dateshift — Add or Subtract a Number of Days 172
  8.7 diffdate — Find the Number of Days Between Two Dates 174
  8.8 day_of_week — Find the Day of the Week for Any Date 175
  8.9 display_date — Show a Date in Text Format 176
  8.10 parse_date — Decipher Various Forms of Date String 179
  8.11 valid_date — Where Was I on November 31st? 183
Summary 184

CHAPTER 9
Good Housekeeping: Monitoring and Tidying Up File Systems

  9.1 dfcmp — Notify User of Major Changes in Disk Usage 185
  9.2 symfix — Remove Broken Symbolic Links 189
  9.3 sym2file — Converts Symbolic Links to Regular Files 191
  9.4 zrm — Remove Empty Files 193
  9.5 undup — Remove Duplicate Files 194
  9.6 lsr — List the Most Recent (or Oldest) Files in a Directory 196
Summary 198

CHAPTER 10
POP Goes the E-Mail

  10.1 popcheck — Display Number of Messages in POP3 Mailbox 199
  10.2 pop3-funcs — Functions for Managing a POP3 Mailbox 200
  10.3 pop3list — Examine a POP3 Mailbox 215
  10.4 pop3filter — Delete and Retrieve Messages from a POP3 Server 222
  10.5 pflog — Print a Summary of a Day's Activity 224
  10.6 viewlog — Continuously View the pop3filter Log File 225
Summary 227

CHAPTER 11
PostScript: More Than an Afterthought

  11.1 ps-labels — Produce Mailing Labels from a List 231
  11.2 ps-envelopes — Print Envelopes from List 237
  11.3 ps-grid — Convert a Crossword Grid to PostScript 242
Summary 249

CHAPTER 12
Screenplay: The screen-funcs Library

  12.1 screen-vars — Variables for Screen Manipulation 252
  12.2 set_attr — Set Screen-Printing Attributes 254
  12.3 set_fg, set_bg, set_fgbg — Set Colors for Printing to the Screen 255
  12.4 cls — Clear the Screen 256
  12.5 printat — Position Cursor by Row and Column 257
  12.6 put_block_at — Print Lines in a Column Anywhere on the Screen 260
  12.7 get_size — Set LINES and COLUMNS Variables 262
  12.8 max_length — Find the Length of the Longest Argument 263
  12.9 print_block_at — Print a Block of Lines Anywhere on the Screen 264
  12.10 vbar, hbar — Print a Vertical or Horizontal Bar 265
  12.11 center — Center a String on N Columns 267
  12.12 flush_right — Align String with the Right Margin 268
  12.13 ruler — Draw a Ruler Across the Width and Height of the Window 269
  12.14 box_block, box_block_at — Print Text Surrounded by a Box 271
  12.15 clear_area, clear_area_at — Clear an Area of the Screen 272
  12.16 box_area, box_area_at — Draw a Box Around an Area 273
  12.17 screen-demo — Saving and Redisplaying Areas of the Screen 275
Summary 277

CHAPTER 13
Backing Up the Drive

  13.1 bu — Backup Directories 280
  13.2 bin-pack — Fill Fixed-Capacity Directories with Files 285
  13.3 unpack — Extract All Files from One or More Archives 290
Summary 292

CHAPTER 14
Aging, Archiving, and Deleting Files

  14.1 date-file — Add a Datestamp to a Filename 293
  14.2 rmold — Remove Old Files 297
  14.3 keepnewest — Remove All but the Newest or Oldest Files 300
Summary 302

CHAPTER 15
Covering All Your Databases

  15.1 lookup — Find the Corresponding Value for a Key 304
  shdb-funcs: Shell Database Function Library 306
  15.2 load_db — Import Database into Shell Array 306
  15.3 split_record — Split a Record into Fields 307
  15.4 csv_split — Extract Fields from CSV Records 309
  15.5 put_record — Assemble a Database Record from an Array 310
  15.6 put_csv — Assemble Fields into a CSV Record 311
  15.7 db-demo — View and Edit a Password File 312
PhoneBase: A Simple Phone Number Database 317
  15.8 ph — Look Up a Phone Number 318
  15.9 phadd — Add an Entry to PhoneBase 319
  15.10 phdel — Delete an Entry from PhoneBase 320
  15.11 phx — Show ph Search Results in an X Window 320
Summary 321

CHAPTER 16
Home on the Web

Playing with Hypertext: The html-funcs Library 323
  16.1 get_element — Extract the First Occurrence of an Element 323
  16.2 split_tags — Put Each Tag on Its Own Line 326
  16.3 html-title — Get the Title from an HTML File 327
HTML on the Fly: The cgi-funcs Library 329
  16.4 x2d2 — Convert a Two-Digit Hexadecimal Number to Decimal 329
  16.5 dehex — Convert Hex Strings (%XX) to Characters 330
  16.6 filedate — Find and Format the Modification Date of a File 332
Creating HTML Files 333
  16.7 mk-htmlindex — Create an HTML Index 333
  16.8 pretext — Create a Wrapper Around a Text File 341
  16.9 text2html — Convert a Text File to HTML 342
  16.10 demo.cgi — A CGI Script 343
Summary 344

CHAPTER 17
Taking Care of Business

  17.1 prcalc — A Printing Calculator 345
  17.2 gle — Keeping Records Without a Shoebox 350
Summary 360

CHAPTER 18
Random Acts of Scripting

The rand-funcs Library 361
  18.1 random — Return One or More Random Integers in a Given Range 362
  18.2 toss — Simulate Tossing a Coin 365
  18.3 randstr — Select a String at Random 366
A Random Sampling of Scripts 367
  18.4 rand-date — Generate Random Dates in ISO Format 367
  18.5 randsort — Print Lines in Random Order 369
  18.6 randomword — Generate Random Words According to Format Specifications 370
  18.7 dice — Roll a Set of Dice 375
  18.8 throw — Throw a Pair of Dice 378
Summary 379

CHAPTER 19
A Smorgasbord of Scripts

  19.1 topntail — Remove Top and Bottom Lines from a File 381
  19.2 flocate — Locate Files by Filename Alone 383
  19.3 sus — Display a POSIX Man Page 384
  19.4 cwbw — Count Words Beginning With 385
  19.5 cci — Configure, Compile, and Install from Tarball 386
  19.6 ipaddr — Find a Computer's Network Address 387
  19.7 ipaddr.cgi — Print the Remote Address of an HTTP Connection 388
  19.8 iprev — Reverse the Order of Digits in an IP Address 389
  19.9 intersperse — Insert a String Between Doubled Characters 389
  19.10 ll — Use a Pager for a Directory Listing Only If Necessary 390
  19.11 name-split — Divide a Person's Full Name into First, Last, and Middle Names 391
  19.12 rot13 — Encode or Decode Text 392
  19.13 showfstab — Show Information from /etc/fstab 393
  19.14 unique — Remove All Duplicate Lines from a File 395
Summary 395

CHAPTER 20
Script Development Management

  20.1 script-setup — Prepare the Scripting Environment 397
  20.2 cpsh — Install Script and Make Backup Copy 402
  20.3 shgrep — Search Scripts for String or Regular Expression 403
  20.4 shcat — Display a Shell Script 404
Summary 405

APPENDIX
Internet Scripting Resources

Introductions to Shell Scripting 407
Intermediate and Advanced Scripting 408
Collections of Scripts 408
Home Pages for Shells 408
Regular Expressions, sed, and awk 408
Miscellaneous Pages 409
History of the Shell 409

INDEX