Mastering Advanced Shell Scripting: Automation & Logging

The journey continues to excel in shell scripting using the simple techniques. As you will find from the previous two tutorials:

Part 1: https://pixelhowl.com/advanced-shell-scripting-master-class/

Part 2: https://pixelhowl.com/advanced-shell-scripting-part-2-automation-server-management/

we have moved through some fairly important and complex concepts with examples and real world sample scripts that you can try on a test machine. Today we will try to go a bit deeper into the real essence, that is the text processing which is quite important as an admin.

Without text manipulation you will have to manually go through the results and get the data out. But we can automate the process so lets dive in without wasting time.

Advanced Text Processing Techniques

Text processing is a fundamental aspect of Bash scripting, and mastering advanced techniques can significantly enhance your ability to manipulate and analyze data. In this section, we’ll explore powerful tools and methods for processing text efficiently in your Bash scripts.

Sed: Stream Editor for Text Transformation

sed is a powerful stream editor that allows you to perform complex text transformations. Here are some advanced sed techniques:

Multi-line Pattern Matching

You can use sed to match and manipulate multi-line patterns:

sed -n '/START/,/END/p' file.txt

This command prints all lines between ‘START’ and ‘END’ (inclusive).

In-place Editing with Backup

Modify files in-place while creating a backup:

sed -i.bak 's/old/new/g' file.txt

This replaces all occurrences of ‘old’ with ‘new’ and creates a backup file named ‘file.txt.bak’.

Using Sed with Variables

You can use variables in sed commands by employing shell quoting:

old_text="apple"
new_text="orange"
sed "s/$old_text/$new_text/g" file.txt

Awk: Pattern Scanning and Processing

awk is a versatile tool for processing structured text data. Here are some advanced awk techniques:

Custom Field Separators

Use custom field separators for parsing structured data:

awk -F':' '{print $1, $3}' /etc/passwd

This prints the first and third fields of the passwd file, using ‘:’ as the separator.

Conditional Processing

Perform actions based on conditions:

awk '$3 > 1000 {print $1, $3}' data.txt

This prints the first and third fields only for lines where the third field is greater than 1000.

Calculating Sums and Averages

Use awk for simple calculations:

awk '{sum += $1} END {print "Average:", sum/NR}' numbers.txt

This calculates the average of numbers in the first column.

Regular Expressions for Pattern Matching

Regular expressions (regex) are powerful tools for pattern matching and text manipulation. Here are some advanced regex techniques:

Lookahead and Lookbehind Assertions

Use lookahead and lookbehind to match patterns based on what comes before or after:

grep -P '(?<=\bfoo\s)\w+' file.txt  # Matches words that come after 'foo'
grep -P '\w+(?=\sbar\b)' file.txt   # Matches words that come before 'bar'

Non-Greedy Matching

Use ? after quantifiers for non-greedy matching:

echo "This is a <tag>example</tag> text" | grep -oP '<.*?>'

This matches the shortest possible string between < and >.

Named Capture Groups

Use named capture groups for more readable regex:

grep -P '(?<year>\d{4})-(?<month>\d{2})-(?<day>\d{2})' dates.txt

Advanced String Manipulation in Bash

Bash itself provides several built-in string manipulation techniques:

Substring Extraction

Extract parts of strings using parameter expansion:

string="Hello, World!"
echo ${string:7:5}  # Outputs: World

String Replacement

Replace parts of strings:

string="Hello, World!"
echo ${string/World/Universe}  # Outputs: Hello, Universe!

Case Conversion

Convert string case:

string="Hello, World!"
echo ${string^^}  # Outputs: HELLO, WORLD!
echo ${string,,}  # Outputs: hello, world!

By mastering these advanced text processing techniques, you’ll be able to handle complex data manipulation tasks efficiently in your Bash scripts. These tools and methods provide powerful ways to extract, transform, and analyze text data, enabling you to create more sophisticated and capable scripts for a wide range of applications.

Implementing Robust Error Handling

Error handling is a crucial aspect of writing reliable and maintainable Bash scripts. Proper error handling helps you identify and address issues quickly, prevent unexpected behavior, and provide meaningful feedback to users. In this section, we’ll explore advanced techniques for implementing robust error handling in your Bash scripts.

Setting Up Error Traps

The trap command allows you to catch signals and execute specific code when they occur. This is particularly useful for cleaning up temporary files or resetting the environment when a script exits unexpectedly:

cleanup() {
    echo "Cleaning up temporary files..."
    rm -f /tmp/tempfile_$$
    exit 1
}

trap cleanup EXIT

# Your script logic here

This setup ensures that the cleanup function is called when the script exits, regardless of how it exits.

Handling Specific Error Conditions

You can use conditional statements to check for specific error conditions and handle them appropriately:

if ! command -v jq &> /dev/null; then
    echo "Error: jq is not installed. Please install it and try again."
    exit 1
fi

if [ ! -f "$config_file" ]; then
    echo "Error: Configuration file not found: $config_file"
    exit 1
fi

Using Set Options for Stricter Error Checking

Bash provides several options that can help catch errors early. Here are some useful ones:

set -e  # Exit immediately if a command exits with a non-zero status
set -u  # Treat unset variables as an error
set -o pipefail  # Return value of a pipeline is the status of the last command to exit with a non-zero status

# Or combine them:
set -euo pipefail

Implementing Custom Error Handling Functions

Create custom functions to standardize error reporting and handling:

error_exit() {
    echo "ERROR: $1" >&2
    exit 1
}

# Usage
[ -z "$important_var" ] && error_exit "Important variable is not set"

Logging Errors and Debugging Information

Implement a logging system to track errors and debugging information:

log_error() {
    echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') - $1" >> error.log
}

log_debug() {
    if [ "$DEBUG" = true ]; then
        echo "[DEBUG] $(date '+%Y-%m-%d %H:%M:%S') - $1" >> debug.log
    fi
}

# Usage
log_error "Failed to connect to the database"
log_debug "Attempting to reconnect..."

Handling Command-line Arguments Safely

Use getopts to handle command-line arguments safely and provide meaningful error messages for invalid inputs:

while getopts ":a:b:h" opt; do
    case ${opt} in
        a )
            a_arg=$OPTARG
            ;;
        b )
            b_arg=$OPTARG
            ;;
        h )
            echo "Usage: $0 [-a ARG] [-b ARG]"
            exit 0
            ;;
        \? )
            echo "Invalid option: $OPTARG" 1>&2
            exit 1
            ;;
        : )
            echo "Invalid option: $OPTARG requires an argument" 1>&2
            exit 1
            ;;
    esac
done
shift $((OPTIND -1))

Implementing Timeout Mechanisms

For operations that might hang, implement a timeout mechanism:

timeout_cmd() {
    timeout=$1
    shift
    command="$@"
    
    ( $command ) & pid=$!
    ( sleep $timeout && kill -HUP $pid ) 2>/dev/null & watcher=$!
    wait $pid 2>/dev/null && pkill -HUP -P $watcher
}

# Usage
timeout_cmd 5 curl http://example.com || echo "Command timed out after 5 seconds"

Validating Input Data

Always validate input data before processing it:

validate_integer() {
    if ! [[ "$1" =~ ^[0-9]+$ ]]; then
        error_exit "Invalid integer: $1"
    fi
}

# Usage
age=$1
validate_integer "$age"

By implementing these advanced error handling techniques, you can create more robust and reliable Bash scripts. Proper error handling not only helps in debugging and maintaining your scripts but also improves the user experience by providing clear and meaningful error messages. As you develop more complex scripts, make error handling an integral part of your development process to ensure your scripts can gracefully handle unexpected situations.

Bonus Script

So combining both the parameters of text manipulation and error handling – you can create a similar script like:

#!/bin/bash

LOG_FILE="/var/log/syslog"  # Change this to the log file you want to process
OUTPUT_FILE="/var/log/filtered_errors.log"
CLEANED_FILE="/var/log/cleaned_errors.log"

# Check if the log file exists
if [ ! -f "$LOG_FILE" ]; then
    echo "Error: Log file $LOG_FILE not found!" >&2
    exit 1
fi

# Extract error and warning messages, handling any failures
if ! grep -iE "error|warning" "$LOG_FILE" > "$OUTPUT_FILE"; then
    echo "Error: Failed to extract errors from $LOG_FILE" >&2
    exit 1
fi

# Use sed to clean up the log entries (remove timestamps and extra spaces)
sed -E 's/^([A-Za-z]{3} [0-9]{1,2} [0-9]{2}:[0-9]{2}:[0-9]{2}) //' "$OUTPUT_FILE" | sed 's/\s\+/ /g' > "$CLEANED_FILE"

echo "Filtered errors and warnings saved to $OUTPUT_FILE"
echo "Cleaned log entries saved to $CLEANED_FILE"

The script will go through the syslog folder and see if the log file is present or not – if not then it will throw the error. This sums all that we have learned – uses text manipulation to manage the file name and also handles errors.