Advanced Shell Scripting : Final Part

We’ve spoken about how sophisticated shell scripting may benefit you as a systems administrator and how easy it is to manage your everyday activities with only a few lines of code. This will not only increase productivity but also make it easier to scale up as your firm grows. If you haven’t checked out our past posts, please do so here:

Chatter 1: https://pixelhowl.com/advanced-shell-scripting-master-class.

Chapter 2: https://pixelhowl.com/advanced-shell-scripting-part-2-automation-server-management.

So it should give you some fundamental ideas about how to design the script. So going with the same theme we are going to optimize the ssh scripts.

Optimizing Script Performance

As your Bash scripts grow in complexity and handle larger datasets, optimizing performance becomes crucial. In this section, we’ll explore various techniques to enhance the efficiency and speed of your Bash scripts, enabling them to handle more demanding tasks with improved responsiveness.

Minimizing External Command Calls

External command calls can be expensive in terms of performance. Whenever possible, use Bash built-ins or parameter expansions instead of external commands:

# Slower
count=$(echo "$string" | wc -c)

# Faster
count=${#string}

Using Process Substitution

Process substitution can help reduce the number of subshells and temporary files:

# Instead of
sort file1 > sorted1
sort file2 > sorted2
diff sorted1 sorted2

# Use
diff <(sort file1) <(sort file2)

Leveraging Associative Arrays for Lookups

For frequent lookups, associative arrays are more efficient than repeatedly greping through files:

#declare -A user_data
while IFS=: read -r username password uid gid info home shell; do
    user_data["$username"]="$uid:$gid:$home"
done < /etc/passwd

# Lookup
echo "${user_data["root"]}"

Optimizing Loops

When working with loops, consider these optimizations:

1. Use for loops instead of while loops when possible:

# Slower
i=0
while [ $i -lt 1000 ]; do
    # do something
    ((i++))
done

# Faster
for ((i=0; i<1000; i++)); do
    # do something
done

1. Avoid unnecessary subshells in loops:

# Slower
for file in *.txt; do
    (
        # operations in subshell
    )
done

# Faster
for file in *.txt; do
    # operations in current shell
done

Using mapfile for Reading Large Files

For reading large files into an array, mapfile (or readarray) is more efficient than a while loop:

mapfile -t lines < large_file.txt

Parallel Processing with xargs or GNU Parallel

For CPU-intensive tasks, leverage parallel processing:

find . -type f -print0 | xargs -0 -P 4 -I {} gzip {}

This compresses files in parallel using 4 processes.

Optimizing Case Statements

For large case statements, consider using associative arrays instead:

#declare -A actions=(
    [start]="service_start"
    [stop]="service_stop"
    [restart]="service_restart"
)

action=${actions[$1]:-unknown_action}
$action

Using printf Instead of echo

printf is generally faster and more predictable than echo:

printf "Name: %s\nAge: %d\n" "$name" "$age"

Avoiding Unnecessary Pipe Usage

Pipes create subshells, which can be expensive. When possible, use parameter expansion or command substitution instead:

# Slower
echo "$var" | tr '[:lower:]' '[:upper:]'

# Faster
echo "${var^^}"

Profiling Your Script

Use the time command to measure the execution time of your script or specific commands:

time ./your_script.sh

For more detailed profiling, you can use tools like strace or bash -x:

bash -x ./your_script.sh

Caching Expensive Operations

If your script performs expensive operations repeatedly, consider caching the results:

get_data() {
    if [ -z "$cached_data" ]; then
        cached_data=$(expensive_operation)
    fi
    echo "$cached_data"
}

Using Memory-Efficient Text Processing

For very large files, use stream editors like sed or awk instead of loading the entire file into memory:

sed 's/pattern/replacement/g' large_file.txt > modified_file.txt

By implementing these optimization techniques, you can significantly improve the performance of your Bash scripts, especially when dealing with large datasets or complex operations. Remember to profile your script before and after optimizations to measure the impact of your changes. Keep in mind that readability and maintainability are also important factors, so strive for a balance between performance and code clarity.

Automating System Administration Tasks

Bash scripting is a powerful tool for automating various system administration tasks, allowing you to streamline repetitive processes, manage system resources efficiently, and maintain consistent configurations across multiple systems. In this section, we’ll explore advanced techniques for automating common system administration tasks using Bash.

Automated Backup and Restore

Create a comprehensive backup script that archives important directories and databases:

#!/bin/bash

backup_dir="/mnt/backups"
date_stamp=$(date +"%Y-%m-%d_%H-%M-%S")
backup_file="$backup_dir/backup_$date_stamp.tar.gz"

# Backup important directories
tar -czf "$backup_file" /etc /home /var/www

# Backup MySQL databases
mysqldump --all-databases | gzip >> "$backup_file"

# Rotate old backups (keep last 7 days)
find "$backup_dir" -type f -mtime +7 -delete

echo "Backup completed: $backup_file"

System Monitoring and Alerting

Develop a script to monitor system resources and send alerts when thresholds are exceeded:

#!/bin/bash

check_cpu_usage() {
    cpu_usage=$(top -bn1 | grep "Cpu(s)" | awk '{print $2 + $4}')
    if (( $(echo "$cpu_usage > 90" | bc -l) )); then
        send_alert "High CPU usage: $cpu_usage%"
    fi
}

check_disk_space() {
    disk_usage=$(df -h / | awk '/\// {print $5}' | sed 's/%//')
    if [ "$disk_usage" -gt 90 ]; then
        send_alert "Low disk space: $disk_usage% used"
    fi
}

send_alert() {
    echo "$1" | mail -s "System Alert" [email protected]
}

check_cpu_usage
check_disk_space

Automated User Management

Create a script to automate user account creation and management:

#!/bin/bash

create_user() {
    username=$1
    password=$2
    
    # Check if user already exists
    if id "$username" &>/dev/null; then
        echo "User $username already exists"
        return 1
    fi
    
    # Create user and set password
    useradd -m -s /bin/bash "$username"
    echo "$username:$password" | chpasswd
    
    # Add user to necessary groups
    usermod -aG sudo,developers "$username"
    
    echo "User $username created successfully"
}

delete_user() {
    username=$1
    
    # Check if user exists
    if ! id "$username" &>/dev/null; then
        echo "User $username does not exist"
        return 1
    fi
    
    # Delete user and their home directory
    userdel -r "$username"
    
    echo "User $username deleted successfully"
}

# Usage
create_user "newuser" "password123"
delete_user "olduser"

Automated Software Updates

Develop a script to keep your system up-to-date automatically:

#!/bin/bash

log_file="/var/log/auto_update.log"

update_system() {
    echo "$(date): Starting system update" >> "$log_file"
    
    # Update package lists
    apt-get update >> "$log_file" 2>&1
    
    # Upgrade packages
    apt-get -y upgrade >> "$log_file" 2>&1
    
    # Remove unnecessary packages
    apt-get -y autoremove >> "$log_file" 2>&1
    
    echo "$(date): System update completed" >> "$log_file"
}

# Check if script is run as root
if [ "$(id -u)" != "0" ]; then
    echo "This script must be run as root" 1>&2
    exit 1
fi

update_system

Log Rotation and Cleanup

Implement a script to manage log files, rotating and compressing old logs:

#!/bin/bash

log_dir="/var/log"
max_age=30 # days

# Compress logs older than 7 days
find "$log_dir" -type f -name "*.log" -mtime +7 -exec gzip {} \;

# Delete logs older than max_age
find "$log_dir" -type f -name "*.log.gz" -mtime +"$max_age" -delete

# Rotate current logs
for log in "$log_dir"/*.log; do
    mv "$log" "${log}.1"
    touch "$log"
done

Network Configuration Management

Create a script to manage network configurations across multiple servers:

#!/bin/bash

update_network_config() {
    local server=$1
    local ip=$2
    local gateway=$3
    
    ssh "$server" "
        sudo sed -i 's/^address.*/address $ip/' /etc/network/interfaces
        sudo sed -i 's/^gateway.*/gateway $gateway/' /etc/network/interfaces
        sudo systemctl restart networking
    "
    
    echo "Updated network configuration for $server"
}

# Read server configurations from a file
while IFS=, read -r server ip gateway; do
    update_network_config "$server" "$ip" "$gateway"
done < server_network_config.csv

Automated Database Maintenance

Develop a script to perform regular database maintenance tasks:

#!/bin/bash

db_user="your_username"
db_pass="your_password"

# Optimize all tables in all databases
mysql -u "$db_user" -p"$db_pass" -e "
    SELECT CONCAT('OPTIMIZE TABLE ', table_schema, '.', table_name, ';')
    FROM information_schema.tables
    WHERE table_schema NOT IN ('information_schema', 'performance_schema', 'mysql')
" | mysql -u "$db_user" -p"$db_pass"

# Analyze all tables
mysql -u "$db_user" -p"$db_pass" -e "
    SELECT CONCAT('ANALYZE TABLE ', table_schema, '.', table_name, ';')
    FROM information_schema.tables
    WHERE table_schema NOT IN ('information_schema', 'performance_schema', 'mysql')
" | mysql -u "$db_user" -p"$db_pass"

echo "Database maintenance completed"

By leveraging these advanced Bash scripting techniques for system administration tasks, you can significantly improve the efficiency and consistency of your IT operations. These scripts serve as a foundation that you can build upon and customize to suit your specific needs. Remember to thoroughly test your scripts in a safe environment before deploying them in production, and always include proper error handling and logging to ensure smooth operation and easy troubleshooting.

Securing Your Bash Scripts

As you develop more advanced Bash scripts, especially those used for system administration or handling sensitive data, it’s crucial to implement robust security measures. Securing your scripts helps protect against unauthorized access, data breaches, and potential system compromises. In this section, we’ll explore various techniques to enhance the security of your Bash scripts.

Implementing Proper File Permissions

Ensure that your scripts have appropriate permissions to prevent unauthorized access or modification:

# Set script to be executable only by owner
chmod 700 myscript.sh

# If the script needs to be run by others, use sudo instead of making it world-executable
sudo ./myscript.sh

Using Secure Temporary Files

When creating temporary files, use mktemp to generate unique, unpredictable filenames:

temp_file=$(mktemp /tmp/myapp.XXXXXX)
trap 'rm -f "$temp_file"' EXIT

Providing you with a detailed set of tutorials for systems administration and linux administration with real life examples and free scripts.

Hope you enjoyed this series, we do have much more coming up around the academy so stay subscribed and please do feel free to share and like this post! It will mean a lot to us!