Text Processing Utilities

# CHAPTER 11

Text Processing Utilities #

1. Introduction #

2. Learning Objectives #

• Extract specific rows of text from massive files using grep.

• Slice specific columns out of explicitly delimited files using cut.

• Perform automated, inline find-and-replace text modifications using sed.

• Extract and format complex tabular whitespace data using awk.

• Translate and manipulate characters using tr.

• Chain sort and uniq together to count duplicate log entries.

3. The Finder (grep) #

# Print all lines in the auth.log that contain the word "Failed"
grep "Failed" /var/log/auth.log

# Pro-Tip: Add -v to INVERT the search (Show everything EXCEPT "Failed")
grep -v "Failed" /var/log/auth.log

4. The Slicer (cut) #

# Print ONLY the usernames (Column 1) from the password file.
# The delimiter is explicitly set to a colon (:).
cat /etc/passwd | cut -d ':' -f 1

5. The Replacer (sed) #

# Pipe text into sed to change "apples" to "oranges"
echo "I like apples" | sed 's/apples/oranges/g'
# Output: I like oranges

*Pro-Tip (In-place Editing):* Normally sed just prints to the screen. If you use -i (or -i '' on macOS), sed will physically edit the target text file and save it permanently without opening it!

# Permanently change Port 22 to 2222 in the SSH configuration
sudo sed -i 's/Port 22/Port 2222/g' /etc/ssh/sshd_config

6. The Engine (awk) #

# Print ONLY the file size (Column 5) and the file name (Column 9)
ls -l | awk '{print $5, $9}'

7. The Organizer (sort and tr) #

# Convert all lowercase text to UPPERCASE
echo "hello world" | tr 'a-z' 'A-Z'

2. sort and uniq: If you extract 5,000 IP addresses from a log file, you will likely have hundreds of duplicates. You use sort to organize them alphabetically, and uniq -c to delete the duplicates while counting them!

# Read IPs, sort them, count duplicates, and sort by highest count (Numeric Reverse)
cat ip_list.txt | sort | uniq -c | sort -nr

8. Diagrams/Visual Suggestions #

9. Best Practices #

• Never sed -i without a backup: The in-place stream editor is ruthlessly destructive. If you make a typo in your search pattern, it will permanently corrupt the file it is editing. Always back up configuration files (cp config config.bak) before executing an automated sed -i command inside a script.

10. Common Mistakes #

• Using uniq without sort: The uniq command is mechanically "dumb". It only deletes duplicates if they are sitting *exactly adjacent* to each other vertically. If "Admin" is on line 1, and "Admin" is on line 5, uniq will not delete the duplicate. You MUST pipe the data through sort first to group identical lines together, and then pipe it into uniq.

11. Mini Project: Build an Access Log Analyzer #

1. 1. nano log_analyzer.sh

1. 2. Write the code:

#!/bin/sh

# Pretend we have an apache access.log file
LOG_FILE="/var/log/apache2/access.log"

if [ ! -f "$LOG_FILE" ]; then
    echo "No log file found. Skipping analysis."
    exit 1
fi

echo "Top 5 IP Addresses hitting the server:"
echo "--------------------------------------"

# The Pipeline Breakdown:
# 1. awk '{print $1}': Extract the very first column (IP addresses)
# 2. sort: Group identical IPs together
# 3. uniq -c: Count the duplicates and collapse them into single lines
# 4. sort -nr: Sort them Numerically (n) in Reverse (r) so the biggest number is at top
# 5. head -n 5: Only display the top 5 lines

cat "$LOG_FILE" | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 5

1. 3. This is a real-world, production-grade forensic data pipeline.

12. Practice Exercises #

1. 1. Differentiate the operational utility of the cut command versus the awk command. In a file where columns are separated by irregular amounts of whitespace (tabs and spaces mixed), which tool is superior?

1. 2. Explain the mandatory workflow requirement regarding the execution order of the sort and uniq commands.

13. MCQs with Answers #

14. Interview Questions #

• Q: A junior engineer runs cat access.log | uniq -c to count duplicate IP addresses, but the terminal output still shows hundreds of duplicated IPs scattered throughout the list. Explain the mechanical limitation of the uniq command that caused this failure, and provide the corrected command pipeline.

• Q: Explain the syntax awk '{print $3}'. What is this command extracting, and how does awk fundamentally determine where data boundaries exist by default?

• Q: You are writing an automated deployment script. You need to permanently change a database connection string inside a config.php file without opening a text editor. Walk me through the exact sed command syntax required to execute this modification securely.

15. FAQs #

16. Summary #

17. Next Chapter Recommendation #