Text Processing Commands

# CHAPTER 11

Text Processing Commands #

1. Introduction #

2. Learning Objectives #

• Extract specific rows of text using grep and Regular Expressions.

• Slice specific columns of text out of delimited files using cut.

• Perform automated, inline find-and-replace text modifications using sed.

• Extract and format complex tabular data using awk.

• Chain sort and uniq together to count duplicate entries in log files.

3. The Slicer (cut) #

# Print ONLY the usernames (Column 1) from the password file.
# The delimiter is a colon (:).
cat /etc/passwd | cut -d ':' -f 1

4. The Replacer (sed) #

# Echo a sentence, and pipe it into sed to change "apples" to "oranges"
echo "I like apples" | sed 's/apples/oranges/g'
# Output: I like oranges

*Pro-Tip: Using -i (In-place)*: Normally sed just prints to the screen. If you add -i, sed will physically edit and save the text file permanently.

# Permanently change Port 22 to 2222 in the SSH config
sudo sed -i 's/Port 22/Port 2222/g' /etc/ssh/sshd_config

5. The Engine (awk) #

# Print ONLY the file size (Column 5) and the file name (Column 9)
ls -l | awk '{print $5, $9}'

6. The Organizer (sort and uniq) #

# Read a file of messy IPs, sort them, count the duplicates, and sort by the highest count
cat ip_list.txt | sort | uniq -c | sort -nr
# Output: 450 192.168.1.5 (This IP attacked you 450 times!)

7. Diagrams/Visual Suggestions #

8. Best Practices #

• Never sed -i without a backup: The sed -i command is ruthlessly destructive. If you make a typo in your Regex, it will permanently corrupt the file it is editing. Always back up configuration files (cp config config.bak) before running a sed -i command in a script.

9. Common Mistakes #

• Using uniq without sort: The uniq command is notoriously "dumb". It only deletes duplicates if they are sitting *exactly next to each other* vertically. If "Apple" is on line 1, and "Apple" is on line 5, uniq will not delete the duplicate. You MUST pipe the data through sort first to group the duplicates together, and then pipe it into uniq.

10. Mini Project: Build an Access Log Analyzer #

1. 1. nano log_analyzer.sh

1. 2. Write the code:

#!/bin/bash

# Pretend we have an apache access.log file
LOG_FILE="/var/log/apache2/access.log"

if [ ! -f "$LOG_FILE" ]; then
    echo "No log file found."
    exit 1
fi

echo "Top 5 IP Addresses hitting the server:"
echo "--------------------------------------"

# 1. Read the file
# 2. awk '{print $1}': Extract the very first column (which is the IP address in Apache logs)
# 3. sort: Group the identical IPs together
# 4. uniq -c: Count the duplicates and collapse them into single lines
# 5. sort -nr: Sort them Numerically (n) in Reverse (r) so the biggest number is at the top
# 6. head -n 5: Only show the top 5 lines

cat "$LOG_FILE" | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 5

1. 3. This is a real-world, production-grade forensic pipeline.

11. Practice Exercises #

1. 1. Differentiate the operational utility of the cut command versus the awk command. In a file where columns are separated by irregular amounts of whitespace (tabs and spaces), which tool is superior?

1. 2. Explain the mandatory workflow requirement regarding the execution order of the sort and uniq commands.

12. MCQs with Answers #

13. Interview Questions #

• Q: A junior engineer runs cat access.log | uniq -c to count duplicate IP addresses, but the output still shows hundreds of duplicated IPs scattered throughout the list. Explain the mechanical limitation of the uniq command that caused this failure, and provide the corrected command pipeline.

• Q: Explain the syntax awk '{print $3}'. What is this command extracting, and how does awk fundamentally determine where data boundaries exist by default?

• Q: You are writing an automated deployment script. You need to permanently change a database connection string inside a config.php file without opening a text editor. Walk me through the exact sed command required to execute this modification safely.

14. FAQs #

15. Summary #

16. Next Chapter Recommendation #