Text Processing: awk, sed, cut

After this lesson, you will be able to: Use awk, sed, and cut to extract, transform, and reformat text, the foundation of every Unix log-analysis workflow.

Logs come in lines; analysis happens with text tools. awk + sed + cut are the three you'll reach for daily.

Prerequisites:Package Management

cut: extract columns

Fastest way to pull a column out of structured text.

tsx

# Extract usernames from /etc/passwd (colon-separated)
cut -d: -f1 /etc/passwd
# -d sets delimiter, -f selects field(s)

cut -d: -f1,3 /etc/passwd       # multiple fields
cut -d',' -f2- data.csv          # field 2 onward
cut -c1-10 file.txt              # first 10 characters per line

awk: column-aware mini-language

awk is a full programming language for text. Most people use 5% of it; that's enough.

tsx

# Print the 7th column (the user name) from ps aux output
ps aux | awk '{print $7}'

# Field separator (default is whitespace)
awk -F: '{print $1, $3}' /etc/passwd

# Conditional: lines where column 3 > 1000
awk -F: '$3 > 1000 {print $1}' /etc/passwd

# Sum column 5
awk '{sum += $5} END {print sum}' /var/log/access.log

# Count unique values in column 7
awk '{print $7}' access.log | sort | uniq -c | sort -rn | head

# NR = current line number; NF = number of fields on current line
awk 'NR==10' file.txt          # 10th line
awk 'NF > 5' file.txt          # lines with more than 5 fields

sed: stream editor for substitutions

sed is what you reach for 'I need to replace X with Y across this file/stream'.

tsx

# Replace 'foo' with 'bar' (first occurrence per line)
sed 's/foo/bar/' file.txt

# Replace ALL occurrences (g = global)
sed 's/foo/bar/g' file.txt

# In-place edit (BSD vs GNU difference: BSD/macOS needs `sed -i ''`)
sed -i 's/foo/bar/g' file.txt

# Delete lines matching a pattern
sed '/^#/d' config.ini             # delete comment lines
sed '/^$/d' file.txt               # delete blank lines

# Insert / append / change a line
sed '5a\\nNew line after line 5' file.txt
sed '1i\\nFirst line inserted' file.txt

# Multiple expressions
sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

When to reach for jq / yq / ripgrep instead

JSON: use `jq`. `cat data.json | jq '.users[].name'`. Trying to parse JSON with awk / sed is a rite of passage and a mistake. YAML: use `yq` (same syntax as jq, but for YAML). Code search across a tree: use `ripgrep` (rg). Respects .gitignore by default; way faster than `grep -r`. Knowing when to put awk/sed away and switch to a structured tool is a senior-engineer move.

Common mistakes only experienced ops engineers avoid

Writing 20-line awk programs instead of using Python. If it's complex, escape to Python. Using sed on JSON. Always wrong; use jq. Forgetting that sed -i differs between BSD/macOS and GNU/Linux. Scripts that work on Mac break on Linux (and vice versa). Not piping `head` after exploratory commands. Don't dump a million-line file to the terminal. Conflating awk's `==` (compare) with `=` (assign).

Quick Check

What does this do? `awk '$9 == 500' /var/log/nginx/access.log | wc -l`

Pick the cleanest explanation.

←Package Management

Back to Linux and the Command Line

vim Basics→