sed edits a stream line by line; awk splits each line into fields and runs a program against them. The split in practice: reach for sed when you are rewriting text, and awk when you care about columns or need to accumulate a total. Both ship on every Unix system, which is why they survive — but the GNU versions on Linux and the BSD versions on macOS disagree in ways that silently corrupt files, and that section is the one worth reading even if you skip the rest.
When to reach for this
- Rewriting a config value across many files in place, without opening an editor —
sed -i. - Pulling one column out of whitespace- or comma-separated output and summing or counting it —
awk. - A script works on your Linux CI runner and mangles files on a colleague's Mac — the portability section below.
How Do You Substitute Text With sed?
| Command | Description |
|---|---|
| sed 's/old/new/' f | Replace the first match on each line |
| sed 's/old/new/g' f | Replace every match on each line |
| sed 's/old/new/2' f | Replace only the second match on each line |
| sed 's/old/new/gI' f | Global and case-insensitive (GNU; BSD uses I too) |
| sed 's|/usr|/opt|' f | Use | as the delimiter when the text contains slashes |
| sed 's/.*/[&]/' f | Wrap the whole match — & is the matched text |
| sed -E 's/(a+)(b+)/\2\1/' f | Swap capture groups; -E enables extended regex |
| sed 's/x/y/w out.txt' f | Write only the changed lines to out.txt |
& in the replacement is the entire match, and \1 through \9 are capture groups. Use -E rather than -r for extended regex: -E works on both GNU and BSD, -r is GNU-only.
How Do You Target Specific Lines?
| Address | Selects |
|---|---|
| sed -n 5p f | Line 5 only (-n suppresses the default print) |
| sed -n 5,10p f | Lines 5 through 10 |
| sed -n 5,$p f | Line 5 to end of file |
| sed -n 0~3p f | Every third line (GNU only) |
| sed '/start/,/end/d' f | Delete from the first match of start through end |
| sed '/^#/d' f | Delete comment lines |
| sed '/^$/d' f | Delete blank lines |
| sed '5!d' f | Delete everything except line 5 |
| sed '$d' f | Delete the last line |
| sed '2i\text' f | Insert a line before line 2 |
| sed '2a\text' f | Append a line after line 2 |
| sed '2c\text' f | Replace line 2 entirely |
sed -n '/start/,/end/p' is the quickest way to pull one stanza out of a long config or a single failing test out of CI output. Note that a range restarts: if start matches again after end, you get a second block.
What Does an awk Program Look Like?
An awk program is a list of pattern { action } pairs. Omit the pattern and the action runs on every line; omit the action and matching lines are printed.
| Command | Description |
|---|---|
| awk '{print $1}' f | Print the first whitespace-separated field |
| awk '{print $NF}' f | Print the last field |
| awk '{print $(NF-1)}' f | Print the second-to-last field |
| awk -F, '{print $2}' f | Split on commas instead of whitespace |
| awk -F'\t' '{print $3}' f | Split on tabs |
| awk 'NR==5' f | Print line 5 |
| awk 'NR>1' f | Skip a header row |
| awk '/error/ {print $0}' f | Print lines matching a pattern |
| awk '$3 > 100' f | Print rows where field 3 exceeds 100 |
| awk 'NF' f | Drop blank lines (NF is 0 on an empty line) |
| awk '!seen[$0]++' f | Deduplicate without sorting, keeping first occurrence |
| awk -v n=5 '$1 > n' f | Pass a shell value in with -v |
Built-in Variables
| Variable | Meaning |
|---|---|
| NR | Current record number, counting across all input files |
| FNR | Record number within the current file |
| NF | Number of fields on the current line |
| FS | Input field separator (same as -F) |
| OFS | Output field separator used by print with commas |
| RS | Input record separator; set to "" for paragraph mode |
| ORS | Output record separator, newline by default |
| FILENAME | Name of the file currently being read |
OFS catches people out: changing it does nothing until a field is reassigned. The idiom is awk -v OFS=, '{$1=$1; print}' — the no-op assignment forces awk to rebuild the record with the new separator.
BEGIN, END, and Aggregation
| Command | Description |
|---|---|
| awk '{s+=$1} END {print s}' f | Sum a column |
| awk '{s+=$1} END {print s/NR}' f | Average a column |
| awk 'END {print NR}' f | Count lines, like wc -l |
| awk '{c[$1]++} END {for (k in c) print c[k], k}' f | Count occurrences per key |
| awk 'BEGIN {FS=":"} {print $1}' /etc/passwd | Set the separator in BEGIN instead of -F |
| awk '{print > $1".txt"}' f | Split input into files named by field 1 |
| awk 'BEGIN {print "start"} END {print "done"}' f | Run once before and once after all input |
The {c[$1]++} END {for (k in c) ...} pattern replaces sort | uniq -c | sort -rn and does it in a single pass, which matters once the input stops fitting in memory comfortably. Associative arrays are the reason awk outlived its replacements.
Which GNU vs BSD Differences Actually Bite?
macOS ships BSD sed and BSD awk; Linux ships GNU sed and gawk or mawk. The in-place edit flag is where scripts break, and it fails destructively rather than loudly.
| Form | GNU (Linux) | BSD (macOS) |
|---|---|---|
| sed -i 's/a/b/' f | Edits in place, no backup | Treats the script as the backup suffix — breaks |
| sed -i '' 's/a/b/' f | Treats '' as the suffix — breaks | Edits in place, no backup |
| sed -i.bak 's/a/b/' f | Edits in place, keeps f.bak | Edits in place, keeps f.bak |
| sed -E | Extended regex | Extended regex |
| sed -r | Extended regex | Not supported |
| \b word boundary | Supported | Not supported |
Run the Linux form on a Mac and BSD sed reads 's/a/b/' as the required backup extension, then tries to parse your filename as the script, giving the famously unhelpful sed: 1: "f": invalid command code f. The variant sed -i -e 's/a/b/' f is worse: BSD takes -e as the suffix and leaves a stray f-e file behind while appearing to work.
The portable idiom is sed -i.bak 's/a/b/' f && rm f.bak. Both implementations parse a glued suffix identically, and the && keeps the backup if sed failed and left the original half-rewritten. If you would rather not think about it, install GNU sed on macOS with brew install gnu-sed and call gsed.
What Are the Common Gotchas?
| Symptom | Cause | Fix |
|---|---|---|
| sed: invalid command code | BSD sed consumed the script as the -i backup suffix | Use sed -i.bak ... && rm file.bak |
| Substitution silently does nothing | The pattern contains an unescaped delimiter, or is a basic-regex construct needing -E | Change the delimiter (s|a|b|) or add -E |
| awk prints nothing for a valid file | The field separator is wrong, so $2 does not exist | Check with awk "{print NF}" before indexing fields |
| awk arithmetic gives 0 | The field has stray whitespace or a currency symbol, so it coerces to 0 | Strip it first: gsub(/[^0-9.]/, "", $1) |
| OFS has no effect on output | awk only rebuilds the record when a field is assigned | Add a no-op assignment: {$1=$1; print} |
| sed -i destroys a symlink | In-place edit writes a new file and replaces the link | Resolve first with readlink -f, or edit the target directly |
For the regex syntax both tools share, see the regex cheat sheet. When the input is JSON rather than lines and columns, stop reaching for these and use jq — parsing JSON with sed is the canonical way to produce a subtly wrong pipeline. For wiring these into scripts, the Bash scripting cheat sheet covers quoting and exit codes.