Use gawk to Filter Logs and Calculate Field Totals

You will finish with a repeatable way to select records, print chosen fields, and calculate a total with gawk. The examples use the installed GNU Awk 5.2.1 package, and read input without changing it.

Allow about fifteen minutes. You need a shell and a readable text file with one record per line. No elevated privileges are needed. If the source file contains confidential data, treat command output as sensitive too: do not paste it into a ticket or terminal transcript without checking it first.

1. Check the installed interpreter

Confirm which executable and package version you are using. This is an ordinary, read-only check:

$ command -v gawk
/usr/bin/gawk
$ dpkg-query -W -f='${Package} ${Version}\n' gawk
gawk 1:5.2.1-2ubuntu0.1
$ gawk --version | head -1
GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)

Your package revision can differ. The guide's syntax is based on the gawk 5.2.1 interface installed here. awk and nawk are compatible command names on many systems, but do not assume they select GNU Awk: check each command with command -v and its version output when an extension matters.

2. Read fields and select matching records

Awk reads input a record at a time. By default, a record is a line and its fields are separated by runs of whitespace. The first field is $1, the second is $2, and $0 is the complete record.

For a log whose first field is a level, print only errors and retain the status code and path:

$ printf '%s\n' \
  'INFO 200 /api' \
  'ERROR 500 /login' \
  'INFO 204 /health' |
  gawk '$1 == "ERROR" { print $2, $3 }'
500 /login

The expression before the braces is a pattern. The statements inside the braces are the action taken for matching records. A record that does not match is ignored. The comma in print $2, $3 uses the output field separator, which is a space by default.

3. Set a delimiter explicitly

Do not rely on whitespace rules for CSV or other structured text. The -F option sets the input field separator. Set OFS inside the program when you want a deliberate separator in output:

$ printf '%s\n' 'name,count,region' 'alpha,12,north' 'beta,7,south' |
  gawk -F, -v OFS=' | ' '{ print $1, $3, $2 }'
name | region | count
alpha | north | 12
beta | south | 7

-v assigns an awk variable before processing starts. Here it sets OFS to the three-character string containing spaces and a vertical bar. The separator is not removed from quoted CSV fields, so this simple form is suitable only for data whose fields do not contain embedded commas or quoting rules. Use a CSV-aware tool for full CSV syntax.

4. Calculate totals and averages

Awk variables start with a numeric value of zero when used arithmetically. Use an action for each data row and an END action for the report:

$ printf '%s\n' 'alpha 12' 'beta 7' 'alpha 3' |
  gawk '{ total += $2; rows++ }
         END { printf "rows=%d total=%d average=%.1f\n", rows, total, total / rows }'
rows=3 total=22 average=7.3

The END block runs after the last input record. It will divide by zero when the input is empty, so guard that case in a reusable command:

$ gawk '{ total += $2; rows++ }
         END { if (rows) printf "rows=%d total=%d average=%.1f\n", rows, total, total / rows; else print "no data" }' data.txt

Check that the numeric field really contains numbers. A malformed value can produce a result that looks plausible while representing the wrong calculation. For money, avoid treating binary floating-point output as an accounting ledger; sum integer minor units or use a decimal-aware tool when exact decimal arithmetic is required.

5. Pass a threshold without editing the program

Use -v for a value that changes between runs. This keeps the awk program readable and avoids constructing source code from shell input:

$ LIMIT=400
$ gawk -v limit="$LIMIT" '$2 >= limit { print $1, $2 }' access.log
ERROR 500
WARN 404

Replace access.log with your file and adjust the field numbers to its format. The shell expands LIMIT before gawk starts; gawk receives the value as the numeric variable limit. Quote the assignment value, especially when it may contain spaces or shell metacharacters.

$ gawk -v limit="$LIMIT" '$2 >= limit { print $1, $2 }' access.log > access.log.filtered
$ wc -l access.log access.log.filtered
$ cmp --silent access.log access.log.filtered; printf 'cmp status: %s\n' "$?"

A non-zero cmp status is expected when the filter removed records, not an indication that either file is damaged. If you later decide to replace the old file, make a backup first and use a temporary destination on the same filesystem. There is no undo for a truncated file unless you have a backup or another original copy.

6. Move a working program into a script

Once a one-liner is stable, put the program in a file so it can be reviewed and rerun. This example expects whitespace-separated records with a status in field two:

$ cat > report.awk <<'AWK'
BEGIN { OFS = " | " }
$2 >= limit { print $1, $2 }
AWK
$ gawk -v limit=400 -f report.awk access.log
ERROR | 500
WARN | 404

The quoted here-document marker prevents the shell from expanding awk variables while the file is created. If the report is wrong, inspect the script and run it against a small sample. Remove the temporary script with rm report.awk only when you are sure it is no longer useful; that deletion is irreversible.