GAWK 1

Gnu Awk

Gawk is the GNU version of the text processing tool. In most GNU/Linux distributions, Gawk is the default awk implementation, so there is usually no difference in daily use.

readlink -f /usr/bin/awk

/usr/bin/gawk

The Gawk command defaults to using ERE mode.

Basic Syntax

gawk [OPTIONS] program file

• OPTIONS: command options.
• program: there is a program in this damn command.
• file: the file being processed, if omitted, read from STDIN.

Omitting file enters interactive mode, where one line is executed at a time.

Execution Process

• Read a line of data:
  • If there is a matching rule:
    • If the match is successful: perform the corresponding operation.
    • If the match fails: do not perform the operation.
  • If there is no matching rule: perform the corresponding operation.

Basic Usage

Create the foo file.

echo -e 'aa 11\nbb 22' > foo

For each line of data, Gawk defaults to using space/tab to separate fields.

• $N: represents the Nth field.
• $0: represents the entire line of data.

gawk '{print $1}' foo

aa
bb

BEGIN/END Structure

• BEGIN: initialization, executed before interpretation.
• BODY: executed once for each record.
• END: end of processing.

Note the use of single quotes 'EOF' to create the file, so that special characters $ are not processed.

cat <<'EOF' > foo.gawk
BEGIN {
    FS=":"
    print "User\tShell"
    print "-------\t-------"
}
{
    print $1 "\t" $6
}
END {
    print "-------\t-------"
}
EOF

head -n 3 /etc/passwd | gawk -f foo.gawk

User    Shell
------- -------
root    /root
daemon  /usr/sbin
bin     /bin
------- -------

Common Options

Specify Separator

The -F option can modify the line separator.

gawk -F: '{print $1}' /etc/passwd | head -n 1

Specify File

The -f option can specify a file.

echo '{print $1 "-dir:" $6}' > foo.gawk

gawk -F: -f foo.gawk /etc/passwd | head -n 1

root-dir:/root

Assign Variable Parameters

The -v option can assign values to variables before BEGIN.

gawk -v n=2 'BEGIN{print 2*n}'

4

If you don't need to use it in BEGIN, you can omit the -v parameter.

echo 'a b c' | gawk '{print $n}' n=2

b

Built-in Variables

Variable $N

$N can also be assigned, and double quotes for strings cannot be omitted.

echo 'hey man' | gawk '{$2="bro"; print $0}'

hey bro

Variable FS

Field Separator, field separator.

gawk 'BEGIN{FS=":"} {print $1}' /etc/passwd | head -n 1

Variable NF

Number of Fields, represents the number of fields in the record.

gawk -F: '$1=="root"{print $1":"$NF}' /etc/passwd

root:/bin/bash

Variable NR

Number of Records, represents the current record number being processed, the default value is 1, and 1 is added after processing each line. Can be used to skip the first line of text, the NR value of the first line is 1.

cat <<EOF > foo
name score
foo  90
bar  80
EOF

gawk '{if (NR>1) {if ($2>85) {print $1,$2}}' foo

foo 90

Variable RS

Record Separator, input record separator, the default value is \n, which means that each record is separated by a newline. Setting RS to "" means that an empty line is used as the record separator. For the following text, it will be divided into two records, upper and lower.

cat <<EOF > foo
apple
sweet
red

banana
sweet
yellow
EOF

Set FS="\n", then you can get each line of record through $N. RS and FS are usually used together.

gawk 'BEGIN{RS=""; FS="\n"} {print $1"\t"$3}' foo

apple  red
banana yellow

Variable OFS

Output Field Separator, output field separator.

echo 'aa,bb' | gawk 'BEGIN{FS=","; OFS="-"} {print $1,$2}'

aa-bb

Variable FIELDWIDTHS

Specify character width for separation.

echo 'abbc' | gawk 'BEGIN{FIELDWIDTHS="1 2 1"} {print $1,$2,$3}'

a bb c

Conditional and Structure

Conditional Expression

==, <, <=, >, >=.

gawk -F: '$7=="/bin/bash"{print $1}' /etc/passwd

Output all users who start with bash.

Conditional Statement

A single statement inside if does not need {}.

echo -e '10\n20' | gawk '{if ($1>15) print $1}'

Multiple statements inside if need {}.

echo -e '10\n20' | gawk '{if ($1>15) {x=2*$i; print x}'

For a single line else statement, the previous statement needs a ;.

echo -e '10\n20' | gawk '{if ($1>15) print $1; else print "no"}'

Multiple lines do not need a semicolon.

echo -e '10\n20' | gawk '{
if ($i>15) {
    x=2*$i
    print x
} else {
    print "no"
}
}'

FOR Statement

Calculate the sum of each field for each line, both += and ++ are supported.

echo '1 2 3' | gawk '{
total=0
for (i=1; i<=NF; i++) {
    total += $i
}
print total
}'

WHILE Statement

Calculate the sum of each field for each line.

echo '1 2 3' | gawk '{
i=1
total=0
while (i<=NF) {
    total += $i
    i++
}
print total
}'

DO-WHILE Statement

Calculate the sum of each field for each line

echo '1 2 3' | gawk '{
i=1
total=0
do {
    total += $i
    i++
} while(i<=NF)
print total
}'

Function Related

Built-in Functions

• int(x): take the integer part of x.
• exp(x): x to the power.
• sqrt(x): square root of x.
• rand(): a random number greater than 0 and less than 1.
• length(x): length of string x.
• tolower(x): convert x to lowercase.
• toupper(x): convert x to uppercase.

There are many more, such as gensub, gsub.

Custom Functions

Custom functions must appear before BEGIN block.

gawk '
function random(ts, num) {
    srand(ts)
    return int(num * rand())
}
BEGIN {
    ts=systime()
    print ts
    print random(ts, 10)
}'

You can use function library files and then reference them.

cat <<'EOF' > funclib.gawk
function random(ts, num) {
    srand(ts)
    return int(num * rand())
}
EOF

The gawk program file is as follows.

cat <<'EOF' > test.gawk
BEGIN {
    ts=systime()
    print ts
    print random(ts, 10)
}
EOF

Use the -f option to reference two files.

gawk -f funclib.gawk -f test.gawk

You cannot use inline program mode when referencing function libraries, you need to reference both.

Other Examples

Custom Variables

Support mathematical operations and floating point numbers, not stronger than bash 🤪.

gawk 'BEGIN{a=2; a=a*2/3; print a}'

1.33333

Array Operations

Features: associative arrays, similar to dictionaries, unordered.

gawk 'BEGIN{arr["name"]="foo"; print arr["name"]}'

You can use numeric subscripts, which are actually dictionaries.

gawk 'BEGIN{arr[3]="foo"; print arr[3]}'

Traverse the array, delete elements.

gawk 'BEGIN{
arr["a"]=1
arr[2]=2
arr["c"]="cat"
delete arr[2]
for (k in arr) {
    print "key:",k," val:", arr[k]
}
}
'

key: a  val: 1
key: c  val: cat

Formatted Printing

Processing floating point numbers.

gawk 'BEGIN{printf "%.2f\n", 2/3}'

0.67

Specify width.

echo -e 'foo\nfoobar' | gawk '{printf "%8s\n", $1}'

Left alignment.

echo -e 'foo\nfoobar' | gawk '{printf "%-8s\n", $1}'