Awk --- A Pattern Scanning And Processing Language (Second Edition)

Awk | A Pattern Scanning and Processing Language
(Second Edition)
Alfred V. Aho
Brian W. Kernighan
Peter J. Weinberger
ABSTRACT
Awk is a programming language whose basic operation is to search a set of
les for patterns, and to perform speci ed actions upon lines or elds of lines
which contain instances of those patterns. Awk makes certain data selection and
transformation operations easy to express; for example, the awk program
length > 72
prints all input lines whose length exceeds 72 characters; the program
NF % 2 == 0
prints all lines with an even number of elds; and the program
f $1 = log($1); print g
replaces the rst eld of each line by its logarithm.
Awk patterns may include arbitrary boolean combinations of regular expres-
sions and of relational operators on strings, numbers, elds, variables, and array
elements. Actions may include the same pattern-matching constructions as in
patterns, as well as arithmetic and string expressions and assignments, if-else,
while, for statements, and multiple output streams.
This report contains a user's guide, a discussion of the design and imple-
mentation of awk, and some timing statistics.
September 1, 1978

Awk | A Pattern Scanning and Processing Language
(Second Edition)
Alfred V. Aho
Brian W. Kernighan
Peter J. Weinberger
1. Introduction
Awk is a programming language designed to
make many common information retrieval and text
manipulation tasks easy to state and to perform.
The basic operation of awk is to scan a set
of input lines in order, searching for lines which
match any of a set of patterns which the user has
speci ed. For each pattern, an action can be spec-
i ed; this action will be performed on each line
that matches the pattern.
Readers familiar with the UNIXy program
grep1 will recognize the approach, although in
awk the patterns may be more general than in
grep, and the actions allowed are more involved
than merely printing the matching line. For exam-
ple, the awk program
fprint $3, $2g
prints the third and second columns of a table in
that order. The program
$2 /AjBjC/
prints all input lines with an A, B, or C in the sec-
ond eld. The program
$1 != prev f print; prev = $1 g
prints all lines in which the rst eld is di erent
from the previous rst eld.
1.1. Usage
The command
awk program [files]
executes the awk commands in the string pro-
gram on the set of named les, or on the standard
input if there are no les. The statements can also
be placed in a le p le, and executed by the com-
mand
awk ,f pfile [files]
y UNIX is a trademark of Bell Laboratories.
1 K. Thompson and D. M. Ritchie, UNIX Programmer's
Manual, Bell Laboratories, May 1975. Sixth Edition
1.2. Program Structure
An awk program is a sequence of statements
of the form:
pattern f action g
pattern f action g
...
Each line of input is matched against each of the
patterns in turn. For each pattern that matches,
the associated action is executed. When all the
patterns have been tested, the next line is fetched
and the matching starts over.
Either the pattern or the action may be left
out, but not both. If there is no action for a pat-
tern, the matching line is simply copied to the out-
put. (Thus a line which matches several patterns
can be printed several times.) If there is no pat-
tern for an action, then the action is performed for
every input line. A line which matches no pattern
is ignored.
Since patterns and actions are both
optional, actions must be enclosed in braces to dis-
tinguish them from patterns.
1.3. Records and Fields
Awk input is divided into ``records'' termi-
nated by a record separator. The default record
separator is a newline, so by default awk processes
its input a line at a time. The number of the cur-
rent record is available in a variable named NR.
Each input record is considered to be
divided into `` elds.'' Fields are normally sepa-
rated by white space | blanks or tabs | but the
input eld separator may be changed, as described
below. Fields are referred to as $1, $2, and so
forth, where $1 is the rst eld, and $0 is the
whole input record itself. Fields may be assigned
to. The number of elds in the current record is
available in a variable named NF.
The variables FS and RS refer to the input
eld and record separators; they may be changed
at any time to any single character. The optional
command-line argument ,Fc may also be used to

- 2 -
set FS to the character c.
If the record separator is empty, an empty
input line is taken as the record separator, and
blanks, tabs and newlines are treated as eld sepa-
rators.
The variable FILENAME contains the
name of the current input le.
1.4. Printing
An action may have no pattern, in which
case the action is executed for all lines. The sim-
plest action is to print some or all of a record; this
is accomplished by the awk command print. The
awk program
f print g
prints each record, thus copying the input to the
output intact. More useful is to print a eld or
elds from each record. For instance,
print $2, $1
prints the rst two elds in reverse order. Items
separated by a comma in the print statement will
be separated by the current output eld separator
when output. Items not separated by commas will
be concatenated, so
print $1 $2
runs the rst and second elds together.
The prede ned variables NF and NR can
be used; for example
f print NR, NF, $0 g
prints each record preceded by the record number
and the number of elds.
Output may be diverted to multiple les;
the program
f print $1 foo1; print $2 foo2 g
writes the rst eld, $1, on the le foo1, and the
second eld on le foo2. The notation can
also be used:
print $1 foo
appends the output to the le foo. (In each case,
the output les are created if necessary.) The le
name can be a variable or a eld as well as a con-
stant; for example,
print $1 $2
uses the contents of eld 2 as a le name.
Naturally there is a limit on the number of
output les; currently it is 10.
Similarly, output can be piped into another
process (on UNIX only); for instance,
print j mail bwk
mails the output to bwk.
The variables OFS and ORS may be used
to change the current output eld separator and
output record separator. The output record sepa-
rator is appended to the output of the print state-
ment.
Awk also provides the printf statement for
output formatting:
printf format expr, expr, ...
formats the expressions in the list according to the
speci cation in format and prints them. For
example,
printf %8.2f %10ldnn, $1, $2
prints $1 as a oating point number 8 digits wide,
with two after the decimal point, and $2 as a
10-digit long decimal number, followed by a new-
line. No output separators are produced automati-
cally; you must add them yourself, as in this exam-
ple. The version of printf is identical to that used
with C.2
2. Patterns
A pattern in front of an action acts as a
selector that determines whether the action is to
be executed. A variety of expressions may be used
as patterns: regular expressions, arithmetic rela-
tional expressions, string-valued expressions, and
arbitrary boolean combinations of these.
2.1. BEGIN and END
The special pattern BEGIN matches the
beginning of the input, before the rst record is
read. The pattern END matches the end of the
input, after the last record has been processed.
BEGIN and END thus provide a way to gain
control before and after processing, for initializa-
tion and wrapup.
As an example, the eld separator can be
set to a colon by
BEGIN f FS = : g
... rest of program ...
Or the input lines may be counted by
END f print NR g
If BEGIN is present, it must be the rst pattern;
END must be the last if used.
2 B. W. Kernighan and D. M. Ritchie, The C Program-
ming Language, Prentice-Hall, Englewood Cli s, New Jer-
sey, 1978.

- 3 -
2.2. Regular Expressions
The simplest regular expression is a literal
string of characters enclosed in slashes, like
/smith/
This is actually a complete awk program which
will print all lines which contain any occurrence of
the name ``smith''. If a line contains ``smith'' as
part of a larger word, it will also be printed, as in
blacksmithing
Awk regular expressions include the regular
expression forms found in the UNIX text editor
ed3 and grep (without back-referencing). In addi-
tion, awk allows parentheses for grouping, j for
alternatives, + for ``one or more'', and ? for ``zero
or one'', all as in lex. Character classes may be
abbreviated: [a,zA,Z0,9] is the set of all letters
and digits. As an example, the awk program
/[Aa]hoj[Ww]einbergerj[Kk]ernighan/
will print all lines which contain any of the names
``Aho,'' ``Weinberger'' or ``Kernighan,'' whether
capitalized or not.
Regular expressions (with the extensions
listed above) must be enclosed in slashes, just as in
ed and sed. Within a regular expression, blanks
and the regular expression metacharacters are sig-
ni cant. To turn of the magic meaning of one of
the regular expression characters, precede it with a
backslash. An example is the pattern
/n/.n//
which matches any string of characters enclosed in
slashes.
One can also specify that any eld or vari-
able matches a regular expression (or does not
match it) with the operators and !. The pro-
gram
$1 /[jJ]ohn/
prints all lines where the rst eld matches ``john''
or ``John.'' Notice that this will also match ``John-
son'', ``St. Johnsbury'', and so on. To restrict it to
exactly [jJ]ohn, use
$1 /^[jJ]ohn$/
The caret ^ refers to the beginning of a line or
eld; the dollar sign $ refers to the end.
2.3. Relational Expressions
An awk pattern can be a relational expres-
sion involving the usual relational operators ,
=, ==, !=, =, and . An example is
$2 $1 + 100
which selects lines where the second eld is at
least 100 greater than the rst eld. Similarly,
NF % 2 == 0
prints lines with an even number of elds.
In relational tests, if neither operand is
numeric, a string comparison is made; otherwise it
is numeric. Thus,
$1 = s
selects lines that begin with an s, t, u, etc. In the
absence of any other information, elds are treated
as strings, so the program
$1 $2
will perform a string comparison.
2.4. Combinations of Patterns
A pattern can be any boolean combination
of patterns, using the operators jj (or), (and),
and ! (not). For example,
$1 = s $1 t $1 != smith
selects lines where the rst eld begins with ``s'',
but is not ``smith''. and jj guarantee that
their operands will be evaluated from left to right;
evaluation stops as soon as the truth or falsehood
is determined.
2.5. Pattern Ranges
The ``pattern'' that selects an action may
also consist of two patterns separated by a comma,
as in
pat1, pat2 f ... g
In this case, the action is performed for each line
between an occurrence of pat1 and the next
occurrence of pat2 (inclusive). For example,
/start/, /stop/
prints all lines between start and stop, while
NR == 100, NR == 200 f ... g
does the action for lines 100 through 200 of the
input.
3. Actions
An awk action is a sequence of action state-
ments terminated by newlines or semicolons.
These action statements can be used to do a vari-
ety of bookkeeping and string manipulating tasks.

- 4 -
3.1. Built-in Functions
Awk provides a ``length'' function to com-
pute the length of a string of characters. This pro-
gram prints each record, preceded by its length:
fprint length, $0g
length by itself is a ``pseudo-variable'' which
yields the length of the current record;
length(argument) is a function which yields the
length of its argument, as in the equivalent
fprint length($0), $0g
The argument may be any expression.
Awk also provides the arithmetic functions
sqrt, log, exp, and int, for square root, base e
logarithm, exponential, and integer part of their
respective arguments.
The name of one of these built-in functions,
without argument or parentheses, stands for the
value of the function on the whole record. The
program
length 10 jj length 20
prints lines whose length is less than 10 or greater
than 20.
The function substr(s, m, n) produces the
substring of s that begins at position m (origin 1)
and is at most n characters long. If n is omitted,
the substring goes to the end of s. The function
index(s1, s2) returns the position where the
string s2 occurs in s1, or zero if it does not.
The function sprintf(f, e1, e2, ...) pro-
duces the value of the expressions e1, e2, etc., in
the printf format speci ed by f. Thus, for exam-
ple,
x = sprintf(%8.2f %10ld, $1, $2)
sets x to the string produced by formatting the
values of $1 and $2.
3.2. Variables, Expressions, and Assign-
ments
Awk variables take on numeric ( oating
point) or string values according to context. For
example, in
x = 1
x is clearly a number, while in
x = smith
it is clearly a string. Strings are converted to
numbers and vice versa whenever context demands
it. For instance,
x = 3 + 4
assigns 7 to x. Strings which cannot be inter-
preted as numbers in a numerical context will gen-
erally have numeric value zero, but it is unwise to
count on this behavior.
By default, variables (other than built-ins)
are initialized to the null string, which has numeri-
cal value zero; this eliminates the need for most
BEGIN sections. For example, the sums of the
rst two elds can be computed by
f s1 += $1; s2 += $2 g
END f print s1, s2 g
Arithmetic is done internally in oating
point. The arithmetic operators are +, ,, , /,
and % (mod). The C increment ++ and decre-
ment ,, operators are also available, and so are
the assignment operators +=, ,=, =, /=, and
%=. These operators may all be used in expres-
sions.
3.3. Field Variables
Fields in awk share essentially all of the
properties of variables | they may be used in
arithmetic or string operations, and may be
assigned to. Thus one can replace the rst eld
with a sequence number like this:
f $1 = NR; print g
or accumulate two elds into a third, like this:
f $1 = $2 + $3; print $0 g
or assign a string to a eld:
f if ($3 1000)
$3 = too big
print
g
which replaces the third eld by ``too big'' when it
is, and in any case prints the record.
Field references may be numerical expres-
sions, as in
f print $i, $(i+1), $(i+n) g
Whether a eld is deemed numeric or string
depends on context; in ambiguous cases like
if ($1 == $2) ...
elds are treated as strings.
Each input line is split into elds automati-
cally as necessary. It is also possible to split any
variable or string into elds:
n = split(s, array, sep)
splits the the string s into array[1], ..., array[n].
The number of elements found is returned. If the
sep argument is provided, it is used as the eld
separator; otherwise FS is used as the separator.

- 5 -
3.4. String Concatenation
Strings may be concatenated. For example
length($1 $2 $3)
returns the length of the rst three elds. Or in a
print statement,
print $1 is $2
prints the two elds separated by `` is ''. Variables
and numeric expressions may also appear in con-
catenations.
3.5. Arrays
Array elements are not declared; they spring
into existence by being mentioned. Subscripts
may have any non-null value, including non-
numeric strings. As an example of a conventional
numeric subscript, the statement
x[NR] = $0
assigns the current input record to the NR-th ele-
ment of the array x. In fact, it is possible in prin-
ciple (though perhaps slow) to process the entire
input in a random order with the awk program
f x[NR] = $0 g
END f ... program ... g
The rst action merely records each input line in
the array x.
Array elements may be named by non-
numeric values, which gives awk a capability
rather like the associative memory of Snobol
tables. Suppose the input contains elds with val-
ues like apple, orange, etc. Then the program
/apple/ f x[apple]++ g
/orange/ f x[orange]++ g
END f print x[apple], x[orange] g
increments counts for the named array elements,
and prints them at the end of the input.
3.6. Flow-of-Control Statements
Awk provides the basic ow-of-control state-
ments if-else, while, for, and statement grouping
with braces, as in C. We showed the if statement
in section 3.3 without describing it. The condition
in parentheses is evaluated; if it is true, the state-
ment following the if is done. The else part is
optional.
The while statement is exactly like that of
C. For example, to print all input elds one per
line,
i = 1
while (i = NF) f
print $i
++i
g
The for statement is also exactly that of C:
for (i = 1; i = NF; i++)
print $i
does the same job as the while statement above.
There is an alternate form of the for state-
ment which is suited for accessing the elements of
an associative array:
for (i in array)
statement
does statement with i set in turn to each element
of array. The elements are accessed in an appar-
ently random order. Chaos will ensue if i is
altered, or if any new elements are accessed during
the loop.
The expression in the condition part of an
if, while or for can include relational operators
like , =, , =, == (``is equal to''), and !=
(``not equal to''); regular expression matches with
the match operators and !; the logical opera-
tors jj, , and !; and of course parentheses for
grouping.
The break statement causes an immediate
exit from an enclosing while or for; the continue
statement causes the next iteration to begin.
The statement next causes awk to skip
immediately to the next record and begin scanning
the patterns from the top. The statement exit
causes the program to behave as if the end of the
input had occurred.
Comments may be placed in awk programs:
they begin with the character # and end with the
end of the line, as in
print x, y # this is a comment
4. Design
The UNIX system already provides several
programs that operate by passing input through a
selection mechanism. Grep, the rst and simplest,
merely prints all lines which match a single speci-
ed pattern. Egrep provides more general pat-
terns, i.e., regular expressions in full generality;
fgrep searches for a set of keywords with a particu-
larly fast algorithm. Sed4 provides most of the
editing facilities of the editor ed, applied to a

- 6 -
stream of input. None of these programs provides
numeric capabilities, logical relations, or variables.
Lex5 provides general regular expression
recognition capabilities, and, by serving as a C
program generator, is essentially open-ended in its
capabilities. The use of lex, however, requires a
knowledge of C programming, and a lex program
must be compiled and loaded before use, which
discourages its use for one-shot applications.
Awk is an attempt to ll in another part of
the matrix of possibilities. It provides general reg-
ular expression capabilities and an implicit
input/output loop. But it also provides convenient
numeric processing, variables, more general selec-
tion, and control ow in the actions. It does not
require compilation or a knowledge of C. Finally,
awk provides a convenient way to access elds
within lines; it is unique in this respect.
Awk also tries to integrate strings and num-
bers completely, by treating all quantities as both
string and numeric, deciding which representation
is appropriate as late as possible. In most cases
the user can simply ignore the di erences.
Most of the e ort in developing awk went
into deciding what awk should or should not do
(for instance, it doesn't do string substitution) and
what the syntax should be (no explicit operator
for concatenation) rather than on writing or
debugging the code. We have tried to make the
syntax powerful but easy to use and well adapted
to scanning les. For example, the absence of dec-
larations and implicit initializations, while proba-
bly a bad idea for a general-purpose programming
language, is desirable in a language that is meant
to be used for tiny programs that may even be
composed on the command line.
In practice, awk usage seems to fall into two
broad categories. One is what might be called
``report generation'' | processing an input to
extract counts, sums, sub-totals, etc. This also
includes the writing of trivial data validation pro-
grams, such as verifying that a eld contains only
numeric information or that certain delimiters are
properly balanced. The combination of textual
and numeric processing is invaluable here.
A second area of use is as a data trans-
former, converting data from the form produced
by one program into that expected by another.
The simplest examples merely select elds, per-
haps with rearrangements.
5 M. E. Lesk, ``Lex | A Lexical Analyzer Generator,''
Comp. Sci. Tech. Rep. No. 39, Bell Laboratories, Murray
Hill, New Jersey, October 1975.
5. Implementation
The actual implementation of awk uses the
language development tools available on the UNIX
operating system. The grammar is speci ed with
yacc;6 the lexical analysis is done by lex; the regu-
lar expression recognizers are deterministic nite
automata constructed directly from the expres-
sions. An awk program is translated into a parse
tree which is then directly executed by a simple
interpreter.
Awk was designed for ease of use rather
than processing speed; the delayed evaluation of
variable types and the necessity to break input
into elds makes high speed dicult to achieve in
any case. Nonetheless, the program has not
proven to be unworkably slow.
Table I below shows the execution (user +
system) time on a PDP-11/70 of the UNIX pro-
grams wc, grep, egrep, fgrep, sed, lex, and awk
on the following simple tasks:
1. count the number of lines.
2. print all lines containing ``doug''.
3. print all lines containing ``doug'', ``ken'' or
``dmr''.
4. print the third eld of each line.
5. print the third and second elds of each line,
in that order.
6. append all lines containing ``doug'', ``ken'',
and ``dmr'' to les ``jdoug'', ``jken'', and
``jdmr'', respectively.
7. print each line pre xed by ``line-number : ''.
8. sum the fourth column of a table.
The program wc merely counts words, lines and
characters in its input; we have already mentioned
the others. In all cases the input was a le con-
taining 10,000 lines as created by the command ls
,l; each line has the form
,rw,rw,rw, 1 ava 123 Oct 15 17:05 xxx
The total length of this input is 452,960 charac-
ters. Times for lex do not include compile or load.
As might be expected, awk is not as fast as
the specialized tools wc, sed, or the programs in
the grep family, but is faster than the more gen-
eral tool lex. In all cases, the tasks were about as
easy to express as awk programs as programs in
these other languages; tasks involving elds were
considerably easier to express as awk programs.
Some of the test programs are shown in awk, sed
and lex.
6 S. C. Johnson, ``Yacc | Yet Another Compiler-
Compiler,'' Comp. Sci. Tech. Rep. No. 32, Bell Laborato-
ries, Murray Hill, New Jersey, July 1975.

- 7 -
Task
Program 1 2 3 4 5 6 7 8
wc 8.6
grep 11.7 13.1
egrep 6.2 11.5 11.6
fgrep 7.7 13.8 16.1
sed 10.2 11.6 15.8 29.0 30.5 16.1
lex 65.1 150.1 144.2 67.7 70.3 104.0 81.7 92.8
awk 15.0 25.6 29.9 33.3 38.9 46.4 71.4 31.1
Table I. Execution Times of Programs. (Times are in sec.)
The programs for some of these jobs are
shown below. The lex programs are generally too
long to show.
AWK:
1. END fprint NRg
2. /doug/
3. /kenjdougjdmr/
4. fprint $3g
5. fprint $3, $2g
6. /ken/ fprint jkeng
/doug/ fprint jdougg
/dmr/ fprint jdmrg
7. fprint NR : $0g
8. fsum = sum + $4g
END fprint sumg
SED:
1. $=
2. /doug/p
3. /doug/p
/doug/d
/ken/p
/ken/d
/dmr/p
/dmr/d
4. /[^ ] [ ][^ ] [ ]n([^ ]n) ./s//n1/p
5. /[^ ] [ ]n([^ ]n) [ ]n([^ ]n) ./s//n2 n1/p
6. /ken/w jken
/doug/w jdoug
/dmr/w jdmr
LEX:
1. %f
int i;
%g
%%
nn i++;
. ;
%%
yywrap() f
printf(%dnn, i);
g
2. %%
^.doug.$ printf(%snn, yytext);
. ;
nn ;

Awk --- A Pattern Scanning And Processing Language (Second Edition)

More Related Content

Similar to Awk --- A Pattern Scanning And Processing Language (Second Edition) (20)

More from Brooke Heidt (20)

Recently uploaded (20)

Awk --- A Pattern Scanning And Processing Language (Second Edition)