![AppleScript
William R. Cook
University of Texas at Austin
wcook@cs.utexas.edu
With contributions from Warren Harris, Kurt Piersol, Dave Curbow, Donn Denman,
Edmund Lai, Ron Lichty, Larry Tesler, Donald Olson, Mitchell Gass and Eric House
Abstract
AppleScript is a scripting language and environment for the
Mac OS. Originally conceived in 1989, AppleScript allows
end users to automate complex tasks and customize Mac
OS applications. To automate tasks, AppleScript provides
standard programming language features (control flow, vari-
ables, data structures) and sends Apple Events to invoke ap-
plication behavior. Apple Events are a variation on standard
remote procedure calls in which messages can identify their
arguments by queries that are interpreted by the remote ap-
plication. This approach avoids the need for remote object
pointers or proxies, and reduces the number of communi-
cation round trips, which are expensive in high latency en-
vironments like the early Macintosh OS. To customize an
application that uses AppleScript’s Open Scripting Architec-
ture, users attach scripts to application objects; these scripts
can then intercept and modify application behavior.
AppleScript was designed for casual users: AppleScript
syntax resembles natural language, and scripts can be cre-
ated easily by recording manual operations on a graphical
interface. AppleScript also supported internationalization in
allowing script to be presented in multiple dialects, including
English, Japanese, or French. Although the naturalistic syn-
tax is easy to read, it can make scripts much more difficult
to write.
Early adoption was hindered by the difficulty of mod-
ifying applications to support Apple Events and the Open
Scripting Architecture. Yet AppleScript is now widely used
and is an essential differentiator of the Mac OS. Apple-
Script’s communication model is a precursor to web ser-
vices, and the idea of embedded scripting has been widely
adopted.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation
on the first page. To copy otherwise, to republish, to post on servers or to redistribute
to lists, requires prior specific permission and/or a fee.
Copyright c
° ACM [to be supplied]...$5.00
Categories and Subject Descriptors D.3 [Programming
Languages]
General Terms Languages, Design, Human Factors
Keywords AppleScript, Scripting, History
1. Introduction
The development of AppleScript was a long and complex
process that spanned multiple teams, experienced several
false starts and changes of plan, and required coordination
between different projects and companies. It is difficult for
any one person, or even a small group of people, to present a
comprehensive history of such a project, especially without
official support from the company for which the work was
done. The email record of the team’s communications have
been lost, and the author no longer has access to internal
specifications and product plans.
Nevertheless, I believe that the development of Apple-
Script is a story worth telling, and I have been encouraged to
attempt it despite the inherent difficulty of the task. I can of-
fer only my own subjective views on the project, as someone
who was intimately involved with all its aspects. I apologize
in advance for errors and distortions that I will inevitably
introduce into the story, in spite of my best efforts to be ac-
curate.
I first heard the idea of AppleScript over lunch with Kurt
Piersol in February of 1991. The meeting was arranged by
our mutual friend James Redfern. I knew James from Brown,
where he was finishing his undergraduate degree after some
time off, and I was working on my PhD. James and I both
moved to California at about the same time, in 1988. We
spent a lot of time together and I had heard a little about what
he was up to, but he claimed it was secret. James arranged
the meeting because Kurt was looking for someone to lead
the AppleScript effort, and I was looking for something new
to do.
For the previous two and a half years I had been work-
ing at HP Labs. I was a member of the Abel group, which
included Walt Hill, Warren Harris, Peter Canning, and Wal-
ter Olthoff. John Mitchell consulted with us from Stanford.
The group was managed by Alan Snyder, whose formaliza-
Permission to make digital/hard copy of part of this work for personal or
classroom use is granted without fee provided that the copies are not made or
distributed for profit or commercial advantage, the copyright notice, the title of
the publication, and its date of appear, and notice is given that copying is by
permission of the ACM, Inc. To copy otherwise, to republish, to post on servers,
or to redistribute to lists, requires prior specific permission and/or a fee.
Permission may be requested from the Publications Dept., ACM, Inc., 2 Penn
Plaza, New York, NY 11201-0701, USA, fax:+1(212) 869-0481,
©2007 ACM 978-1-59593-766-7/2007/06-ART1 $5.00
DOI 10.145/1238844.1238845
http://doi.acm.org/10.1145/1238844.1238845
Permission to make digital/hard copy of part of this work for personal or
classroom use is granted without fee provided that the copies are not
made or distributed for profit or commercial advantage, the copyright
notice, the title of the publication, and its date of appear, and notice is
given that copying is by permission of the ACM, Inc. To copy
otherwise, to republish, to post on servers, or to redistribute to lists,
requires prior specific permission and/or a fee. Permission may be
requested from the Publications Dept., ACM, Inc., 2 Penn Plaza, New
York, NY 11201-0701, USA, fax:+1(212) 869-0481,
permissions@acm.org
©2007 ACM 978-1-59593-766-7/2007/06-ART1 $5.00
DOI 10.1145/1238844.1238845
http://doi.acm.org/10.1145/1238844.1238845
1-1](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-9-320.jpg)
![tion of object concepts [35] was one basis for the develop-
ment of CORBA. At HP Labs I finished writing my PhD the-
sis, A Denotational Semantics of Inheritance [14, 19], which
the Abel group used as the foundation for a number of pa-
pers. We published papers on inheritance and subtyping [18],
object-oriented abstraction [7, 16], mixins [5], F-Bounded
polymorphism [6], and a fundamental flaw in the Eiffel type
system [15]. I continued some of the work, analyzing the
Smalltalk collection hierarchy [17], after I left HP.
Near the end of 1990 HP Labs was undergoing a reorgani-
zation and I was not sure I would fit into the new plan. In ad-
dition, I was interested in making a change. I had developed
several fairly large systems as an undergraduate, including a
text editor and a graphical software design tool [39, 20] that
were used by hundreds of other students for many years. As
a graduate student I had focused on theory, but I wanted to
learn the process of commercial software development. Ap-
ple seemed like a reasonable place to try, so I agreed to talk
with Kurt.
1.1 AppleScript Vision—Over Lunch
Kurt and I hit it off immediately at lunch. Kurt Piersol is a
large, friendly, eloquent man who was looking for people to
work on the AppleScript project. Kurt graduated with a B.S.
from the University of Louisville’s Speed Scientific School.
He then worked at Xerox, building systems in Smalltalk-
80 and productizing research systems from Xerox PARC.
Kurt was hired to work on AppleScript in 1989. He was
originally asked to work on a development environment for
the new language, but eventually took on the role of steering
the project.
Kurt and I discussed the advantages or disadvantages
of command-line versus graphical user interfaces. With
command-line interfaces, commonly used on Unix, users
frequently write scripts that automate repeated sequences of
program executions. The ability to pipe the output of one
program into another program is a simple but powerful form
of inter-application communication that allows small pro-
grams to be integrated to perform larger tasks. For example,
one can write a script that sends a customized version of
a text file to a list of users. The sed stream editor can cre-
ate the customized text file, which is then piped into the
mail command for delivery. This new script can be saved
as a mail-merge command, so that it is available for manual
execution or invocation from other scripts. One appealing
aspect of this model is its compositionality: users can create
new commands that are invoked in the same way as built-in
commands. This approach works well when atomic com-
mands all operate on a common data structure, in this case
text streams. It was not obvious that it would work for more
complex structured data, like images, databases, or office
documents, or for long-running programs that interact with
users.
With a graphical user interface (GUI) important func-
tions, including the mail-merge command described above,
are usually built into a larger product, e.g. a word processor.
A GUI application offers pre-packaged integrated function-
ality, so users need not combine basic commands to perform
common tasks. Although careful design of graphical inter-
faces eliminates the need for automation for basic use, there
are still many tasks that users perform repeatedly within a
graphical interface. The designers of the graphical interface
cannot include commands to cover all these situations —
if they did, then some users would execute these new com-
mands repeatedly. No finite set of commands can ever satisfy
all situations.
Most users are happy with GUI applications and do not
need a command-line interface or a scripting language. But
there are clearly some limitations to the GUI approach on
which Macintosh OS was based. Power users and system
integrators were frustrated by the inability to build custom
solutions by assembling multiple applications and special-
izing them to particular tasks. Allowing users to automate
the tasks that are relevant to them relieves pressure on the
graphical user interface to include more and more special-
ized features.
The vision for AppleScript was to provide a kind of
command-line interface to augment the power of GUI ap-
plications and to bring this power to casual users.
1.2 Automation and Customization
Kurt and I talked about two ways in which scripts and
GUI applications could interact: for automation and for cus-
tomization. Automation means that a script directs an ap-
plication to perform a sequence of actions—the actions are
performed “automatically” rather than manually. With au-
tomation, the script is in control and one or more applica-
tions respond to script requests. Customization occurs when
a script is invoked from within an application—the script
can perform “custom” actions that replace or augment the
normal application behavior. With customization, the appli-
cation manages and invokes scripts that users have attached
to application objects. Automation is useful even without
customization, but customization requires automation to be
useful.
We discussed whether there was sufficient benefit in pro-
viding a standard platform for scripting, when custom so-
lutions for each application might be better. Some applica-
tions already had their own macro capability or a proprietary
scripting language. However, this approach requires users to
learn a different scripting language to automate each appli-
cation. These languages typically include some variation on
the basic elements of any programming language, including
variables, loops, conditionals, data types, procedures, and
exceptions. In addition, they may include special forms or
constructs specific to the application in question. For exam-
ple, a spreadsheet language can refer to cells, sheets, equa-
tions and evaluation.
One benefit of a standard scripting platform is that appli-
cations can then be integrated with each other. This capa-
1-2](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-10-320.jpg)
![bility is important because users typically work with mul-
tiple applications at the same time. In 1990, the user’s op-
tions for integrating applications on the Macintosh OS were
limited to shared files or copy/paste with the clipboard. If
a repeated task involves multiple applications, there is lit-
tle hope that one application will implement a single com-
mand to perform the action. Integrating graphical applica-
tions can be done at several levels: visual integration, behav-
ioral integration, or data integration. Visual integration in-
volves embedding one graphical component inside another;
examples include a running Java applet inside a web page, or
a specialized organization chart component displayed inside
a word-processing document. Behavioral integration occurs
when two components communicate with each other; exam-
ples include workflow or invocation of a spell-check com-
ponent from within a word processor. Data integration oc-
curs whenever one application reads a file (or clipboard data)
written by another application. A given system can include
aspects of all three forms of integration.
I agreed with Kurt that the most important need at that
time was for behavioral integration. To compete with custom
application-specific scripting languages, AppleScript would
have to allow application-specific behaviors to be incorpo-
rated into the language in as natural a way as possible, while
preserving the benefits of a common underlying language.
The core of the language should support the standard fea-
tures of a programming language, including variables, pro-
cedures, and basic data types. An application then provides a
vocabulary of specific terminology that apply to the domain:
a photo-processing application would manipulate images,
pixels and colors, while a spreadsheet application would ma-
nipulate cells, sheets, and graphs. The idea of AppleScript
was to implement the “computer science boilerplate” once,
while seamlessly integrating the vocabulary of the applica-
tion domain so that users of the language can manipulate
domain objects naturally. We discussed the vision of Apple-
Script as a pervasive architecture for inter-application com-
munication, so that it is easy to integrate multiple applica-
tions with a script, or invoke the functionality of one appli-
cation from a script in another application. We hoped that
scripting would create a “network effect”, by which each
new scriptable application improves the value of scripting
for all other applications.
1.3 AppleScript Begins
Soon after, Kurt offered me a job and I accepted quickly.
This event illustrates one of the recurring characteristics of
AppleScript: the basic idea is so compelling that it is enthusi-
astically embraced by almost every software developer who
is exposed to it.
What was not immediately obvious was how difficult the
vision was to achieve—not for strictly technical reasons, but
because AppleScript required a fundamental refactoring, or
at least augmentation, of almost the entire Macintosh code
base. The demonstrable benefits of AppleScript’s vision has
led developers to persevere in this massive task for the last
twenty years; yet the work is truly Sisyphean, in that the slow
incremental progress has been interrupted by major steps
backward, first when the hardware was changed from the
Motorola 68000 chip to the IBM PowerPC, and again when
the operating system was reimplemented for Mac OS X.
At this point it is impossible to identify one individual
as the originator of the AppleScript vision. The basic idea
is simple and has probably been independently discovered
many times. The AppleScript team was successful in elab-
orating the original vision into a practical system used by
millions of people around the world.
2. Background
When I started working at Apple in April 1991 I had never
used a Macintosh computer. My first task was to understand
the background and context in which AppleScript had to be
built.
The main influences I studied were the Macintosh op-
erating system, HyperCard, and Apple Events. HyperCard
was a good source of inspiration because it was a flexible
application development environment with a scripting lan-
guage embedded within it. A previous team had designed
and implemented Apple Events to serve as the underlying
mechanism for inter-application communication. The Apple
Events Manager had to be shipped early so that it could be
included in the Macintosh System 7 OS planned for summer
1991. When I started at Apple, the Apple Event Manager
was in final beta testing. The fact that AppleScript and Apple
Events were not designed together proved to be a continuing
source of difficulties.
Macintosh systems at that time had 4 to 8 megabytes of
random-access memory (RAM) and a 40- to 60-megabyte
hard drive. They had 25-50 MHz Motorola 68000 series
processors. The entire company was internally testing Sys-
tem 7.0, a major revision of the Macintosh OS.
Applications on the Macintosh OS were designed around
a main event processing loop, which handled lower-level
keyboard and mouse events from the operating system [12].
The OS allowed an application to post a low-level event
to another application, providing a simple form of inter-
application communication. In this way one application
could drive another application, by sending synthetic mouse
and keyboard events that select menus or data and enter text
into an application. This technique was used in two utility
applications, MacroMaker and QuicKeys, which recorded
and played back low-level user interface events. It was also
used in the Macintosh help system, which could post low-
level events to show the user how to use the system. Scripts
that send low-level events are fragile, because they can fail
if the position or size of user interface elements changes
between the time the script is recorded and when it is run.
They are also limited in the actions they can perform; low-
1-3](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-11-320.jpg)
![level events can be used to change the format of the current
cell in a spreadsheet, but cannot read the contents of a cell.
In the following section I describe these systems as they
were described to me at the start of the AppleScript project,
in April 1991.
2.1 HyperCard
HyperCard [27, 30], originally released in 1987, was the
most direct influence on AppleScript. HyperCard is a com-
bination of a simple database, a collection of user interface
widgets, and an English-like scripting language. These el-
ements are organized around the metaphor of information
on a collection of index cards. A collection of such cards
is called a stack. A card could contain text, pictures, but-
tons, and other graphical objects. Each object has many
properties, including location on the card, size, font, style,
etc. Cards with similar structure could use a common back-
ground; a background defines the structure, but not the con-
tent, of multiple cards. For example, a stack for household
information might contain recipe cards and contact cards.
The recipe cards use a recipe background that includes text
fields for the recipe title, ingredients, and steps. The contact
cards use a contact background with appropriate fields, in-
cluding a photo.
HyperCard scripts are written in HyperTalk, an English-
like scripting language [28]. The language is for the most
part a standard structured, imperative programming lan-
guage. However, it introduced a unique approach to data
structures: the stack, cards, objects and properties are used
to store data. These structures are organized in a contain-
ment hierarchy: stacks contain cards, cards contain objects,
and properties exist on stacks, cards, and objects. This pre-
defined structure makes it easy to build simple stacks, but
more difficult to create custom data structures.
Scripts in a stack can refer to the objects, collections of
objects, and their properties by using chunk expressions. A
chunk expression is best understood as a kind of query. For
example, the following chunk expression refers to the text
style property of a word element in a field of the current
card:
the textStyle of word 2
of card field ”Description”
A chunk expression can refer to properties and elements of
objects. A property is a single-valued attribute, for example
textStyle . Elements are collections of objects identified by a
type, for example word and card field . Element access may
be followed by a name, index or range to select element(s)
from the collection. Properties access distributes over col-
lections; the following expression represents a a collection
of 10 text style properties:
the textStyle of character 1 to 10
of card field ”Description”
HyperCard has a built-in set of property and collection
names.
Each object has a script containing procedures defined for
that object. If the procedure name is an event name, then the
procedure is a handler for that event— it is called when the
event occurs for that object. For example, a button script may
have handlers for mouseDown, mouseUp and mouseMove
events. The following handler shows the next card when a
button is released.
on mouseUp
go to next card
end mouseUp
Actions can be performed on chunk expressions to mod-
ify the stack, its cards, the objects, or their properties. For
example, clicking a button may run a script that moves to
the next card, adds/removes cards, or modifies the contents
of one or more cards. HyperCard has a set of predefined ac-
tions, including set, go, add, close, etc. For example, the
text style can be updated to a predefined constant bold:
set the textStyle of character 1 to 10
of card field ”Description” to bold
HyperCard 2.0 was released in 1990. HyperCard was
very influential and widely used. Developers could easily
create some applications in HyperCard, but to create more
complex applications, they had to switch to more difficult
general-purpose development tools. The need for unification
of these approaches was recognized early at Apple, leading
to the formation of a research and development project to
build a new development platform for the Mac, discussed
in the next section. Looking forward, the rapid develop-
ment capabilities pioneered by HyperCard were added to
more sophisticated general-purpose development environ-
ments. This gradually reduced the need for systems like Hy-
perCard, which was discontinued in 2004.
2.2 Family Farm
Many of the ideas that eventually emerged in AppleScript
were initially explored as part of a research project code-
named Family Farm, which was undertaken in the Advanced
Technology Group (ATG) at Apple, starting in 1989. The re-
search team was led by Larry Tesler and included Mike Farr,
Mitchell Gass, Mike Gough, Jed Harris, Al Hoffman, Ruben
Kleiman, Edmund Lai, and Frank Ludolph. Larry received
a B.S. in Mathematics from Stanford University in 1965. In
1963, he founded and ran IPC, one of the first software de-
velopment firms in Palo Alto, CA. From 1968-73, he did
research at the Stanford A.I. Lab on cognitive modeling and
natural language understanding, where he designed and de-
veloped PUB, one of the first markup languages with embed-
ded tags and scripting. From 1973-80, he was a researcher
at Xerox PARC, working on object-oriented languages, user
interfaces, and desktop publishing. He joined Apple in 1980
1-4](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-12-320.jpg)
![Document
Paragraph
Word
Character
Size Style
Size Style
Size Style
Name
Property
Element
A
B
A contains B
Key:
Root
Figure 1. The properties and elements in a simple object
model.
mand that can delete paragraphs or characters from a word-
processing document, or files from the file system. Having a
small number of generic verbs, including set, copy, delete,
insert, open and close, proved to be more extensible.
2.3.1 Object Specifiers
Object specifiers are symbolic references to objects in an
application. One application can create object specifiers that
refer to objects in another application, called the target.
The specifiers can then be included in an Apple Event and
sent to the target application. These symbolic references
are interpreted by the target application to locate the actual
remote objects. The target application then performs the
action specified by the verb in the Apple Event upon the
objects.
For example, a single Apple Event can be used to copy a
paragraph from one location to another in a document; the
source and target location are included in the event as object
specifiers. The content of the paragraph remains local in the
target application. Traditional RPC models would require
the client to retrieve the paragraph contents, and then send
it back to the target as a separate insertion operation.
Object specifiers provide a view of application data that
includes elements and properties. An element name repre-
sents a collection of values; a property has a single value.
The value of a property may be either a basic value, for ex-
ample an integer or a string, or another object. Elements are
always objects.
A simple object model is illustrated in Figure 1. A
document has multiple paragraph elements and a name
property. A paragraph has style and size properties and
contains word and character elements.
The distinction between properties and elements is re-
lated to the concept of cardinality in entity-relationship mod-
eling [9] and UML class diagrams [32]. Cardinality indicates
the maximum number of objects that may be involved in
a relationship. The most important distinction is between
single-valued (cardinality of 1) relationships and multi-
valued (cardinality greater than 1). The entity-relationship
model also includes attributes, which identify scalar, prim-
itive data values. An AppleScript property is used for both
attributes and single-valued relationships. Elements are used
to describe multivalued relationships.
The name identifying a set of elements is called a class
name, identifying a specific kind of contained object and/or
its role. For example, a Family object might have elements
parents and children , which are elements that refer to
sets of Person objects. Object specifiers allow application-
specific names for elements and properties, which generalize
the fixed set of predefined names available in HyperCard.
Object specifiers also generalize HyperCard chunk ex-
pressions in other ways. One extension was the addition of
conditions to select elements of a collection based on their
properties. This extension made object specifiers a form of
query language with significant expressive power.
Supporting Apple Events was frequently quite difficult.
To create a new scriptable application, the software architect
must design a scripting interface in addition to the traditional
GUI interface. If the existing application architecture sepa-
rates views (graphical presentations) from models (underly-
ing information representation) [34], then adding scripting
is usually possible. In this case the internal methods on the
model may be exposed to scripts. There are two reasons why
direct access to an existing object model may not be suffi-
cient:
1. Users often want to control the user interface of an appli-
cation, not just internal data of the application. Thus the
scripting interface should provide access to both the view
objects and the model objects.
2. The interfaces of the internal object model may not be
suitable as an external scripting interface. In this case
the scripting interface is usually implemented to provide
another abstract view of the internal model.
Even with a good architecture, it can be difficult to retrofit
an existing GUI application to include a second interface
for scripting. If the application does not use a model-view
architecture, then adding scripting is much more difficult.
The Apple Events team created a support library to assist
application writers in interpreting object specifiers. It inter-
prets nesting and conditions in the object specifiers, while
using application callbacks to perform primitive property
and element operations.
2.3.2 Apple Events Implementation
Object specifiers are represented in Apple Events as nested
record structures, called descriptors [11]. Descriptors use a
self-describing, tagged tree data structured designed to be
easily transported or stored in a flattened binary format. De-
scriptors can either contain primitive data, a list of descrip-
tors, or a labeled product of descriptors. Primitive data types
include numbers (small and large integers and floats), pic-
tures, styled and unstyled text, process IDs, files, and aliases.
All the structures, types, and record fields are identified by
1-6](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-14-320.jpg)
![four-byte type codes. These codes are chosen to be human-
readable to facilitate low-level debugging.
Each kind of object specifier is a record with fields for the
class, property name, index, and container. The container is
either another object specifier record or null, representing the
default or root container of the application. Events may be
sent synchronously or asynchronously. The default behavior
of Apple Events is stateless—the server does not maintain
state for each client session. However, Apple Events sup-
ports a simple form of transaction: multiple events can be
tagged with a transaction ID, which requires an application
to perform the events atomically or else signal an error.
Apple Events was first released with Macintosh System
7 in 1991. The entire Apple Events system was designed,
implemented, and shipped in the Macintosh OS before any
products using it were built. It was a complex problem: ap-
plications could not be built until the infrastructure existed,
but the infrastructure could not be validated until many ap-
plications had used it. In the end, the Apple Events team
adopted a “build it and they will come” approach. They de-
signed the system as well as they could to meet predicted
needs. Only a few small sample applications were developed
to validate the model. In addition, the operating system team
defined four standard events with a single simple parame-
ter: open application, open documents, print documents, and
quit. These first basic events did not use object specifiers; the
open and print events used a vector of path names as argu-
ments.
Later projects, including AppleScript, had to work around
many of the Apple Events design choices that were made
essentially within a vacuum. For example, Apple Events
included some complex optimized message that were never
used because they were too unwieldy. For example, if an
array of values all has a common prefix, this prefix can be
defined once and omitted in each element of the array. This
was originally motivated by a desire to omit repetitive type
information. This optimization is not used by AppleScript
because it is difficult to detect when it could be used, and the
reduction in message length provided by the optimization
does not significantly affect performance.
3. The AppleScript Language
My primary task was to lead the design and implementation
of the AppleScript language. After I decided to join Apple
I mentioned the opportunity to Warren Harris. I enjoyed
working with Warren at HP and thought he would be a great
addition to the AppleScript effort. Warren has a BS and
MS inree EE from the University of Kansas. At HP Warren
was still working on his “Abel Project Posthumous Report”,
which contained all the ideas we had discussed, but had not
time to complete, while working together at HP Labs [25].
Warren talked to Kurt and eventually decided to join the
AppleScript team as a software engineer. He quickly became
the co-architect and primary implementor of the language.
3.1 Requirements
AppleScript is intended to be used by all users of the Mac-
intosh OS. This does not imply that all users would use Ap-
pleScript to the same degree or in the same way—there is
clearly a wide range of sophistication in users of the Mac-
intosh OS, and many of them have no interest in, or need
for, learning even the simplest form of programming lan-
guage. However, these users could invoke scripts created by
other users, so there were important issues of packaging of
scripts, in addition to developing them. More sophisticated
users might be able to record or modify a script even if they
could not write it. Finally, it should be possible for non-
computer specialists to write scripts after some study.
The language was primarily aimed at casual program-
mers, a group consisting of programmers from all experi-
ence levels. What distinguishes casual programming is that
it is infrequent and in service of some larger goal—the pro-
grammer is trying to get something else done, not create a
program. Even an experienced software developer can be a
casual programmer in AppleScript.
The team recognized that scripting is a form of program-
ming and requires more study and thought than using a
graphical interface. The team felt that the language should
be easy enough to learn and use to make it accessible to
non-programmers, and that such users would learn enough
programming as they went along to do the things that they
needed to do.
Programs were planned to be a few hundred lines long
at most, written by one programmer and maintained by a
series of programmers over widely spaced intervals. Scripts
embedded inside an application were also expected to be
small. In this case, the application provides the structure
to interconnect scripts; one script may send messages to
an application object that contains another script. Because
scripts are small, compilation speed was not a significant
concern. Readability was important because it was one way
to check that scripts did not contain malicious code.
In the early ’90s computer memory was still expensive
enough that code size was a significant issue, so the Ap-
pleScript compiler and execution engine needed to be as
small as possible. Portability was not an issue, since the lan-
guage was specifically designed for the Macintosh OS. Per-
formance of scripts within an application was not a signifi-
cant concern, because complex operations like image filter-
ing or database lookup would be performed by applications
running native code, not by scripts. Due to the high latency
of process switches needed for communication, optimization
of communication costs was the priority.
3.2 Application Terminologies
An application terminology is a dictionary that defines the
names of all the events, properties, and elements supported
by an application. A terminology can define names as plural
1-7](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-15-320.jpg)
![reference ::=
propertyName
| ‘beginning’
| ‘end’
| ‘before’ term
| ‘after’ term
| ‘some’ singularClass
| ‘ first ’ singularClass
| ‘ last ’ singularClass
| term (‘ st ’ | ‘nd’ | ‘rd’ | ‘th’) anyClass
| ‘middle’ anyClass
| plural [‘ from’ term toOrThrough term]
| anyClass term [toOrThrough term]
| singularClass ‘before’ term
| singularClass ‘after’ term
| term (‘of’ | ‘in’ | ‘’ s’) term
| term (‘whose’ | ‘where’ | ‘that’) term
plural ::= pluralClass | ‘every’ anyClass
toOrThrough ::= ‘to’ | ‘thru’ | ‘through’
call ::= message ‘(’ expr∗ ‘)’
| message [‘in’ | ‘of ’] [term] arguments
| term name arguments
message ::= name | terminologyMessage
arguments ::= ( preposition expression | flag | record)∗
flag ::= (‘with’ | ‘without’) [name]+
record ::= ‘given’ (name ‘:’ expr)∗
preposition ::=
‘to’ | ‘from’ | ‘thru’ | ‘through’
| ‘by’ | ‘on’ | ‘into’ | terminologyPreposition
Figure 2. AppleScript grammar for object references and
message sends.
or masculine/feminine, and this information can be used by
the custom parser for a dialect.
One of the main functions of AppleScript is to send and
receive Apple Events (defined in Section 2.3). Sending an
Apple Event is analogous to an object-oriented message
send, while handling an Apple Event is done by defining a
method whose name is an Apple Event. One view of this
arrangement is that each application is an object that re-
sponds to a variety of messages containing complex argu-
ment types. AppleScript encourages a higher-level view in
which each application manages a set of objects to which
messages may be sent.
AppleScript also provides special syntax for manipulat-
ing Apple Event object specifiers, which are called “object
references” in AppleScript documentation. When an oper-
ation is performed on an object reference, an Apple Event
is created containing the appropriate verb and object spec-
ifier. Thus AppleScript needed to provide a concise syntax
for expressing Apple Events. These AppleScript expressions
create object specifiers:
the first word of paragraph 22
name of every figure of document ‘‘taxes ’’
the modification date of every file whose size > 1024
The first example is a reference to a particular word of a
paragraph. The second refers to the collection of names
associated with all figures in a document named “taxes”. The
last one refers to the modification dates of large files.
For example, the object reference name of window 1
identifies the name of the first window in an application.
Object references are like first-class pointers that can be
dereferenced or updated. An object reference is automati-
cally dereferenced when a primitive value is needed:
print the name of window 1
The primitive value associated with an object reference can
be updated:
set the name of window 1 to ‘‘Taxes’’
These examples illustrate that object specifiers can act as
both l-values and r-values.
Figure 2 includes the part of the AppleScript grammar
relating to object specifiers. The grammar makes use of four
nonterminals that represent symbols from the application
terminology: for property, singularClass , pluralClass ,
and anyClass. As mentioned in Section 2.3.1, a terminology
has properties and elements, which are identified by a class
name. Property names are identified by lexical analysis and
passed to the parser. For class names the terminology can in-
clude both plural and singular forms or a generic “any” form.
For example, name is a property, window is a singular class,
and windows is a plural class. The grammar then accepts
windows from 1 to 10 and every window from 1 to 10,
Figure 2 also summarizes the syntax of messages. Argu-
ments can be given by position or name after the given key-
word. In addition, a set of standard prepositions can be used
as argument labels. This allows messages of the form:
copy paragraph 1 to end of document
The first parameter is paragraph 1, and the second argu-
ment is a prepositional argument named to with value
end of document.
One of the most difficult aspects of the language design
arose from the fundamental ambiguity of object references:
an object reference is itself a first-class value, but it also de-
notes a particular object (or set of objects) within an applica-
tion. In a sense an object reference is like a symbolic pointer,
which can be dereferenced to obtain its value; the referenced
value can also be updated or assigned through the object
reference. The difficulties arose because of a desire to hide
the distinction between a reference and its value. The solu-
tion to this problem was to dereference them automatically
when necessary, and require special syntax to create an ob-
ject reference instead of accessing its value. The expression
a reference to o creates a first-class reference to an object
1-8](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-16-320.jpg)
![described by o. Although the automatic dereferencing han-
dles most cases, a script can explicitly dereference r using
the expression value of r. The examples above can be ex-
pressed using a reference value:
set x to a reference to the name of window 1
The variable x can then be used in place of the original
reference. The following statements illustrate the effect of
operations involving x:
print the value of x
print x
set the value of x to ‘‘ Taxes’’
set x to ‘‘ Taxes’’
The first and second statements both print the name of the
window. In the first statement the dereference is explicit, but
in the second it happens implicitly because print expects a
string, not a reference. The third statement changes the name
of the window, while the last one changes the values of the
variable x but leaves the window unchanged.
An object reference can be used as a base for further
object specifications.
set x to a reference to window 1
print the name of x
Figure 3 and 4 describe the custom terminology dictio-
nary for iChat, a chat, or instant messaging, program for the
Macintosh. It illustrates the use of classes, elements, proper-
ties, and custom events. Terminologies for large applications
are quite extensive: the Microsoft Word 2004 AppleScript
Reference is 529 pages long, and the Adobe Photoshop CS2
AppleScript Scripting Reference is 251 pages. They each
have dozens of classes and hundreds of properties.
In addition to the terminology interpreted by individ-
ual applications, AppleScript has its own terminology to
identify applications on multiple machines. The expression
application ‘‘ name’’ identifies an application. The expres-
sion application ‘‘ appName’’ of machine ‘‘machineName’’
refers to an application running on a specific machine. A
block of commands can be targeted at an application using
the tell statement:
tell application ‘‘ Excel ’’ on machine x
put 3.14 into cell 1 of row 2 of window 1
end
This is an example of a static tell command, because
the name of the target application is known statically, at
compile time. The target of the tell statement can also be
a dynamic value rather than an application literal. In this
case the terminology is not loaded from the application. To
communicate with a dynamic application using a statically
specified terminology, a dynamic tell can be nested inside
a static tell ; the outer one sets the static terminology, while
the inner one defines the dynamic target. This brings up the
possibility that applications may receive messages that they
Class application: iChat application
Plural form: applications
Elements: account, service , window, document
Properties:
idle time integer
Time in seconds that I have been idle.
image picture
My image as it appears in all services.
status message string
My status message, visible to other people
while I am online.
status string
My status on all services:
away/offline/available.
Class service: An instant-messaging service
Plural form: services
Elements: account
Properties:
status string
The status of the service.:
disconnecting/connected/connecting/disconnected.
id string
The unique id of the service.
name string
The name of the service.
image picture
The image for the service.
Class account: An account on a service
Plural form: accounts
Properties:
status string
away/offline/available/idle/unknown.
id string
The account’s service and handle. For ex-
ample: AIM:JohnDoe007.
handle string
The account’s online name.
name string
The account’s name as it appears in the
buddy list.
status message
The account’s status message.
capabilities list
The account’s messaging capabilities.
image picture
The account’s custom image.
idle time integer
The time in seconds the account has been
idle.
Figure 3. iChat Suite: Classes in the iChat scripting termi-
nology [13].
1-9](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-17-320.jpg)
![Events
log in service
Log in a service with an account. If the account password
is not in the keychain the user will be prompted to enter
one.
log out service
Logs out of a service, or all services if none is specified.
send message to account
Send account a text message or video invitation.
Figure 4. iChat Suite: Events in the iChat scripting termi-
nology [13].
do not understand. In such situations, the application should
return an error.
Integration of multiple applications opens the possibility
that a single command may involve object references from
multiple applications. The target of a message is determined
by examining the arguments of the message. If all the argu-
ments are references to the same application, then that appli-
cation is the target. But if the arguments contain references
to different applications, one of the applications must be cho-
sen as the target. Since applications can interpret only their
own object specifiers, the other object references must be
evaluated to primitive values, which can then be sent to the
target application.
copy the name of the first window
of application ”Excel”
to the end of the first paragraph
of app ”Scriptable Text Editor”
This example illustrates copying between applications
without using the global clipboard. AppleScript picks the
target for an event by examining the first object reference in
the argument list. If the argument list contains references to
other applications, the values of the references are retrieved
and passed in place of the references in the argument list.
Standard events and reference structures are defined in
the Apple Event Registry. The Registry is divided into suites
that apply to domains of application. Suites contain spec-
ifications for classes and their properties, and events. Cur-
rently there are suites for core operations, text manipulation,
databases, graphics, collaboration, and word services (spell-
checking, etc.).
Jon Pugh, with a BSCS from Western Washington Uni-
versity in 1983, was in charge of the Apple Events registry.
He also helped out with quality assurance and evangelism.
Since then he has worked on numerous scriptable applica-
tions, and created “Jon’s Commands,” a shareware library of
AppleScript extensions.
Terminologies also provide natural-language names for
the four-letter codes used within an Apple Event. This meta-
data is stored in an application as a resource. As discussed in
Section 3.4 below, the terminology resources in an applica-
tion are used when parsing scripts targeting that application.
3.3 Programming Language Features
AppleScript’s programming language features include vari-
ables and assignment, control flow, and basic data struc-
tures. Control flow constructs include conditionals, a vari-
ety of looping constructs, subroutines, and exceptions. Sub-
routines allow positional, prepositional, and keyword para-
meters. Data structures include records, lists, and objects.
Destructuring bind, also known as pattern matching, can be
used to break apart basic data structures. Lists and records
are mutable. AppleScript also supports objects and a simple
transaction mechanism.
AppleScript has a simple object model. A script object
contains properties and methods. Methods are dynamically
dispatched, so script objects support a simple form of object-
oriented programming. The following simple script declara-
tion binds the name Counter to a new script object represent-
ing a counter:
script Counter
property count : 0
to increment
set count to count + 1
return count
end increment
end script
A script declaration can be placed inside a method to
create multiple instances. Such a method is called a factory
method, and is similar to a constructor method in Java.
Since script can access any lexically enclosing variables, all
the objects created by a factory have access to the state of
the object that constructed them. The resulting pattern of
object references resembles the class/metaclass system in
Smalltalk [23], although in much simpler form.
AppleScript’s object model is a prototype model similar
to that employed by Self [37], as opposed to the container-
based inheritance model of HyperTalk. Script objects sup-
port single inheritance by delegating unhandled commands
to the value in their parent property [36]. JavaScript later
adopted a model similar to AppleScript.
The top level of every script is an implicit object decla-
ration. Top-level properties are persistent, in that changes to
properties are saved when the application running the script
quits. A standalone script can be stored in a script file and
executed by opening it in the Finder. Such a script can di-
rect other applications to perform useful functions, but may
also call other script files. Thus, script files provide a simple
mechanism for modularity.
AppleScript provides no explicit support for threading or
synchronization. However, the application hosting a script
can invoke scripts from multiple threads: the execution
engine was thread-safe when run by the non-preemptive
scheduling in the original Macintosh OS. It is not safe when
1-10](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-18-320.jpg)
![English the first character of every word whose style is bold
Japanese
French le premier caractère de tous les mots dont style est gras
Professional { words | style == bold }.character[1]
Figure 5. Illustration of dialects.
run on multiple preemptive threads on Mac OS X. Script
objects can also be sent to another machine, making mobile
code possible.
3.4 Parsing and Internationalization
The AppleScript parser integrates the terminology of ap-
plications with its built-in language constructs. For ex-
ample, when targeting the Microsoft ExcelTM
application,
spreadsheet terms are known by the parser—nouns like cell
and formula, and verbs like recalculate . The statement
tell application ‘‘ Excel ’’ introduces a block in which
the Excel terminology is available. The terminology can
contain names that are formed from multiple words; this
means that the lexical analyzer must be able to recognize
multiple words as a single logical identifier. As a result,
lexical analysis depends upon the state of the parser: on en-
tering a tell block, the lexical analysis tables are modified
with new token definitions. The tables are reset when the
parser reaches the end of the block. This approach increases
flexibility but makes parsing more difficult. I believe the
added complexity in lexing/parsing makes it more difficult
for users to write scripts.
Apple also required that AppleScript, like most of its
other products, support localization, so that scripts could be
read and written in languages other than English. Scripts are
stored in a language-independent internal representation. A
dialect defines a presentation for the internal language. Di-
alects contain lexing and parsing tables, and printing rou-
tines. A script can be presented using any dialect—so a script
written using the English dialect can be viewed in Japanese.
Examples are given in Figure 5. For complete localization,
the application terminologies must also include entries for
multiple languages. Apple developed dialects for Japanese
and French. A “professional” dialect, which resembles Java,
was created but not released.
There are numerous difficulties in parsing a programming
language that resembles a natural language. For example,
Japanese does not have explicit separation between words.
This is not a problem for language keywords and names
from the terminology, but special conventions were required
to recognize user-defined identifiers. Other languages have
complex conjugation and agreement rules that are difficult to
implement. Nonetheless, the internal representation of Ap-
pleScript and the terminology resources contain information
to support these features.
The AppleScript parser was created using Yacc [29], a
popular LALR parser generator. Poor error messages are a
common problem with LALR parsing [1]. I wrote a tool
that produces somewhat better error messages by including
a simplified version of the follow set at the point where the
error occurred. The follow set was simplified by replacing
some common sets of symbols (like binary operators) with
a generic name, so that the error message would be “ex-
pected binary operator” instead of a list of every binary op-
erator symbol. Despite these improvements, obscure error
messages continue to be one of the biggest impediments to
using AppleScript.
3.5 AppleScript Implementation
During the design of AppleScript in mid-1991, we consid-
ered building AppleScript on top of an existing language
or runtime. We evaluated Macintosh Common Lisp (MCL),
Franz Lisp, and Smalltalk systems from ParcPlace and Dig-
italk. These were all good systems, but were not suitable as
a foundation for AppleScript for the same reason: there was
not sufficient separation between the development environ-
ment and the runtime environment. Separating development
from execution is useful because it a allows compiled script
to be executed in a limited runtime environment with low
overhead. The full environment would be needed only when
compiling or debugging a script.
Instead, we developed our own runtime and compiler. The
runtime includes a garbage collector and byte-code inter-
preter. The compiler and runtime were loaded separately to
minimize memory footprint.
One AppleScript T-shirt had the slogan “We don’t patch
out the universe”. Many projects at Apple were implemented
by “patching”: installing new functions in place of kernel
operating system functions. The operating system had no
protection mechanisms, so any function could be patched.
Patches typically had to be installed in a particular order, or
else they would not function. In addition, a bug in a patch
could cause havoc for a wide range of applications.
AppleScript did not patch any operating system func-
tions. Instead the system was carefully packaged as a thread-
safe QuickTime component. QuickTime components are a
lightweight dynamic library mechanism introduced by the
QuickTime team. Only one copy of the AppleScript com-
piler and runtime was loaded and shared by all applica-
tions on a machine. The careful packaging is one of the rea-
sons AppleScript was able to continue running unchanged
through numerous operating system upgrades, and even onto
the PowerPC.
1-11](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-19-320.jpg)
![Integers
Lists
Strings
Compile
GetSource
Display
Data
CoerceFromDesc
CoerceToDesc
Execute
Storage
Format
Load
Store
ExecuteEvent
Text
Objects
Commands
Script
Text
Data
Key: Interface
…
…
Scripts
Figure 6. Overview of the Open Scripting API.
The AppleScript runtime is implemented in C++. The en-
tire system, including dialects, is 118K lines of code, includ-
ing comments and header files. Compiling the entire Apple-
Script runtime took over an hour on the machines used by
the development team. After the alpha milestone, the devel-
opment team was not allowed to produce official builds for
the testing team. Instead, the code had to be checked in and
built by a separate group on a clean development environ-
ment. This process typically took 8 to 12 hours, because the
process was not fully automated, so there was sometimes a
significant lag between identifying a bug and delivering a fix
to the quality assurance team. This was a significant source
of frustration for the overall team.
4. Script Management
Open Scripting Architecture (OSA) allows any application
to manipulate and execute scripts [11]. The Open Scripting
API is centered around the notion of a script, as shown in
Figure 6. A script is either a data value or a program. Many
of the routines in the API are for translating between scripts
and various external formats. Compile parses script text and
creates a script object, while GetSource translates a script
object back into human-readable script text. Display trans-
lates a value into a printed representation. When applied to
a string, e.g. ‘‘ Gustav’’, GetSource returns a program lit-
eral ‘‘ Gustav’’, while Display just returns the text Gustav.
CoerceFromDesc and CoerceToDesc convert AppleScript
values to and from Apple Event descriptors. Load and Store
convert to/from compact binary byte-streams that can be in-
cluded in a file.
The Execute function runs a script in a context. A context
is a script that contains bindings for global variables.
At its simplest, the script management API supports the
construction of a basic script editor that can save scripts as
stand-alone script applications.
The OSA API does not include support for debugging,
although this was frequently discussed by the team. How-
ever, other companies have worked around this problem and
created effective debugging tools (Section 6.3).
4.1 Embedding
The script management API also supports attaching scripts
to objects in an existing application. Such scripts can be trig-
gered during normal use of the application. This usage is
supported by the ExecuteEvent function, which takes as in-
put a script and an Apple Event. The event is interpreted as a
method call to the script. The corresponding method decla-
ration in the script is invoked. In this way an application can
pass Apple Events to scripts that are attached to application
objects.
Embedded scripts allow default application behavior to
be customized, extended, or even replaced. For example, the
Finder can run a script whenever a file is added to a folder,
or an email package can run a script when new mail is re-
ceived. Scripts can also be attached to new or existing menu
items to add new functionality to an application. By em-
bedding a universal scripting languages, application devel-
opers do not need to build proprietary scripting languages,
and users do not need to learn multiple languages. Users
can also access multiple applications from a single script.
AppleScript demonstrated the idea that a single scripting
language could be used for all applications, while allowing
application-specific behaviors to be incorporated so that the
language was specialized for use in each application.
Embedding can also be used to create entire applications.
In this case there is no predefined application structure to
which scripts are attached. Instead, the user builds the ap-
plication objects — for data and user interfaces, and then at-
taches scripts to them. Several application development tools
based on AppleScript are described in Section 6.
4.2 Multiple Scripting Languages
Halfway through the development of AppleScript, Apple
management decided to allow third-party scripting lan-
guages to be used in addition to AppleScript. A new API
for managing scripts and scripting language runtime engines
had to be designed and implemented. These changes con-
tributed to delays in shipping AppleScript. However, they
also led to a more robust architecture for embedding.
In February of 1992, just before the first AppleScript al-
pha release, Dave Winer convinced Apple management that
having one scripting language would not be good for the
Macintosh. At that time, Dave Winer was an experienced
Macintosh developer, having created one of the first outliner
applications, ThinkTank. In the early 1990s, Dave created an
alternative scripting system, called Frontier. Before I joined
the project, Apple had discussed the possibility of buying
Frontier and using it instead of creating its own language.
For some reason the deal fell through, but Dave continued
developing Frontier. Apple does not like to take business
away from developers, so when Dave complained that the
impending release of AppleScript was interfering with his
product, Apple decided the AppleScript should be opened up
to multiple scripting languages. The AppleScript team mod-
1-12](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-20-320.jpg)
![ified the OSA APIs so that they could be implemented by
multiple scripting systems, not just AppleScript. As a result,
OSA is a generic interface between clients of scripting ser-
vices and scripting systems that support a scripting language.
Each script is tagged with the scripting system that created it,
so clients can handle multiple kinds of script without know-
ing which scripting system they belong to.
Dave Winer’s Frontier is a complete scripting and ap-
plication development environment that eventually became
available as an Open Scripting component. Dave went on
to participate in the design of web services and SOAP [4].
Tcl, JavaScript, Python and Perl have also been packaged as
Open Scripting components.
4.3 Recording Events as Scripts
The AppleScript infrastructure supports recording of events
in recordable applications, which publish events in response
to user actions. Donn Denman, a senior software engineer
on the AppleScript team with a BS in Computer Science
and Math from Antioch College, designed and implemented
much of the infrastructure for recording. At Apple he worked
on Basic interpreters. He was involved in some of the early
discussions of AppleScript, worked on Apple Events and ap-
plication terminologies in AppleScript. In addition, Donn
created MacroMaker, a low-level event record and play-
back system for Macintosh OS System 5. Working on Mar-
coMaker gave Donn a lot of experience in how recording
should work.
Recording allows automatic generation of scripts for re-
peated playback of what would otherwise be repetitive tasks.
Recorded scripts can be subsequently generalized by users
for more flexibility. This approach to scripting alleviates the
“staring at a blank page” syndrome that can be so crippling
to new scripters. Recording is also useful for learning the
terminology of basic operations of an application, because it
helps users to connect actions in the graphical interface with
their symbolic expression in script.
Recording high-level events is different from recording
low-level events of the graphical user interface. Low-level
events include mouse and keyboard events. Low-level events
can also express user interface actions, e.g. “perform Open
menu item in the File menu”, although the response to this
event is usually to display a dialog box, not to open a par-
ticular file. Additional low-level events are required to ma-
nipulate dialog boxes by clicking on interface elements and
buttons. Low-level events do not necessarily have the same
effect if played back on a different machine, or when dif-
ferent windows are open. High-level events are more robust
because they express the intent of an action more directly.
For example, a high-level event can indicate which file to
open.
Recording is supported by a special mode in the Apple
Event manager, based on the idea that a user’s actions in
manipulating a GUI interface can be described by a corre-
sponding Apple Event. For example, if the user selects the
File Open menu, then finds and selects a file named “Re-
sumé” in a folder named “Personal”, the corresponding Ap-
ple Event would be a FileOpen event containing the path
“Personal:Resumé”. To be recordable, an application must
post Apple Events that describe the actions a user performs
with the GUI.
Recordable applications can be difficult to build, since
they must post an Apple Event describing each operation
performed by a user. The AppleScript team promoted an ar-
chitecture that turned this difficulty into a feature. We advo-
cated that applications should be factored into two parts, a
GUI and a back end, where the only communication from
the GUI to the back end is via Apple Events. With this ar-
chitecture, all the core functionality of the application must
be exposed via Apple Events, so the application is inher-
ently scriptable. The GUI’s job becomes one of translating
low-level user input events (keystrokes and mouse move-
ments) into high-level Apple Events. An application built in
this way is inherently recordable; the Apple Event manager
simply records the Apple Events that pass from the GUI to
the back end. If an application is already scriptable, it can
be made recordable by arranging for the user interface to
communicate with the application model only through Ap-
ple Events.
The reality of recording is more complex, however. If
there is a type Apple Event to add characters into a docu-
ment, the GUI must forward each character immediately to
the back end so that the user will see the result of typing.
During recording, if the user types “Hello” the actions will
record an undesirable script:
type ‘‘ H”
type ‘‘ e”
type ‘‘ l”
type ‘‘ l”
type ‘‘ o”
It would be better to record type ‘‘ Hello”. To get this
effect, the GUI developer could buffer the typing and send
a single event. But then the user will not see the effect of
typing immediately. AppleEvents has the ability to specify
certain events as record-only, meaning that it is a summary
of a user’s actions and should not be executed. Creating such
summaries makes developing a recordable application quite
difficult.
In 2006 twenty-five recordable applications were listed
on Apple’s website and in the AppleScript Sourcebook [8],
one of several repositories of information about AppleScript.
Some, but fewer than half, of the major productivity appli-
cations are recordable. Recordable applications include Mi-
crosoft Word and Excel, Netscape Navigator, Quark Express
(via a plugin) and CorelDRAW.
One of the inherent difficulties of recording is the am-
biguity of object specification. As the language of events
becomes more rich, there may be many ways to describe a
given user action. Each version might be appropriate for a
1-13](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-21-320.jpg)
![given situation, but the system cannot pick the correct action
without knowing the intent of the user. For example, when
closing a window, is the user closing the front window or
the window specifically named “Example”? This is a well-
known problem in research on programming by example,
where multiple examples of a given action can be used to dis-
ambiguate the user’s intent. Allen Cypher did fundamental
research on this problem while at Apple. He built a prototype
system called Eager that anticipated user actions by watch-
ing previous actions [21, 22]. AppleScript does not have
built-in support for analyzing multiple examples. There are
also ambiguities when recording and embedding are com-
bined: if a recorded event causes a script to execute, should
the original event or the events generated by the script be
recorded? Application designers must decide what is most
appropriate for a given situation. Cypher worked with Dave
Curbow in writing guidelines to help developers make these
difficult choices [26].
Recording can also be used for other purposes. For ex-
ample, a help system can include step-by-step instructions
defined by a script. The steps can be played as normal
scripts, or the user can be given the option of performing
the steps manually under the supervision of the help system.
By recording the user’s actions, the help system can pro-
vide feedback or warnings when the user’s actions do not
correspond to the script.
5. Development Process
Unlike most programming languages, AppleScript was de-
signed within a commercial software development project.
The team members are listed in Figure 7. AppleScript was
designed by neither an individual nor a committee; the
team used a collaborative design process, with significant
user testing and evaluation. The project leaders guided the
process and made final decisions: there was lively debate
within the group about how things should work. The ex-
tended team included project management, user interface
design and testing, documentation, and product marketing.
The AppleScript project had a strong quality assurance
(QA) team. They created a large test suite which was run
against nightly builds. From a management viewpoint, the
QA group also had significant control over the project, be-
cause they were required to give final approval of a release.
The project was code-named “Gustav” after Donn’s mas-
sive Rottweiler dog. The dog slimed everything it came in
contact with, and was the impetus behind a T-shirt that read
“Script Happens”. The project tag line was “Pure Guava”
because Gary Bond was designing a t-shirt that said “Apple-
Script: Pure Gold” and Warren Harris got him to change it
to Pure Guava after the Ween album he was in love with at
the time.
AppleScript and the associated tools were designed and
implemented between 1990 and 1993. Figure 8 gives a time-
line of the development process. The line labeled “changes”
Jens Alfke
Developer
Greg Anderson
Developer,
Scriptable Finder
Mike Askins
Engineering Project Man-
ager
Gary Bond
QA
Scott Bongiorno
QA, User Testing
B. Bruce Brinson
Developer
Kevin Calhoun
Manager
Jennifer Chaffee
User Interface Design
Dan Clifford
Developer
William Cook
Architect, Developer,
Manager
Sean Cotter
Documentation
Dave Curbow
User Interface Design
Donn Denman
Developer
Sue Dumont
Developer, QA
Mike Farr
Marketing
Mitch Gass
Documentation
Laura Clark Hamersley
Marketing
Warren Harris
Architect, Developer
Eric House QA, Developer
Ron Karr
QA, Apple Events Devel-
oper
Edmund Lai
Developer, Apple Events
Ron Lichty
Manager, Finder
Bennet Marks
Developer
Mark Minshull
Manager
Kazuhisa Ohta
Developer, Dialects
Donald Olson
QA, Manager
Chuck Piercey
Marketing
Kurt Piersol
Architect
James Redfern
QA, Developer
Brett Sher
Developer, QA
Laile Di Silvestro
QA, Developer
Sal Soghoian
Product Manager
Francis Stanbach
Developer,
Scriptable Finder
Kazuhiko Tateda
Japanese Dialect
Larry Tesler
Manager, VP
Mark Thomas
Evangelist
Susan Watkins
Marketing
Figure 7. AppleScript and related project team members.
1-14](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-22-320.jpg)
![inheritance, lexical closures, etc.), because the goal for this
aspect of the language was to apply well-known constructs,
not define new ones. There was no standard formal seman-
tic framework for the novel aspects of AppleScript, espe-
cially the notion of references for access to external objects
residing in an application. The project did not have the lux-
ury of undertaking extensive research into semantic founda-
tions; its charter was to develop and ship a practical language
in a short amount of time. Sketches of a formal semantics
were developed, but the primary guidance for language de-
sign came from solving practical problems and from user
studies, rather than from a priori formal analysis.
The public documentation was developed by professional
writers who worked closely with the team throughout the
project. The primary document is Inside Macintosh: Inter-
application Communication, which includes details on the
Apple Event Manager and Scripting Components [11]. The
AppleScript language is also thoroughly documented [2],
and numerous third-party books have been written about
it, for examples see [31, 24]. Mitch Gass and Sean Cotter
documented Apple Events and AppleScript for external use.
Mitch has a bachelor’s degrees in comparative literature and
computer science, and worked at Tandem and Amiga before
joining Apple. Mitch worked during the entire project to
provide that documentation, and in the process managed to
be a significant communication point for the entire team.
5.2 User Testing
Following Apple’s standard practice, we user-tested the lan-
guage in a variety of ways. We identified novice users and
asked them, “What do you think this script does?” The fol-
lowing questions illustrate the kinds of questions asked dur-
ing user testing.
Part I. Please answer the following multiple choice ques-
tions about AppleScript.
3. Given the handler:
on doit from x to y with z
return (x ∗ y) + z
end doit
What does the following statement evaluate to?
doit with 3 from 8 to 5
a) 29
b) 43
c) error
d) other:
Part II. Please state which of the following AppleScript
statements you prefer.
8. a) put ‘‘a ’’, {‘‘b ’’, ‘‘ c’’} into x
b) put {‘‘a ’’, {‘‘b ’’, ‘‘ c’’}} into x
9. a) window named ‘‘fred’’
b) window ‘‘fred ’’
10. a) window 1
b) window #1
11. a) word 34
b) word #34
12. a) ‘‘ apple ’’ < ‘‘betty ’’
b) ‘‘ apple ’’ comes before ‘‘betty ’’
Part III. This section shows sequences of commands and
then asks questions about various variables after they are
executed.
15. Given the commands:
put {1, 2, 3} into x
put x into y
put 4 into item 1 of x
What is x?
a) {1, 2, 3}
b) {4, 2, 3}
c) error
d) other:
What is y?
a) {1, 2, 3}
b) {4, 2, 3}
c) error
d) other:
Part IV. In this section, assume that all AppleScript state-
ments refer to window 1, which contains the following
text:
this is a test
of the emergency broadcast system
18. What does the following statement evaluate to?
count every line of window 1
a) 2
b) 4, 5
c) 9
d) 14, 33
e) 47
f) error
g) other:
1-16](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-24-320.jpg)
![a task that took a team of twenty up to a month for a single
catalog.
An AppleScript script automates the entire process. The
script reads the price and descriptive text from the FileMaker
Pro database and inserts it into appropriate QuarkXPress
fields. The script applies special formatting: it deletes the
decimal point in the prices and superscripts the cents (e.g.
3499
). To make the text fit precisely in the width of the en-
closing box, the script computes a fractional expansion fac-
tor for the text by dividing the width of the box by the width
of the text (this task was previously done with a calculator).
It adjusts colors and sets the first line of the description in
boldface type. Finally, it adds special markers like “Buy 2
get 1 free” and “Sale price $1799
” where specified by the
database.
Once this process is automated, one person can produce
the entire catalog in under a day, a tiny fraction of the time
taken by the manual process. It also reduced errors during
copying and formatting. Of course, creating and maintaining
the scripts takes time, but the overall time is significantly
reduced over the long run.
6.3 Scripting Tools
AppleScript included a simple and elegant script editor cre-
ated by Jens Alfke, who had graduated from Caltech and
worked with Kurt Piersol at Xerox Parc on Smallktalk-80
applications. Jens was one of the key developers on the Ap-
pleScript team; he focused on tools, consistency of the APIs
and usability of the overall system.
Soon after AppleScript was released, more powerful
script development environments were created outside Ap-
ple. They addressed one of the major weaknesses of Apple-
Script: lack of support for debugging. One developer outside
Apple who took on this challenge is Cal Simone, who has
also been an unofficial evangelist for AppleScript since its
inception. Cal created Scripter, which allows users to single-
step through a script. It works by breaking a script up into
individual lines that are compiled and executed separately.
The enclosing tell statements are preserved around each
line as it is executed. Scripter also allows inspection of local
variables and execution of immediate commands within the
context of the suspended script. Script Debugger uses a dif-
ferent technique: it adds a special Apple Event between each
line of a script. The Apple Event is caught by the debugger
and the processing of the script is suspended. The current
values of variables can then be inspected. To continue the
script, the debugger simply returns from the event.
AppleScript also enables creation of sophisticated inter-
face builders. The interface elements post messages when
a user interacts with them. The user arranges the elements
into windows, menus, and dialogs. Scripts may be attached
to any object in the interface to intercept the messages be-
ing sent by the interface elements and provide sophisticated
behavior and linking between the elements. Early applica-
tion builders included FrontmostTM
, a window and dialog
builder, and AgentBuilderTM
, which specialized in commu-
nication front-ends. Version 2.2 of HyperCard, released in
1992, included support for OSA, so that AppleScript or any
OSA language could be used in place of HyperTalk.
Two major application builders have emerged recently.
FaceSpan, originally released in 1994, has grown into a full-
featured application development tool. FaceSpan includes
an integrated script debugger. Apple released AppleScript
Studio in 2002 as part of its XCode development platform.
A complete application can be developed with a wide range
of standard user interface elements to which scripts can be
attached. AppleScript Studio won Macworld Best of Show
Awards at the 2001 Seybold Conference in San Francisco.
In 2005 Apple released Automator, a tool for creating
sequences of actions that define workflows. Automator se-
quences are not stored or executed as AppleScripts, but can
contain AppleScripts as primitive actions. The most inter-
esting thing about Automator is that each action has an input
and an output, much like a command in a Unix pipe. The
resulting model is quite intuitive and easy to use for simple
automation tasks.
Although Apple Events are normally handled by appli-
cations, it is also possible to install system event handlers.
When an Apple Event is delivered to an application, the ap-
plication may handle the event or indicate that it was not han-
dled. When an application does not handle an event, the Ap-
ple Event manager searches for a system event handler. Sys-
tem event handlers are packaged in script extensions (also
known as OSAX) and are installed on the system via Script-
ing Additions that are loaded when the system starts up.
6.4 Scriptable Applications
Eventually, a wide range of scriptable applications became
available: there are currently 160 scriptable applications
listed on the Apple web site and the AppleScript source-
book [8]. Every kind of application is present, including
word processors, databases, file compression utilities, and
development tools. Many of the most popular applications
are scriptable, including Microsoft Office, Adobe Photo-
shop, Quark Expression, FileMaker, and Illustrator. In ad-
dition, most components of the Mac OS are scriptable,
including the Finder, QuickTime Player, Address Book,
iTunes, Mail, Safari Browser, AppleWorks, DVD Player,
Help Viewer, iCal, iChat, iSync, iPhoto, and control panels.
Other systems also benefitted from the infrastructure cre-
ated by AppleScript. The Macintosh AVTM
speech recogni-
tion system uses AppleScript, so any scriptable application
can be driven using speech.
7. Evolution
After version 1.1, the evolution of AppleScript was driven
primarily by changes in the Macintosh OS. Since Apple-
Script was first released, the operating system has undergone
two major shifts, first when Apple moved from the Motorola
1-18](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-26-320.jpg)
![68000 to the PowerPC chip, and then when it moved from
the Classic Macintosh OS to the Unix-based OS X. Few
changes were made to the language itself, while scriptable
applications and operating system components experienced
rapid expansion and evolution. A detailed history with dis-
cussion of new features, bugs, and fixes can be found in the
AppleScript Sourcebook [8], which we summarize here.
The first upgrade to AppleScript, version 1.1.2, was cre-
ated for Macintosh OS 8.0, introduced in July 1997. De-
spite the rigorous source code configuration process (see
Section 5), Apple could not figure out how to compile the
system and contracted with Warren Harris to help with the
job. A number of bugs were fixed and some small enhance-
ments were made to conform to Macintosh OS 8.0 standards.
At the same time several system applications and extensions
were changed in ways that could break old scripts. The most
important improvement was a new scriptable Finder, which
eliminated the need for a Finder scripting extension.
In 1997 AppleScript was at the top of the list of features
to eliminate in order to save money. Cal Simone, mentioned
in Section 6.3, successfully rallied customers to rescue Ap-
pleScript.
In October 1998 Apple released AppleScript 1.3 with
UNICODE support recompiled as a native PowerPC exten-
sion; however, the Apple Events Manager was still emulated
as Motorola 68000 code. The dialect feature was no longer
supported; English became the single standard dialect. This
version came much closer to realizing the vision of uni-
form access to all system resources from scripts. At least
30 different system components, including File Sharing, Ap-
ple Video Player and Users & Groups, were now scriptable.
New scriptable applications appeared as well, including Mi-
crosoft Internet Explorer and Outlook Express.
The PowerPC version of AppleScript received an Eddy
Award from MacWorld as “Technology of the Year” for
1998 and was also demonstrated in Steve Jobs’ Seybold
1998 address. In 2006, MacWorld placed AppleScript as
#17 on its list of the 30 most significant Mac products ever.
AppleScript was a long-term investment in fundamental in-
frastructure that took many years to pay dividends.
The most significant language changes involved the tell
statement. For example, the machine class used to identify
remote applications was extended to accept URLs (see Sec-
tion 3.2), allowing AppleScript control of remote applica-
tions via TCP/IP.
When Mac OS X was released in March 2001, it included
AppleScript 1.6. In porting applications and system compo-
nents to OS X, Apple sometimes sacrificed scripting support.
As a result, there was a significant reduction in the number of
scriptable applications after the release of OS X. Full script-
ability is being restored slowly in later releases.
In October 2006, Google reported an estimated 8,570,000
hits for the word “AppleScript”.
8. Evaluation
AppleScript was developed by a small group with a short
schedule, a tight budget and a big job. There was neither
time nor money to fully research design choices.
AppleScript and Apple Events introduced a new approach
to remote communication in high-latency environments [33].
Object references are symbolic paths, or queries, that iden-
tify one or more objects in an application. When a command
is applied to an object reference, both the command and the
object reference are sent (as an Apple Event containing an
object specifier) to the application hosting the target object.
The application interprets the object specifier and then per-
forms the action on the specified objects.
In summary, AppleScript views an application as a form
of object-oriented database. The application publishes a
specialized terminology containing verbs and nouns that
describe the logical structure and behavior of its objects.
Names in the terminology are composed using a standard
query language to create programs that are executed by the
remote application. The execution model does not involve
remote object references and proxies as in CORBA. Rather
than send each field access and method individually to the
remote application and creating proxies to represent inter-
mediate values, AppleScript sends the entire command to
the remote application for execution. From a pure object-
oriented viewpoint, the entire application is the only real ob-
ject; the “objects” within it are identified only by symbolic
references, or queries.
After completing AppleScript, I learned about COM
and was impressed with its approach to distributed object-
oriented programming. Its consistent use of interfaces en-
ables interoperability between different systems and lan-
guages. Although interface negotiation is complex, invoking
a method through an interface is highly optimized. This ap-
proach allows fine-grained objects that are tightly coupled
through shared binary interfaces. For many years I believed
that COM and CORBA would beat the AppleScript com-
munication model in the long run. However, recent develop-
ments have made me realize that this may not be the case.
AppleScript uses a large-granularity messaging model
that has many similarities to the web service standards that
began to emerge in 1999 [10]. Both are loosely coupled and
support large-granularity communication. Apple Events data
descriptors are similar to XML in that they describe arbi-
trary labeled tree structures without fixed semantics. Apple-
Script terminologies are similar to web service description
language (WSDL) files. It is perhaps not an accident that
Dave Winer, who worked extensively with AppleScript and
Apple Events, is also one of the original developers of web
service models. There may be useful lessons to be learned
for web services, given that AppleScript represents a sig-
nificant body of experience with large-granularity messag-
ing. One difference is that AppleScript includes a standard
query model for identifying remote objects. A similar ap-
1-19](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-27-320.jpg)
![proach could be useful for web services. As I write in 2006,
I suspect that COM and CORBA will be overwhelmed by
web services, although the outcome is far from certain now.
AppleScript is also similar to traditional database inter-
faces like ODBC [38]. In AppleScript the query model is
integrated directly into the language, rather than being exe-
cuted as strings as in ODBC. A similar approach has been
adopted by Microsoft for describing queries in .NET lan-
guages [3].
User tests revealed that casual users don’t easily under-
stand the idea of references, or having multiple references to
the same value. It is easier to understand a model in which
values are copied or moved, rather than assigning references.
The feedback from user tests in early 1993 was too late in the
development cycle to address this issue with anything more
than a cosmetic change, to use copy instead of set for as-
signment.
Writing scriptable applications is difficult. Just as user
interface design requires judgment and training, creating a
good scripting interface requires a lot of knowledge and
careful design. It is too difficult for application developers
to create terminologies that work well in the naturalistic
grammar. They must pay careful attention to the linguistic
properties of the names they choose.
The experiment in designing a language that resembled
natural languages (English and Japanese) was not success-
ful. It was assumed that scripts should be presented in “nat-
ural language” so that average people could read and write
them. This lead to the invention of multi-token keywords
and the ability to disambiguate tokens without spaces for
Japanese Kanji. In the end the syntactic variations and flex-
ibility did more to confuse programmers than to help them
out. It is not clear whether it is easier for novice users to
work with a scripting language that resembles natural lan-
guage, with all its special cases and idiosyncrasies. The main
problem is that AppleScript only appears to be a natural
language: in fact, it is an artificial language, like any other
programming language. Recording was successful, but even
small changes to the script may introduce subtle syntactic er-
rors that baffle users. It is easy to read AppleScript, but quite
hard to write it.
When writing programs or scripts, users prefer a more
conventional programming language structure. Later ver-
sions of AppleScript dropped support for dialects. In hind-
sight, we believe that AppleScript should have adopted the
Professional Dialect that was developed but never shipped.
Finally, readability was no substitute for an effective se-
curity mechanism. Most people just run scripts—they don’t
read or write them.
9. Conclusion
AppleScript is widely used today and is a core technol-
ogy of Mac OS X. Many applications, including Quark Ex-
press, Microsoft Office, and FileMaker, support scripting.
Small scripts are used to automate repetitive tasks. Larger
scripts have been developed for database publishing, docu-
ment preparation, and even web applications.
There are many interesting lessons to be learned from
AppleScript. On a technical level, its model of pluggable
embedded scripting languages has become commonplace.
The communication mechanism of Apple Events, which is
certainly inferior to RPC mechanisms for single-machine or
in-process interactions, may turn out to be a good model
for large-granularity communication models such as web
services. Many of the current problems in AppleScript can
be traced to the use of syntax based on natural language;
however, the ability to create pluggable dialects may provide
a solution in the future, by creating a new syntax based on
conventional programming languages.
Acknowledgments
Thanks to Jens Alfke, Paul Berkowitz, Bill Cheeseman,
Chris Espinosa, Michael Farr, Steve Goldband Tom Ham-
mer, David Hoerl, Alexander Kellett, Wayne Malkin, Matt
Neuburg, Chuck Piercey, Hamish Sanderson, and Stephen
Weyl, for discussions about this paper. Special thanks to
Andrew Black and Kathleen Fisher for their guidance, en-
couragement, flexibility, and careful reading of my work in
progress.
References
[1] Alfred V. Aho and Jeffrey D. Ullman. Principles of
Compiler Design (Addison-Wesley series in computer science
and information processing). Addison-Wesley Longman
Publishing Co., Inc., Boston, MA, USA, 1977.
[2] Apple Computer Inc. AppleScript Language Guide. Addison-
Wesley, 1993.
[3] Gavin Bierman, Erik Meijer, and Wolfram Schulte. The
essence of data access in cω. In European Conference on
Object-Oriented Programming. Springer Verlag, 2005.
[4] Don Box, David EhneBuske, Gopal Kakivaya, Andrew
Layman, Noah Mendelsohn, Henrik Frystyk Nielson, Satish
Thatte, and Dave Winer. Simple object access protocol 1.1.
http://www.w3.org/TR/SOAP.
[5] Gilad Bracha and William Cook. Mixin-based inheritance.
In Proc. of ACM Conf. on Object-Oriented Programming,
Systems, Languages and Applications, pages 303–311, 1990.
[6] Peter Canning, William Cook, Walt Hill, John Mitchell,
and Walter Olthoff. F-bounded polymorphism for object-
oriented programming. In Proc. of Conf. on Functional
Programming Languages and Computer Architecture, pages
273–280, 1989.
[7] Peter Canning, William Cook, Walt Hill, and Walter Olthoff.
Interfaces for strongly-typed object-oriented programming.
In Proc. of ACM Conf. on Object-Oriented Programming,
Systems, Languages and Applications, pages 457–467, 1989.
[8] Bill Cheeseman. Applescript sourcebook. http://www.
AppleScriptSourcebook.com.
1-20](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-28-320.jpg)
![[9] Peter P. Chen. The entity-relationship model — toward a
unified view of data. ACM Transactions on Database Systems
(TODS), 1(1):9–36, 1976.
[10] Roberto Chinnici, Martin Gudgin, Jean-Jacques Moreau, and
Sanjiva Weerawarana. Web Services Description Language
Version 1.2, July 2002. W3C Working Draft 9.
[11] Apple Computer. Inside Macintosh: Interraplication Com-
munication. Addison-Wesley, 1991.
[12] Apple Computer. Inside Macintosh: Macintosh Toolbox
Essentials. Addison-Wesley, 1991.
[13] Apple Computer. ichat 2.0 dictionary. FooDoo Lounge web
site by Richard Morton, 2002-2005.
[14] William Cook. A Denotational Semantics of Inheritance.
PhD thesis, Brown University, 1989.
[15] William Cook. A proposal for making Eiffel type-safe. In
Proc. European Conf. on Object-Oriented Programming,
pages 57–70. British Computing Society Workshop Series,
1989. Also in The Computer Journal, 32(4):305–311, 1989.
[16] William Cook. Object-oriented programming versus abstract
data types. In Proc. of the REX Workshop/School on the
Foundations of Object-Oriented Languages, volume 173 of
Lecture Notes in Computer Science. Springer-Verlag, 1990.
[17] William Cook. Interfaces and specifications for the Smalltalk
collection classes. In Proc. of ACM Conf. on Object-Oriented
Programming, Systems, Languages and Applications, 1992.
[18] William Cook, Walt Hill, and Peter Canning. Inheritance is
not subtyping. In Proc. of the ACM Symp. on Principles of
Programming Languages, pages 125–135, 1990.
[19] William Cook and Jens Palsberg. A denotational semantics
of inheritance and its correctness. In Proc. of ACM Conf.
on Object-Oriented Programming, Systems, Languages and
Applications, pages 433–444, 1989.
[20] William R. Cook and Victor Law. An algorithm editor for
structured design (abstract). In Proc. of the ACM Computer
Science Conference, 1983.
[21] Allen Cypher. Eager: programming repetitive tasks by
example. In CHI ’91: Proceedings of the SIGCHI conference
on Human factors in computing systems, pages 33–39, New
York, NY, USA, 1991. ACM Press.
[22] Allen Cypher, editor. Watch What I Do – Programming by
Demonstration. MIT Press, Cambridge, MA, USA, 1993.
Full text available at web.media.mit.edu/ lieber/PBE/.
[23] A. Goldberg and D. Robson. Smalltalk-80: the Language and
Its Implementation. Addison-Wesley, 1983.
[24] A. Goldstein. AppleScript: The Missing Manual. O’Reilly,
2005.
[25] Warren Harris. Abel posthumous report. HP Labs, 1993.
[26] Apple Computer Inc. Scripting interface guidelines. Techni-
cal Report TN2106, Apple Computer Inc.
[27] Apple Computer Inc. HyperCard User’s Guide. Addison
Wesley, 1987.
[28] Apple Computer Inc. HyperCard Script Language Guide:
The HyperTalk Language. Addison Wesley, 1988.
[29] Steven C. Johnson. Yacc: Yet another compiler compiler.
In UNIX Programmer’s Manual, volume 2, pages 353–387.
Holt, Rinehart, and Winston, New York, NY, USA, 1979.
[30] S. Michel. HyperCard: The Complete Reference. Osborne
Mc-GrawHill, 1989.
[31] M. Neuburg. AppleScript : The Definitive Guide. O’Reilly,
2003.
[32] Object Management Group. OMG Unified Modeling
Language Specification Version 1.5, March 2003.
[33] David A. Patterson. Latency lags bandwith. Commun. ACM,
47(10):71–75, 2004.
[34] Trygve Reenskaug. Models — views — controllers.
Technical report, Xerox PARC, December 1979.
[35] Alan Snyder. The essence of objects: Concepts and terms.
IEEE Softw., 10(1):31–42, 1993.
[36] Lynn Andrea Stein, Henry Lieberman, and David Ungar.
A shared view of sharing: The treaty of orlando. In Won
Kim and Frederick H. Lochovsky, editors, Object-Oriented
Concepts, Databases and Applications, pages 31–48. ACM
Press and Addison-Wesley, 1989.
[37] D. Ungar and R.B. Smith. Self: The power of simplicity.
In Proc. of ACM Conf. on Object-Oriented Programming,
Systems, Languages and Applications, pages 227–242, 1987.
[38] Murali Venkatrao and Michael Pizzo. SQL/CLI – a new
binding style for SQL. SIGMOD Record, 24(4):72–77, 1995.
[39] Steven R. Wood. Z — the 95% program editor. In
Proceedings of the ACM SIGPLAN SIGOA symposium on
Text manipulation, pages 1–7, New York, NY, USA, 1981.
ACM Press.
1-21](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-29-320.jpg)
![The Evolution of Lua
Roberto Ierusalimschy
Department of Computer Science,
PUC-Rio, Rio de Janeiro, Brazil
roberto@inf.puc-rio.br
Luiz Henrique de Figueiredo
IMPA–Instituto Nacional de
Matemática Pura e Aplicada, Brazil
lhf@impa.br
Waldemar Celes
Department of Computer Science,
PUC-Rio, Rio de Janeiro, Brazil
celes@inf.puc-rio.br
Abstract
We report on the birth and evolution of Lua and discuss how
it moved from a simple configuration language to a versatile,
widely used language that supports extensible semantics,
anonymous functions, full lexical scoping, proper tail calls,
and coroutines.
Categories and Subject Descriptors K.2 [HISTORY OF
COMPUTING]: Software; D.3 [PROGRAMMING LAN-
GUAGES]
1. Introduction
Lua is a scripting language born in 1993 at PUC-Rio, the
Pontifical Catholic University of Rio de Janeiro in Brazil.
Since then, Lua has evolved to become widely used in all
kinds of industrial applications, such as robotics, literate
programming, distributed business, image processing, exten-
sible text editors, Ethernet switches, bioinformatics, finite-
element packages, web development, and more [2]. In par-
ticular, Lua is one of the leading scripting languages in game
development.
Lua has gone far beyond our most optimistic expecta-
tions. Indeed, while almost all programming languages come
from North America and Western Europe (with the notable
exception of Ruby, from Japan) [4], Lua is the only language
created in a developing country to have achieved global rel-
evance.
From the start, Lua was designed to be simple, small,
portable, fast, and easily embedded into applications. These
design principles are still in force, and we believe that they
account for Lua’s success in industry. The main characteris-
tic of Lua, and a vivid expression of its simplicity, is that it
offers a single kind of data structure, the table, which is the
Lua term for an associative array [9]. Although most script-
ing languages offer associative arrays, in no other language
do associative arrays play such a central role. Lua tables
provide simple and efficient implementations for modules,
prototype-based objects, class-based objects, records, arrays,
sets, bags, lists, and many other data structures [28].
In this paper, we report on the birth and evolution of Lua.
We discuss how Lua moved from a simple configuration
language to a powerful (but still simple) language that sup-
ports extensible semantics, anonymous functions, full lexical
scoping, proper tail calls, and coroutines. In §2 we give an
overview of the main concepts in Lua, which we use in the
other sections to discuss how Lua has evolved. In §3 we re-
late the prehistory of Lua, that is, the setting that led to its
creation. In §4 we relate how Lua was born, what its original
design goals were, and what features its first version had. A
discussion of how and why Lua has evolved is given in §5.
A detailed discussion of the evolution of selected features
is given in §6. The paper ends in §7 with a retrospective of
the evolution of Lua and in §8 with a brief discussion of the
reasons for Lua’s success, especially in games.
2. Overview
In this section we give a brief overview of the Lua language
and introduce the concepts discussed in §5 and §6. For a
complete definition of Lua, see its reference manual [32].
For a detailed introduction to Lua, see Roberto’s book [28].
For concreteness, we shall describe Lua 5.1, which is the
current version at the time of this writing (April 2007), but
most of this section applies unchanged to previous versions.
Syntactically, Lua is reminiscent of Modula and uses
familiar keywords. To give a taste of Lua’s syntax, the code
below shows two implementations of the factorial function,
one recursive and another iterative. Anyone with a basic
knowledge of programming can probably understand these
examples without explanation.
function factorial(n) function factorial(n)
if n == 0 then local a = 1
return 1 for i = 1,n do
else a = a*i
return n*factorial(n-1) end
end return a
end end
Permission to make digital/hard copy of part of this work for personal or
classroom use is granted without fee provided that the copies are not
made or distributed for profit or commercial advantage, the copyright
notice, the title of the publication, and its date of appear, and notice is
given that copying is by permission of the ACM, Inc. To copy
otherwise, to republish, to post on servers, or to redistribute to lists,
requires prior specific permission and/or a fee. Permission may be
requested from the Publications Dept., ACM, Inc., 2 Penn Plaza, New
York, NY 11201-0701, USA, fax:+1(212) 869-0481,
permissions@acm.org
©2007 ACM 978-1-59593-766-7/2007/06-ART2 $5.00
DOI 10.1145/1238844.1238846
http://doi.acm.org/10.1145/1238844.1238846
2-1](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-30-320.jpg)
![Semantically, Lua has many similarities with Scheme,
even though these similarities are not immediately clear be-
cause the two languages are syntactically very different. The
influence of Scheme on Lua has gradually increased during
Lua’s evolution: initially, Scheme was just a language in the
background, but later it became increasingly important as
a source of inspiration, especially with the introduction of
anonymous functions and full lexical scoping.
Like Scheme, Lua is dynamically typed: variables do not
have types; only values have types. As in Scheme, a variable
in Lua never contains a structured value, only a reference to
one. As in Scheme, a function name has no special status in
Lua: it is just a regular variable that happens to refer to a
function value. Actually, the syntax for function definition
‘function foo() · · · end’ used above is just syntactic
sugar for the assignment of an anonymous function to a
variable: ‘foo = function () · · · end’. Like Scheme, Lua
has first-class functions with lexical scoping. Actually, all
values in Lua are first-class values: they can be assigned
to global and local variables, stored in tables, passed as
arguments to functions, and returned from functions.
One important semantic difference between Lua and
Scheme — and probably the main distinguishing feature of
Lua — is that Lua offers tables as its sole data-structuring
mechanism. Lua tables are associative arrays [9], but with
some important features. Like all values in Lua, tables are
first-class values: they are not bound to specific variable
names, as they are in Awk and Perl. A table can have any
value as key and can store any value. Tables allow sim-
ple and efficient implementation of records (by using field
names as keys), sets (by using set elements as keys), generic
linked structures, and many other data structures. Moreover,
we can use a table to implement an array by using natural
numbers as indices. A careful implementation [31] ensures
that such a table uses the same amount of memory that an
array would (because it is represented internally as an actual
array) and performs better than arrays in similar languages,
as independent benchmarks show [1].
Lua offers an expressive syntax for creating tables in
the form of constructors. The simplest constructor is the
expression ‘{}’, which creates a new, empty table. There are
also constructors to create lists (or arrays), such as
{"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"}
and to create records, such as
{lat= -22.90, long= -43.23, city= "Rio de Janeiro"}
These two forms can be freely mixed. Tables are indexed
using square brackets, as in ‘t[2]’, with ‘t.x’ as sugar for
‘t["x"]’.
The combination of table constructors and functions
turns Lua into a powerful general-purpose procedural data-
description language. For instance, a bibliographic database
in a format similar to the one used in BibTEX [34] can be
written as a series of table constructors such as this:
article{"spe96",
authors = {"Roberto Ierusalimschy",
"Luiz Henrique de Figueiredo",
"Waldemar Celes"},
title = "Lua: an Extensible Extension Language",
journal = "Software: Practice & Experience",
year = 1996,
}
Although such a database seems to be an inert data file,
it is actually a valid Lua program: when the database is
loaded into Lua, each item in it invokes a function, because
‘article{· · ·}’ is syntactic sugar for ‘article({· · ·})’,
that is, a function call with a table as its single argument.
It is in this sense that such files are called procedural data
files.
We say that Lua is an extensible extension language [30].
It is an extension language because it helps to extend ap-
plications through configuration, macros, and other end-user
customizations. Lua is designed to be embedded into a host
application so that users can control how the application be-
haves by writing Lua programs that access application ser-
vices and manipulate application data. It is extensible be-
cause it offers userdata values to hold application data and
extensible semantics mechanisms to manipulate these values
in natural ways. Lua is provided as a small core that can be
extended with user functions written in both Lua and C. In
particular, input and output, string manipulation, mathema-
tical functions, and interfaces to the operating system are all
provided as external libraries.
Other distinguishing features of Lua come from its im-
plementation:
Portability: Lua is easy to build because it is implemented
in strict ANSI C.1
It compiles out-of-the-box on most
platforms (Linux, Unix, Windows, Mac OS X, etc.), and
runs with at most a few small adjustments in virtually
all platforms we know of, including mobile devices (e.g.,
handheld computers and cell phones) and embedded mi-
croprocessors (e.g., ARM and Rabbit). To ensure porta-
bility, we strive for warning-free compilations under as
many compilers as possible.
Ease of embedding: Lua has been designed to be easily
embedded into applications. An important part of Lua is
a well-defined application programming interface (API)
that allows full communication between Lua code and
external code. In particular, it is easy to extend Lua by
exporting C functions from the host application. The API
allows Lua to interface not only with C and C++, but also
with other languages, such as Fortran, Java, Smalltalk,
Ada, C# (.Net), and even with other scripting languages
(e.g., Perl and Ruby).
1 Actually, Lua is implemented in “clean C”, that is, the intersection of C
and C++. Lua compiles unmodified as a C++ library.
2-2](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-31-320.jpg)
![Small size: Adding Lua to an application does not bloat it.
The whole Lua distribution, including source code, doc-
umentation, and binaries for some platforms, has always
fit comfortably on a floppy disk. The tarball for Lua 5.1,
which contains source code, documentation, and exam-
ples, takes 208K compressed and 835K uncompressed.
The source contains around 17,000 lines of C. Under
Linux, the Lua interpreter built with all standard Lua li-
braries takes 143K. The corresponding numbers for most
other scripting languages are more than an order of mag-
nitude larger, partially because Lua is primarily meant to
be embedded into applications and so its official distri-
bution includes only a few libraries. Other scripting lan-
guages are meant to be used standalone and include many
libraries.
Efficiency: Independent benchmarks [1] show Lua to be
one of the fastest languages in the realm of interpreted
scripting languages. This allows application developers
to write a substantial fraction of the whole application
in Lua. For instance, over 40% of Adobe Lightroom is
written in Lua (that represents around 100,000 lines of
Lua code).
Although these are features of a specific implementation,
they are possible only due to the design of Lua. In particular,
Lua’s simplicity is a key factor in allowing a small, efficient
implementation [31].
3. Prehistory
Lua was born in 1993 inside Tecgraf, the Computer Graph-
ics Technology Group of PUC-Rio in Brazil. The cre-
ators of Lua were Roberto Ierusalimschy, Luiz Henrique
de Figueiredo, and Waldemar Celes. Roberto was an assis-
tant professor at the Department of Computer Science of
PUC-Rio. Luiz Henrique was a post-doctoral fellow, first at
IMPA and later at Tecgraf. Waldemar was a Ph.D. student in
Computer Science at PUC-Rio. All three were members of
Tecgraf, working on different projects there before getting
together to work on Lua. They had different, but related,
backgrounds: Roberto was a computer scientist interested
mainly in programming languages; Luiz Henrique was a
mathematician interested in software tools and computer
graphics; Waldemar was an engineer interested in appli-
cations of computer graphics. (In 2001, Waldemar joined
Roberto as faculty at PUC-Rio and Luiz Henrique became a
researcher at IMPA.)
Tecgraf is a large research and development laboratory
with several industrial partners. During the first ten years
after its creation in May 1987, Tecgraf focused mainly on
building basic software tools to enable it to produce the inter-
active graphical programs needed by its clients. Accordingly,
the first Tecgraf products were drivers for graphical termi-
nals, plotters, and printers; graphical libraries; and graphical
interface toolkits. From 1977 until 1992, Brazil had a pol-
icy of strong trade barriers (called a “market reserve”) for
computer hardware and software motivated by a national-
istic feeling that Brazil could and should produce its own
hardware and software. In that atmosphere, Tecgraf’s clients
could not afford, either politically or financially, to buy cus-
tomized software from abroad: by the market reserve rules,
they would have to go through a complicated bureaucratic
process to prove that their needs could not be met by Brazil-
ian companies. Added to the natural geographical isolation
of Brazil from other research and development centers, those
reasons led Tecgraf to implement from scratch the basic
tools it needed.
One of Tecgraf’s largest partners was (and still is) Petro-
bras, the Brazilian oil company. Several Tecgraf products
were interactive graphical programs for engineering appli-
cations at Petrobras. By 1993, Tecgraf had developed little
languages for two of those applications: a data-entry appli-
cation and a configurable report generator for lithology pro-
files. These languages, called DEL and SOL, were the an-
cestors of Lua. We describe them briefly here to show where
Lua came from.
3.1 DEL
The engineers at Petrobras needed to prepare input data files
for numerical simulators several times a day. This process
was boring and error-prone because the simulation programs
were legacy code that needed strictly formatted input files —
typically bare columns of numbers, with no indication of
what each number meant, a format inherited from the days
of punched cards. In early 1992, Petrobras asked Tecgraf to
create at least a dozen graphical front-ends for this kind of
data entry. The numbers would be input interactively, just
by clicking on the relevant parts of a diagram describing
the simulation — a much easier and more meaningful task
for the engineers than editing columns of numbers. The
data file, in the correct format for the simulator, would be
generated automatically. Besides simplifying the creation of
data files, such front-ends provided the opportunity to add
data validation and also to compute derived quantities from
the input data, thus reducing the amount of data needed from
the user and increasing the reliability of the whole process.
To simplify the development of those front-ends, a team
led by Luiz Henrique de Figueiredo and Luiz Cristovão
Gomes Coelho decided to code all front-ends in a uni-
form way, and so designed DEL (“data-entry language”),
a simple declarative language to describe each data-entry
task [17]. DEL was what is now called a domain-specific lan-
guage [43], but was then simply called a little language [10].
A typical DEL program defined several “entities”. Each
entity could have several fields, which were named and
typed. For implementing data validation, DEL had predi-
cate statements that imposed restrictions on the values of
entities. DEL also included statements to specify how data
was to be input and output. An entity in DEL was essen-
tially what is called a structure or record in conventional
2-3](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-32-320.jpg)
![programming languages. The important difference — and
what made DEL suitable for the data-entry problem — is that
entity names also appeared in a separate graphics metafile,
which contained the associated diagram over which the en-
gineer did the data entry. A single interactive graphical inter-
preter called ED (an acronym for ‘entrada de dados’, which
means ‘data entry’ in Portuguese) was written to interpret
DEL programs. All those data-entry front-ends requested
by Petrobras were implemented as DEL programs that ran
under this single graphical application.
DEL was a success both among the developers at Tec-
graf and among the users at Petrobras. At Tecgraf, DEL
simplified the development of those front-ends, as originally
intended. At Petrobras, DEL allowed users to tailor data-
entry applications to their needs. Soon users began to de-
mand more power from DEL, such as boolean expressions
for controlling whether an entity was active for input or not,
and DEL became heavier. When users began to ask for con-
trol flow, with conditionals and loops, it was clear that ED
needed a real programming language instead of DEL.
3.2 SOL
At about the same time that DEL was created, a team lead by
Roberto Ierusalimschy and Waldemar Celes started working
on PGM, a configurable report generator for lithology pro-
files, also for Petrobras. The reports generated by PGM con-
sisted of several columns (called “tracks”) and were highly
configurable: users could create and position the tracks, and
could choose colors, fonts, and labels; each track could have
a grid, which also had its set of options (log/linear, verti-
cal and horizontal ticks, etc.); each curve had its own scale,
which had to be changed automatically in case of overflow;
etc. All this configuration was to be done by the end-users,
typically geologists and engineers from Petrobras working
in oil plants and off-shore platforms. The configurations had
to be stored in files, for reuse. The team decided that the best
way to configure PGM was through a specialized description
language called SOL, an acronym for Simple Object Lan-
guage.
Because PGM had to deal with many different objects,
each with many different attributes, the SOL team decided
not to fix those objects and attributes into the language. In-
stead, SOL allowed type declarations, as in the code below:
type @track{ x:number, y:number=23, id=0 }
type @line{ t:@track=@track{x=8}, z:number* }
T = @track{ y=9, x=10, id="1992-34" }
L = @line{ t=@track{x=T.y, y=T.x}, z=[2,3,4] }
This code defines two types, track and line, and creates
two objects, a track T and a line L. The track type contains
two numeric attributes, x and y, and an untyped attribute, id;
attributes y and id have default values. The line type con-
tains a track t and a list of numbers z. The track t has as
default value a track with x=8, y=23, and id=0. The syntax
of SOL was strongly influenced by BibTEX [34] and UIL, a
language for describing user interfaces in Motif [39].
The main task of the SOL interpreter was to read a report
description, check whether the given objects and attributes
were correctly typed, and then present the information to the
main program (PGM). To allow the communication between
the main program and the SOL interpreter, the latter was
implemented as a C library that was linked to the main
program. The main program could access all configuration
information through an API in this library. In particular, the
main program could register a callback function for each
type, which the SOL interpreter would call to create an
object of that type.
4. Birth
The SOL team finished an initial implementation of SOL
in March 1993, but they never delivered it. PGM would
soon require support for procedural programming to allow
the creation of more sophisticated layouts, and SOL would
have to be extended. At the same time, as mentioned before,
ED users had requested more power from DEL. ED also
needed further descriptive facilities for programming its user
interface. Around mid-1993, Roberto, Luiz Henrique, and
Waldemar got together to discuss DEL and SOL, and con-
cluded that the two languages could be replaced by a single,
more powerful language, which they decided to design and
implement. Thus the Lua team was born; it has not changed
since.
Given the requirements of ED and PGM, we decided that
we needed a real programming language, with assignments,
control structures, subroutines, etc. The language should
also offer data-description facilities, such as those offered
by SOL. Moreover, because many potential users of the
language were not professional programmers, the language
should avoid cryptic syntax and semantics. The implemen-
tation of the new language should be highly portable, be-
cause Tecgraf’s clients had a very diverse collection of com-
puter platforms. Finally, since we expected that other Tec-
graf products would also need to embed a scripting lan-
guage, the new language should follow the example of SOL
and be provided as a library with a C API.
At that point, we could have adopted an existing scripting
language instead of creating a new one. In 1993, the only real
contender was Tcl [40], which had been explicitly designed
to be embedded into applications. However, Tcl had unfa-
miliar syntax, did not offer good support for data description,
and ran only on Unix platforms. We did not consider LISP
or Scheme because of their unfriendly syntax. Python was
still in its infancy. In the free, do-it-yourself atmosphere that
then reigned in Tecgraf, it was quite natural that we should
try to develop our own scripting language. So, we started
working on a new language that we hoped would be simpler
to use than existing languages. Our original design decisions
were: keep the language simple and small, and keep the im-
2-4](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-33-320.jpg)
![plementation simple and portable. Because the new language
was partially inspired by SOL (sun in Portuguese), a friend
at Tecgraf (Carlos Henrique Levy) suggested the name ‘Lua’
(moon in Portuguese), and Lua was born. (DEL did not in-
fluence Lua as a language. The main influence of DEL on
the birth of Lua was rather the realization that large parts
of complex applications could be written using embeddable
scripting languages.)
We wanted a light full language with data-description fa-
cilities. So we took SOL’s syntax for record and list con-
struction (but not type declaration), and unified their imple-
mentation using tables: records use strings (the field names)
as indices; lists use natural numbers. An assignment such as
T = @track{ y=9, x=10, id="1992-34" }
which was valid in SOL, remained valid in Lua, but with
a different meaning: it created an object (that is, a table)
with the given fields, and then called the function track on
this table to validate the object or perhaps to provide default
values to some of its fields. The final value of the expression
was that table.
Except for its procedural data-description constructs, Lua
introduced no new concepts: Lua was created for production
use, not as an academic language designed to support re-
search in programming languages. So, we simply borrowed
(even unconsciously) things that we had seen or read about
in other languages. We did not reread old papers to remem-
ber details of existing languages. We just started from what
we knew about other languages and reshaped that according
to our tastes and needs.
We quickly settled on a small set of control structures,
with syntax mostly borrowed from Modula (while, if, and
repeat until). From CLU we took multiple assignment
and multiple returns from function calls. We regarded mul-
tiple returns as a simpler alternative to reference parameters
used in Pascal and Modula and to in-out parameters used in
Ada; we also wanted to avoid explicit pointers (used in C).
From C++ we took the neat idea of allowing a local vari-
able to be declared only where we need it. From SNOBOL
and Awk we took associative arrays, which we called tables;
however, tables were to be objects in Lua, not attached to
variables as in Awk.
One of the few (and rather minor) innovations in Lua was
the syntax for string concatenation. The natural ‘+’ operator
would be ambiguous, because we wanted automatic coer-
cion of strings to numbers in arithmetic operations. So, we
invented the syntax ‘..’ (two dots) for string concatenation.
A polemic point was the use of semicolons. We thought
that requiring semicolons could be a little confusing for en-
gineers with a Fortran background, but not allowing them
could confuse those with a C or Pascal background. In typi-
cal committee fashion, we settled on optional semicolons.
Initially, Lua had seven types: numbers (implemented
solely as reals), strings, tables, nil, userdata (pointers to
C objects), Lua functions, and C functions. To keep the lan-
guage small, we did not initially include a boolean type:
as in Lisp, nil represented false and any other value repre-
sented true. Over 13 years of continuous evolution, the only
changes in Lua types were the unification of Lua functions
and C functions into a single function type in Lua 3.0 (1997)
and the introduction of booleans and threads in Lua 5.0
(2003) (see §6.1). For simplicity, we chose to use dynamic
typing instead of static typing. For applications that needed
type checking, we provided basic reflective facilities, such
as run-time type information and traversal of the global en-
vironment, as built-in functions (see §6.11).
By July 1993, Waldemar had finished the first implemen-
tation of Lua as a course project supervised by Roberto.
The implementation followed a tenet that is now central to
Extreme Programming: “the simplest thing that could pos-
sibly work” [7]. The lexical scanner was written with lex
and the parser with yacc, the classic Unix tools for imple-
menting languages. The parser translated Lua programs into
instructions for a stack-based virtual machine, which were
then executed by a simple interpreter. The C API made it
easy to add new functions to Lua, and so this first version
provided only a tiny library of five built-in functions (next,
nextvar, print, tonumber, type) and three small exter-
nal libraries (input and output, mathematical functions, and
string manipulation).
Despite this simple implementation — or possibly be-
cause of it — Lua surpassed our expectations. Both PGM
and ED used Lua successfully (PGM is still in use today;
ED was replaced by EDG [12], which was mostly written
in Lua). Lua was an immediate success in Tecgraf and soon
other projects started using it. This initial use of Lua at Tec-
graf was reported in a brief talk at the VII Brazilian Sympo-
sium on Software Engineering, in October 1993 [29].
The remainder of this paper relates our journey in im-
proving Lua.
5. History
Figure 1 shows a timeline of the releases of Lua. As can be
seen, the time interval between versions has been gradually
increasing since Lua 3.0. This reflects our perception that
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006
1.0 1.1 2.1 2.2 2.4 2.5 3.0 3.1 3.2 4.0 5.0 5.1
Figure 1. The releases of Lua.
2-5](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-34-320.jpg)
![for many instructions, to try to improve performance. How-
ever, this turned out to be too complicated for little per-
formance gain and we removed those variants in Lua 3.2.
Since Lua 5.0, the virtual machine is register-based [31].
This change gave the code generator more opportunities for
optimization and reduced the number of instructions of typi-
cal Lua programs. (Instruction dispatch is a significant frac-
tion of the time spent in the virtual machine [13].) As far
as we know, the virtual machine of Lua 5.0 was the first
register-based virtual machine to have wide use.
5.1 Lua 1
The initial implementation of Lua was a success in Tec-
graf and Lua attracted users from other Tecgraf projects.
New users create new demands. Several users wanted to use
Lua as the support language for graphics metafiles, which
abounded in Tecgraf. Compared with other programmable
metafiles, Lua metafiles have the advantage of being based
on a truly procedural language: it is natural to model com-
plex objects by combining procedural code fragments with
declarative statements. In contrast, for instance, VRML [8]
must use another language (Javascript) to model procedural
objects.
The use of Lua for this kind of data description, especially
large graphics metafiles, posed challenges that were unusual
for typical scripting languages. For instance, it was not un-
common for a diagram used in the data-entry program ED
to have several thousand parts described by a single Lua ta-
ble constructor with several thousand items. That meant that
Lua had to cope with huge programs and huge expressions.
Because Lua precompiled all programs to bytecode for a vir-
tual machine on the fly, it also meant that the Lua compiler
had to run fast, even for large programs.
By replacing the lex-generated scanner used in the first
version by a hand-written one, we almost doubled the speed
of the Lua compiler on typical metafiles. We also modified
Lua’s virtual machine to handle a long constructor by adding
key-value pairs to the table in batches, not individually as in
the original virtual machine. These changes solved the initial
demands for better performance. Since then, we have always
tried to reduce the time spent on precompilation.
In July 1994, we released a new version of Lua with those
optimizations. This release coincided with the publication of
the first paper describing Lua, its design, and its implementa-
tion [15]. We named the new version ‘Lua 1.1’. The previous
version, which was never publicly released, was then named
‘Lua 1.0’. (A snapshot of Lua 1.0 taken in July 1993 was
released in October 2003 to celebrate 10 years of Lua.)
Lua 1.1 was publicly released as software available in
source code by ftp, before the open-source movement got its
current momentum. Lua 1.1 had a restrictive user license: it
was freely available for academic purposes but commercial
uses had to be negotiated. That part of the license did not
work: although we had a few initial contacts, no commer-
cial uses were ever negotiated. This and the fact that other
scripting languages (e.g, Tcl) were free made us realize that
restrictions on commercial uses might even discourage aca-
demic uses, since some academic projects plan to go to mar-
ket eventually. So, when the time came to release the next
version (Lua 2.1), we chose to release it as unrestricted free
software. Naively, we wrote our own license text as a slight
collage and rewording of existing licenses. We thought it
was clear that the new license was quite liberal. Later, how-
ever, with the spread of open-source licenses, our license text
became a source of noise among some users; in particular,
it was not clear whether our license was compatible with
GPL. In May 2002, after a long discussion in the mailing
list, we decided to release future versions of Lua (starting
with Lua 5.0) under the well-known and very liberal MIT
license [3]. In July 2002, the Free Software Foundation con-
firmed that our previous license was compatible with GPL,
but we were already committed to adopting the MIT license.
Questions about our license have all but vanished since then.
5.2 Lua 2
Despite all the hype surrounding object-oriented program-
ming (which in the early 1990s had reached its peak) and
the consequent user pressure to add object-oriented features
to Lua, we did not want to turn Lua into an object-oriented
language because we did not want to fix a programming
paradigm for Lua. In particular, we did not think that Lua
needed objects and classes as primitive language concepts,
especially because they could be implemented with tables if
needed (a table can hold both object data and methods, since
functions are first-class values). Despite recurring user pres-
sure, we have not changed our minds to this day: Lua does
not force any object or class model onto the programmer.
Several object models have been proposed and implemented
by users; it is a frequent topic of discussion in our mailing
list. We think this is healthy.
On the other hand, we wanted to allow object-oriented
programming with Lua. Instead of fixing a model, we de-
cided to provide flexible mechanisms that would allow the
programmer to build whatever model was suitable to the ap-
plication. Lua 2.1, released in February 1995, marked the in-
troduction of these extensible semantics mechanisms, which
have greatly increased the expressiveness of Lua. Extensible
semantics has become a hallmark of Lua.
One of the goals of extensible semantics was to allow ta-
bles to be used as a basis for objects and classes. For that,
we needed to implement inheritance for tables. Another goal
was to turn userdata into natural proxies for application data,
not merely handles meant to be used solely as arguments to
functions. We wanted to be able to index userdata as if they
were tables and to call methods on them. This would allow
Lua to fulfill one of its main design goals more naturally:
to extend applications by providing scriptable access to ap-
plication services and data. Instead of adding mechanisms
to support all these features directly in the language, we de-
cided that it would be conceptually simpler to define a more
2-7](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-36-320.jpg)
![general fallback mechanism to let the programmer intervene
whenever Lua did not know how to proceed.
We introduced fallbacks in Lua 2.1 and defined them for
the following operations: table indexing, arithmetic oper-
ations, string concatenation, order comparisons, and func-
tion calls.3
When one of these operations was applied to
the “wrong” kind of values, the corresponding fallback
was called, allowing the programmer to determine how
Lua would proceed. The table indexing fallbacks allowed
userdata (and other values) to behave as tables, which was
one of our motivations. We also defined a fallback to be
called when a key was absent from a table, so that we
could support many forms of inheritance (through dele-
gation). To complete the support for object-oriented pro-
gramming, we added two pieces of syntactic sugar: method
definitions of the form ‘function a:foo(· · ·)’ as sugar
for ‘function a.foo(self,· · ·)’ and method calls of the
form ‘a:foo(· · ·)’ as sugar for ‘a.foo(a,· · ·)’. In §6.8 we
discuss fallbacks in detail and how they evolved into their
later incarnations: tag methods and metamethods.
Since Lua 1.0, we have provided introspective functions
for values: type, which queries the type of a Lua value;
next, which traverses a table; and nextvar, which traverses
the global environment. (As mentioned in §4, this was par-
tially motivated by the need to implement SOL-like type
checking.) In response to user pressure for full debug fa-
cilities, Lua 2.2 (November 1995) introduced a debug API
to provide information about running functions. This API
gave users the means to write in C their own introspective
tools, such as debuggers and profilers. The debug API was
initially quite simple: it allowed access to the Lua call stack,
to the currently executing line, and provided a function to
find the name of a variable holding a given value. Following
the M.Sc. work of Tomás Gorham [22], the debug API was
improved in Lua 2.4 (May 1996) by functions to access local
variables and hooks to be called at line changes and function
calls.
With the widespread use of Lua at Tecgraf, many large
graphics metafiles were being written in Lua as the output
of graphical editors. Loading such metafiles was taking in-
creasingly longer as they became larger and more complex.4
Since its first version, Lua precompiled all programs to byte-
code just before running them. The load time of a large pro-
gram could be substantially reduced by saving this bytecode
to a file. This would be especially relevant for procedural
data files such as graphics metafiles. So, in Lua 2.4, we in-
troduced an external compiler, called luac, which precom-
piled a Lua program and saved the generated bytecode to a
binary file. (Our first paper about Lua [15] had already an-
3 We also introduced fallbacks for handling fatal errors and for monitoring
garbage collection, even though they were not part of extensible semantics.
4 Surprisingly, a substantial fraction of the load time was taken in the lexer
for converting real numbers from text form to floating-point representation.
Real numbers abound in graphics metafiles.
ticipated the possibility of an external compiler.) The format
of this file was chosen to be easily loaded and reasonably
portable. With luac, programmers could avoid parsing and
code generation at run time, which in the early days were
costly. Besides faster loading, luac also allowed off-line
syntax checking and protection from casual user changes.
Many products (e.g., The Sims and Adobe Lightroom) dis-
tribute Lua scripts in precompiled form.
During the implementation of luac, we started to restruc-
ture Lua’s core into clearly separated modules. As a conse-
quence, it is now quite easy to remove the parsing modules
(lexer, parser, and code generator), which currently repre-
sent 35% of the core code, leaving just the module that loads
precompiled Lua programs, which is merely 3% of the core
code. This reduction can be significant when embedding Lua
in small devices such as mobile devices, robots and sensors.5
Since its first version, Lua has included a library for
string-processing. The facilities provided by this library
were minimal until Lua 2.4. However, as Lua matured, it
became desirable to do heavier text processing in Lua. We
thought that a natural addition to Lua would be pattern
matching, in the tradition of Snobol, Icon, Awk, and Perl.
However, we did not want to include a third-party pattern-
matching engine in Lua because such engines tend to be very
large; we also wanted to avoid copyright issues that could be
raised by including third-party code in Lua.
As a student project supervised by Roberto in the second
semester of 1995, Milton Jonathan, Pedro Miller
Rabinovitch, Pedro Willemsens, and Vinicius Almendra pro-
duced a pattern-matching library for Lua. Experience with
that design led us to write our own pattern-matching en-
gine for Lua, which we added to Lua 2.5 (November 1996)
in two functions: strfind (which originally only found
plain substrings) and the new gsub function (a name taken
from Awk). The gsub function globally replaced substrings
matching a given pattern in a larger string. It accepted either
a replacement string or a function that was called each time
a match was found and was intended to return the replace-
ment string for that match. (That was an innovation at the
time.) Aiming at a small implementation, we did not include
full regular expressions. Instead, the patterns understood by
our engine were based on character classes, repetitions, and
captures (but not alternation or grouping). Despite its sim-
plicity, this kind of pattern matching is quite powerful and
was an important addition to Lua.
That year was a turning point in the history of Lua be-
cause it gained international exposure. In June 1996 we pub-
lished a paper about Lua in Software: Practice & Experi-
ence [30] that brought external attention to Lua, at least in
5 Crazy Ivan, a robot that won RoboCup in 2000 and 2001 in Denmark,
had a “brain” implemented in Lua. It ran directly on a Motorola Coldfire
5206e processor without any operating system (in other words, Lua was the
operating system). Lua was stored on a system ROM and loaded programs
at startup from the serial port.
2-8](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-37-320.jpg)
![academic circles.6
In December 1996, shortly after Lua 2.5
was released, the magazine Dr. Dobb’s Journal featured
an article about Lua [16]. Dr. Dobb’s Journal is a popular
publication aimed directly at programmers, and that article
brought Lua to the attention of the software industry. Among
several messages that we received right after that publication
was one sent in January 1997 by Bret Mogilefsky, who was
the lead programmer of Grim Fandango, an adventure game
then under development by LucasArts. Bret told us that he
had read about Lua in Dr. Dobb’s and that they planned to re-
place their home-brewed scripting language with Lua. Grim
Fandango was released in October 1998 and in May 1999
Bret told us that “a tremendous amount of the game was
written in Lua” (his emphasis) [38].7
Around that time, Bret
attended a roundtable about game scripting at the Game De-
velopers’ Conference (GDC, the main event for game pro-
grammers) and at the end he related his experience with the
successful use of Lua in Grim Fandango. We know of several
developers who first learned about Lua at that event. After
that, Lua spread by word of mouth among game developers
to become a definitely marketable skill in the game industry
(see §8).
As a consequence of Lua’s international exposure, the
number of messages sent to us asking questions about Lua
increased substantially. To handle this traffic more effi-
ciently, and also to start building a Lua community, so that
other people could answer Lua questions, in February 1997
we created a mailing list for discussing Lua. Over 38,000
messages have been posted to this list since then. The use
of Lua in many popular games has attracted many people to
the list, which now has over 1200 subscribers. We have been
fortunate that the Lua list is very friendly and at the same
time very technical. The list has become the focal point of
the Lua community and has been a source of motivation
for improving Lua. All important events occur first in the
mailing list: release announcements, feature requests, bug
reports, etc.
The creation of a comp.lang.lua Usenet newsgroup
was discussed twice in the list over all these years, in
April 1998 and in July 1999. The conclusion both times
was that the traffic in the list did not warrant the creation
of a newsgroup. Moreover, most people preferred a mailing
list. The creation of a newsgroup seems no longer relevant
because there are several web interfaces for reading and
searching the complete list archives.
6 In November 1997, that article won the First Prize (technological cate-
gory) in the II Compaq Award for Research and Development in Computer
Science, a joint venture of Compaq Computer in Brazil, the Brazilian Min-
istry of Science and Technology, and the Brazilian Academy of Sciences.
7 Grim Fandango mentioned Lua and PUC-Rio in its final credits. Several
people at PUC-Rio first learned about Lua from that credit screen, and
were surprised to learn that Brazilian software was part of a hit game. It
has always bothered us that Lua is widely known abroad but has remained
relatively unknown in Brazil until quite recently.
5.3 Lua 3
The fallback mechanism introduced in Lua 2.1 to support
extensible semantics worked quite well but it was a global
mechanism: there was only one hook for each event. This
made it difficult to share or reuse code because modules that
defined fallbacks for the same event could not co-exist eas-
ily. Following a suggestion by Stephan Herrmann in Decem-
ber 1996, in Lua 3.0 (July 1997) we solved the fallback clash
problem by replacing fallbacks with tag methods: the hooks
were attached to pairs (event, tag) instead of just to events.
Tags had been introduced in Lua 2.1 as integer labels that
could be attached to userdata (see §6.10); the intention was
that C objects of the same type would be represented in Lua
by userdata having the same tag. (However, Lua did not force
any interpretation on tags.) In Lua 3.0 we extended tags to
all values to support tag methods. The evolution of fallbacks
is discussed in §6.8.
Lua 3.1 (July 1998) brought functional programming to
Lua by introducing anonymous functions and function clo-
sures via “upvalues”. (Full lexical scoping had to wait until
Lua 5.0; see §6.6.) The introduction of closures was mainly
motivated by the existence of higher-order functions, such as
gsub, which took functions as arguments. During the work
on Lua 3.1, there were discussions in the mailing list about
multithreading and cooperative multitasking, mainly moti-
vated by the changes Bret Mogilefsky had made to Lua 2.5
and 3.1 alpha for Grim Fandango. No conclusions were
reached, but the topic remained popular. Cooperative multi-
tasking in Lua was finally provided in Lua 5.0 (April 2003);
see §6.7.
The C API remained largely unchanged from Lua 1.0
to Lua 3.2; it worked over an implicit Lua state. However,
newer applications, such as web services, needed multiple
states. To mitigate this problem, Lua 3.1 introduced multiple
independent Lua states that could be switched at run time.
A fully reentrant API would have to wait until Lua 4.0. In
the meantime, two unofficial versions of Lua 3.2 with ex-
plicit Lua states appeared: one written in 1998 by Roberto
Ierusalimschy and Anna Hester based on Lua 3.2 alpha for
CGILua [26], and one written in 1999 by Erik Hougaard
based on Lua 3.2 final. Erik’s version was publicly avail-
able and was used in the Crazy Ivan robot. The version for
CGILua was released only as part of the CGILua distribu-
tion; it never existed as an independent package.
Lua 3.2 (July 1999) itself was mainly a maintenance re-
lease; it brought no novelties except for a debug library that
allowed tools to be written in Lua instead of C. Neverthe-
less, Lua was quite stable by then and Lua 3.2 had a long
life. Because the next version (Lua 4.0) introduced a new,
incompatible API, many users just stayed with Lua 3.2 and
never migrated to Lua 4.0. For instance, Tecgraf never mi-
grated to Lua 4.0, opting to move directly to Lua 5.0; many
products at Tecgraf still use Lua 3.2.
2-9](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-38-320.jpg)
![Packaging library functions inside tables had a big practi-
cal impact, because it affected any program that used at least
one library function. For instance, the old strfind function
was now called string.find (field ‘find’ in string library
stored in the ‘string’ table); openfile became io.open;
sin became math.sin; and so on. To make the transition
easier, we provided a compatibility script that defined the
old functions in terms of the new ones:
strfind = string.find
openfile = io.open
sin = math.sin
...
Nevertheless, packaging libraries in tables was a major
change. In June 2002, when we released the next work
version incorporating this change, we dropped the name
‘Lua 4.1’ and named it ‘Lua 5.0 work0’. Progress to the
final version was steady from then on and Lua 5.0 was re-
leased in April 2003. This release froze Lua enough to allow
Roberto to finish his book, which was published in Decem-
ber 2003 [27].
Soon after the release of Lua 5.0 we started working
on Lua 5.1. The initial motivation was the implementation
of incremental garbage collection in response to requests
from game developers. Lua uses a traditional mark-and-
sweep garbage collector, and, until Lua 5.0, garbage col-
lection was performed atomically. As a consequence, some
applications might experience potentially long pauses dur-
ing garbage collection.10
At that time, our main concern was
that adding the write barriers needed to implement an incre-
mental garbage collector would have a negative impact on
Lua performance. To compensate for that we tried to make
the collector generational as well. We also wanted to keep
the adaptive behavior of the old collector, which adjusted the
frequency of collection cycles according to the total memory
in use. Moreover, we wanted to keep the collector simple,
like the rest of Lua.
We worked on the incremental generational garbage col-
lector for over a year. But since we did not have access to
applications with strong memory requirements (like games),
it was difficult for us to test the collector in real scenarios.
From March to December 2004 we released several work
versions trying to get concrete feedback on the performance
of the collector in real applications. We finally received re-
ports of bizarre memory-allocation behavior, which we later
managed to reproduce but not explain. In January 2005,
Mike Pall, an active member of the Lua community, came
up with memory-allocation graphs that explained the prob-
lem: in some scenarios, there were subtle interactions be-
tween the incremental behavior, the generational behavior,
and the adaptive behavior, such that the collector “adapted”
10 Erik Hougaard reported that the Crazy Ivan robot would initially drive
off course when Lua performed garbage collection (which could take a half
second, but that was enough). To stay in course, they had to stop both motors
and pause the robot during garbage collection.
for less and less frequent collections. Because it was getting
too complicated and unpredictable, we gave up the genera-
tional aspect and implemented a simpler incremental collec-
tor in Lua 5.1.
During that time, programmers had been experimenting
with the module system introduced in Lua 5.0. New pack-
ages started to be produced, and old packages migrated to the
new system. Package writers wanted to know the best way
to build modules. In July 2005, during the development of
Lua 5.1, an international Lua workshop organized by Mark
Hamburg was held at Adobe in San Jose. (A similar work-
shop organized by Wim Couwenberg and Daniel Silverstone
was held in September 2006 at Océ in Venlo.) One of the
presentations was about the novelties of Lua 5.1, and there
were long discussions about modules and packages. As a re-
sult, we made a few small but significant changes in the mod-
ule system. Despite our “mechanisms, not policy” guideline
for Lua, we defined a set of policies for writing modules
and loading packages, and made small changes to support
these policies better. Lua 5.1 was released in February 2006.
Although the original motivation for Lua 5.1 was incremen-
tal garbage collection, the improvement in the module sys-
tem was probably the most visible change. On the other
hand, that incremental garbage collection remained invisible
shows that it succeeded in avoiding long pauses.
6. Feature evolution
In this section, we discuss in detail the evolution of some of
the features of Lua.
6.1 Types
Types in Lua have been fairly stable. For a long time, Lua
had only six basic types: nil, number, string, table, function,
and userdata. (Actually, until Lua 3.0, C functions and Lua
functions had different types internally, but that difference
was transparent to callers.) The only real change happened
in Lua 5.0, which introduced two new types: threads and
booleans.
The type thread was introduced to represent coroutines.
Like all other Lua values, threads are first-class values.
To avoid creating new syntax, all primitive operations on
threads are provided by a library.
For a long time we resisted introducing boolean values in
Lua: nil was false and anything else was true. This state of
affairs was simple and seemed sufficient for our purposes.
However, nil was also used for absent fields in tables and
for undefined variables. In some applications, it is important
to allow table fields to be marked as false but still be seen
as present; an explicit false value can be used for this. In
Lua 5.0 we finally introduced boolean values true and false.
Nil is still treated as false. In retrospect, it would probably
have been better if nil raised an error in boolean expres-
sions, as it does in other expressions. This would be more
consistent with its role as proxy for undefined values. How-
2-11](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-40-320.jpg)
![ever, such a change would probably break many existing
programs. LISP has similar problems, with the empty list
representing both nil and false. Scheme explicitly represents
false and treats the empty list as true, but some implementa-
tions of Scheme still treat the empty list as false.
6.2 Tables
Lua 1.1 had three syntactical constructs to create tables:
‘@()’, ‘@[]’, and ‘@{}’. The simplest form was ‘@()’,
which created an empty table. An optional size could be
given at creation time, as an efficiency hint. The form ‘@[]’
was used to create arrays, as in ‘@[2,4,9,16,25]’. In
such tables, the keys were implicit natural numbers start-
ing at 1. The form ‘@{}’ was used to create records, as in
‘@{name="John",age=35}’. Such tables were sets of key-
value pairs in which the keys were explicit strings. A table
created with any of those forms could be modified dynam-
ically after creation, regardless of how it had been created.
Moreover, it was possible to provide user functions when
creating lists and records, as in ‘@foo[]’ or ‘@foo{}’. This
syntax was inherited from SOL and was the expression of
procedural data description, a major feature of Lua (see §2).
The semantics was that a table was created and then the
function was called with that table as its single argument.
The function was allowed to check and modify the table at
will, but its return values were ignored: the table was the
final value of the expression.
In Lua 2.1, the syntax for table creation was unified and
simplified: the leading ‘@’ was removed and the only con-
structor became ‘{· · ·}’. Lua 2.1 also allowed mixed con-
structors, such as
grades{8.5, 6.0, 9.2; name="John", major="math"}
in which the array part was separated from the record
part by a semicolon. Finally, ‘foo{· · ·}’ became sugar for
‘foo({· · ·})’. In other words, table constructors with func-
tions became ordinary function calls. As a consequence, the
function had to explicitly return the table (or whatever value
it chose). Dropping the ‘@’ from constructors was a trivial
change, but it actually changed the feel of the language, not
merely its looks. Trivial changes that improve the feel of a
language are not to be overlooked.
This simplification in the syntax and semantics of ta-
ble constructors had a side-effect, however. In Lua 1.1, the
equality operator was ‘=’. With the unification of table con-
structors in Lua 2.1, an expression like ‘{a=3}’ became am-
biguous, because it could mean a table with either a pair
("a", 3) or a pair (1, b), where b is the value of the equal-
ity ‘a=3’. To solve this ambiguity, in Lua 2.1 we changed the
equality operator from ‘=’ to ‘==’. With this change, ‘{a=3}’
meant a table with the pair ("a", 3), while ‘{a==3}’ meant
a table with the pair (1, b).
These changes made Lua 2.1 incompatible with Lua 1.1
(hence the change in the major version number). Neverthe-
less, since at that time virtually all Lua users were from Tec-
graf, this was not a fatal move: existing programs were easily
converted with the aid of ad-hoc tools that we wrote for this
task.
The syntax for table constructors has since remained
mostly unchanged, except for an addition introduced in
Lua 3.1: keys in the record part could be given by any ex-
pression, by enclosing the expression inside brackets, as in
‘{[10*x+f(y)]=47}’. In particular, this allowed keys to
be arbitrary strings, including reserved words and strings
with spaces. Thus, ‘{function=1}’ is not valid (because
‘function’ is a reserved word), but ‘{["function"]=1}’
is valid. Since Lua 5.0, it is also possible to freely intermix
the array part and the record part, and there is no need to use
semicolons in table constructors.
While the syntax of tables has evolved, the semantics of
tables in Lua has not changed at all: tables are still asso-
ciative arrays and can store arbitrary pairs of values. How-
ever, frequently in practice tables are used solely as arrays
(that is, with consecutive integer keys) or solely as records
(that is, with string keys). Because tables are the only data-
structuring mechanism in Lua, we have invested much ef-
fort in implementing them efficiently inside Lua’s core. Un-
til Lua 4.0, tables were implemented as pure hash tables,
with all pairs stored explicitly. In Lua 5.0 we introduced a
hybrid representation for tables: every table contains a hash
part and an array part, and both parts can be empty. Lua de-
tects whether a table is being used as an array and automat-
ically stores the values associated to integer indices in the
array part, instead of adding them to the hash part [31]. This
division occurs only at a low implementation level; access
to table fields is transparent, even to the virtual machine. Ta-
bles automatically adapt their two parts according to their
contents.
This hybrid scheme has two advantages. First, access
to values with integer keys is faster because no hashing is
needed. Second, and more important, the array part takes
roughly half the memory it would take if it were stored in
the hash part, because the keys are implicit in the array part
but explicit in the hash part. As a consequence, if a table is
being used as an array, it performs as an array, as long as
its integer keys are densely distributed. Moreover, no mem-
ory or time penalty is paid for the hash part, because it
does not even exist. Conversely, if the table is being used
as a record and not as an array, then the array part is likely
to be empty. These memory savings are important because
it is common for a Lua program to create many small ta-
bles (e.g., when tables are used to represent objects). Lua
tables also handle sparse arrays gracefully: the statement
‘a={[1000000000]=1}’ creates a table with a single entry
in its hash part, not an array with one billion elements.
Another reason for investing effort into an efficient im-
plementation of tables is that we can use tables for all kinds
of tasks. For instance, in Lua 5.0 the standard library func-
tions, which had existed since Lua 1.1 as global variables,
2-12](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-41-320.jpg)
![were moved to fields inside tables (see §5.5). More recently,
Lua 5.1 brought a complete package and module system
based on tables.
Tables play a prominent role in Lua’s core. On two oc-
casions we have been able to replace special data structures
inside the core with ordinary Lua tables: in Lua 4.0 for repre-
senting the global environment (which keeps all global vari-
ables) and in Lua 5.0 for implementing extensible seman-
tics (see §6.8). Starting with Lua 4.0, global variables are
stored in an ordinary Lua table, called the table of globals,
a simplification suggested by John Belmonte in April 2000.
In Lua 5.0 we replaced tags and tag methods (introduced
in Lua 3.0) by metatables and metamethods. Metatables are
ordinary Lua tables and metamethods are stored as fields
in metatables. Lua 5.0 also introduced environment tables
that can be attached to Lua functions; they are the tables
where global names in Lua functions are resolved at run
time. Lua 5.1 extended environment tables to C functions,
userdata, and threads, thus replacing the notion of global en-
vironment. These changes simplified both the implementa-
tion of Lua and the API for Lua and C programmers, be-
cause globals and metamethods can be manipulated within
Lua without the need for special functions.
6.3 Strings
Strings play a major role in scripting languages and so the
facilities to create and manipulate strings are an important
part of the usability of such languages.
The syntax for literal strings in Lua has had an interesting
evolution. Since Lua 1.1, a literal string can be delimited
by matching single or double quotes, and can contain C-like
escape sequences. The use of both single and double quotes
to delimit strings with the same semantics was a bit unusual
at the time. (For instance, in the tradition of shell languages,
Perl expands variables inside double-quoted strings, but not
inside single-quoted strings.) While these dual quotes allow
strings to contain one kind of quote without having to escape
it, escape sequences are still needed for arbitrary text.
Lua 2.2 introduced long strings, a feature not present in
classical programming languages, but present in most script-
ing languages.11
Long strings can run for several lines and
do not interpret escape sequences; they provide a convenient
way to include arbitrary text as a string, without having to
worry about its contents. However, it is not trivial to de-
sign a good syntax for long strings, especially because it
is common to use them to include arbitrary program text
(which may contain other long strings). This raises the ques-
tion of how long strings end and whether they may nest.
Until Lua 5.0, long strings were wrapped inside matching
‘[[· · ·]]’ and could contain nested long strings. Unfortu-
nately, the closing delimiter ‘]]’ could easily be part of a
valid Lua program in an unbalanced way, as in ‘a[b[i]]’,
11 ‘Long string’ is a Lua term. Other languages use terms such as ‘verbatim
text’ or ‘heredoc’.
or in other contexts, such as ‘<[!CDATA[· · ·]]>’ from XML.
So, it was hard to reliably wrap arbitrary text as a long string.
Lua 5.1 introduced a new form for long strings: text de-
limited by matching ‘[===[· · ·]===]’, where the number of
‘=’ characters is arbitrary (including zero). These new long
strings do not nest: a long string ends as soon as a closing de-
limiter with the right number of ‘=’ is seen. Nevertheless, it
is now easy to wrap arbitrary text, even text containing other
long strings or unbalanced ‘]= · · · =]’ sequences: simply use
an adequate number of ‘=’ characters.
6.4 Block comments
Comments in Lua are signaled by ‘--’ and continue to the
end of the line. This is the simplest kind of comment, and
is very effective. Several other languages use single-line
comments, with different marks. Languages that use ‘--’ for
comments include Ada and Haskell.
We never felt the need for multi-line comments, or block
comments, except as a quick way to disable code. There
was always the question of which syntax to use: the famil-
iar ‘/* · · · */’ syntax used in C and several other languages
does not mesh well with Lua’s single-line comments. There
was also the question of whether block comments could nest
or not, always a source of noise for users and of complexity
for the lexer. Nested block comments happen when program-
mers want to ‘comment out’ some block of code, to disable
it. Naturally, they expect that comments inside the block of
code are handled correctly, which can only happen if block
comments can be nested.
ANSI C supports block comments but does not allow
nesting. C programmers typically disable code by using the
C preprocessor idiom ‘#if 0 · · · #endif’. This scheme has
the clear advantage that it interacts gracefully with existing
comments in the disabled code. With this motivation and in-
spiration, we addressed the need for disabling blocks of code
in Lua — not the need for block comments — by introducing
conditional compilation in Lua 3.0 via pragmas inspired in
the C preprocessor. Although conditional compilation could
be used for block comments, we do not think that it ever
was. During work on Lua 4.0, we decided that the support
for conditional compilation was not worth the complexity in
the lexer and in its semantics for the user, especially after
not having reached any consensus about a full macro facil-
ity (see §7). So, in Lua 4.0 we removed support for con-
ditional compilation and Lua remained without support for
block comments.12
Block comments were finally introduced in Lua 5.0, in
the form ‘--[[· · ·]]’. Because they intentionally mimicked
the syntax of long strings (see §6.3), it was easy to modify
the lexer to support block comments. This similarity also
helped users to grasp both concepts and their syntax. Block
12 A further motivation was that by that time we had found a better way to
generate and use debug information, and so the pragmas that controlled this
were no longer needed. Removing conditional compilation allowed us to
get rid of all pragmas.
2-13](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-42-320.jpg)
![comments can also be used to disable code: the idiom is to
surround the code between two lines containing ‘--[[’ and
‘--]]’. The code inside those lines can be re-enabled by
simply adding a single ‘-’ at the start of the first line: both
lines then become harmless single-line comments.
Like long strings, block comments could nest, but they
had the same problems as long strings. In particular, valid
Lua code containing unbalanced ‘]]’s, such as ‘a[b[i]]’,
could not be reliably commented out in Lua 5.0. The new
scheme for long strings in Lua 5.1 also applies to block com-
ments, in the form of matching ‘--[===[· · ·]===]’, and so
provides a simple and robust solution for this problem.
6.5 Functions
Functions in Lua have always been first-class values. A func-
tion can be created at run time by compiling and executing
a string containing its definition.13
Since the introduction of
anonymous functions and upvalues in Lua 3.1, programmers
are able to create functions at run time without resorting to
compilation from text.
Functions in Lua, whether written in C or in Lua, have
no declaration. At call time they accept a variable number
of arguments: excess arguments are discarded and missing
arguments are given the value nil. (This coincides with the
semantics of multiple assignment.) C functions have always
been able to handle a variable number of arguments. Lua 2.5
introduced vararg Lua functions, marked by a parameter
list ending in ‘...’ (an experimental feature that became
official only in Lua 3.0). When a vararg function was called,
the arguments corresponding to the dots were collected into
a table named ‘arg’. While this was simple and mostly
convenient, there was no way to pass those arguments to
another function, except by unpacking this table. Because
programmers frequently want to just pass the arguments
along to other functions, Lua 5.1 allows ‘...’ to be used
in argument lists and on the right-hand side of assignments.
This avoids the creation of the ‘arg’ table if it is not needed.
The unit of execution of Lua is called a chunk; it is
simply a sequence of statements. A chunk in Lua is like
the main program in other languages: it can contain both
function definitions and executable code. (Actually, a func-
tion definition is executable code: an assignment.) At the
same time, a chunk closely resembles an ordinary Lua func-
tion. For instance, chunks have always had exactly the same
kind of bytecode as ordinary Lua functions. However, before
Lua 5.0, chunks needed some internal magic to start execut-
ing. Chunks began to look like ordinary functions in Lua 2.2,
when local variables outside functions were allowed as an
undocumented feature (that became official only in Lua 3.1).
Lua 2.5 allowed chunks to return values. In Lua 3.0 chunks
became functions internally, except that they were executed
13 Some people maintain that the ability to evaluate code from text at run
time and within the environment of the running program is what character-
izes scripting languages.
right after being compiled; they did not exist as functions at
the user level. This final step was taken in Lua 5.0, which
broke the loading and execution of chunks into two steps,
to provide host programmers better control for handling and
reporting errors. As a consequence, in Lua 5.0 chunks be-
came ordinary anonymous functions with no arguments. In
Lua 5.1 chunks became anonymous vararg functions and
thus can be passed values at execution time. Those values
are accessed via the new ‘...’ mechanism.
From a different point of view, chunks are like modules
in other languages: they usually provide functions and vari-
ables to the global environment. Originally, we did not in-
tend Lua to be used for large-scale programming and so we
did not feel the need to add an explicit notion of modules
to Lua. Moreover, we felt that tables would be sufficient for
building modules, if necessary. In Lua 5.0 we made that feel-
ing explicit by packaging all standard libraries into tables.
This encouraged other people to do the same and made it
easier to share libraries. We now feel that Lua can be used for
large-scale programming, especially after Lua 5.1 brought a
package system and a module system, both based on tables.
6.6 Lexical scoping
From an early stage in the development of Lua we started
thinking about first-class functions with full lexical scoping.
This is an elegant construct that fits well within Lua’s philos-
ophy of providing few but powerful constructs. It also makes
Lua apt for functional programming. However, we could not
figure out a reasonable implementation for full lexical scop-
ing. Since the beginning Lua has used a simple array stack
to keep activation records (where all local variables and tem-
poraries live). This implementation had proved simple and
efficient, and we saw no reason to change it. When we allow
nested functions with full lexical scoping, a variable used by
an inner function may outlive the function that created it, and
so we cannot use a stack discipline for such variables.
Simple Scheme implementations allocate frames in the
heap. Already in 1987, Dybvig [20] described how to use
a stack to allocate frames, provided that those frames did
not contain variables used by nested functions. His method
requires that the compiler know beforehand whether a vari-
able appears as a free variable in a nested function. This does
not suit the Lua compiler because it generates code to ma-
nipulate variables as soon as it parses an expression; at that
moment, it cannot know whether any variable is later used
free in a nested function. We wanted to keep this design for
implementing Lua, because of its simplicity and efficiency,
and so could not use Dybvig’s method. For the same rea-
son, we could not use advanced compiler techniques, such
as data-flow analysis.
Currently there are several optimization strategies to
avoid using the heap for frames (e.g., [21]), but they all
need compilers with intermediate representations, which the
Lua compiler does not use. McDermott’s proposal for stack
frame allocation [36], which is explicitly addressed to inter-
2-14](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-43-320.jpg)
![preters, is the only one we know of that does not require in-
termediate representation for code generation. Like our cur-
rent implementation [31], his proposal puts variables in the
stack and moves them to the heap on demand, if they go out
of scope while being used by a nested closure. However, his
proposal assumes that environments are represented by as-
sociation lists. So, after moving an environment to the heap,
the interpreter has to correct only the list header, and all ac-
cesses to local variables automatically go to the heap. Lua
uses real records as activation records, with local-variable
access being translated to direct accesses to the stack plus an
offset, and so cannot use McDermott’s method.
For a long time those difficulties kept us from introducing
nested first-class functions with full lexical scoping in Lua.
Finally, in Lua 3.1 we settled on a compromise that we called
upvalues. In this scheme, an inner function cannot access
and modify external variables when it runs, but it can access
the values those variables had when the function was cre-
ated. Those values are called upvalues. The main advantage
of upvalues is that they can be implemented with a simple
scheme: all local variables live in the stack; when a function
is created, it is wrapped in a closure containing copies of
the values of the external variables used by the function. In
other words, upvalues are the frozen values of external vari-
ables.14
To avoid misunderstandings, we created a new syn-
tax for accessing upvalues: ‘%varname’. This syntax made
it clear that the code was accessing the frozen value of that
variable, not the variable itself. Upvalues proved to be very
useful, despite being immutable. When necessary, we could
simulate mutable external variables by using a table as the
upvalue: although we could not change the table itself, we
could change its fields. This feature was especially useful for
anonymous functions passed to higher-order functions used
for table traversal and pattern matching.
In December 2000, Roberto wrote in the first draft of
his book [27] that “Lua has a form of proper lexical scop-
ing through upvalues.” In July 2001 John D. Ramsdell ar-
gued in the mailing list that “a language is either lexically
scoped or it is not; adding the adjective ‘proper’ to the phrase
‘lexical scoping’ is meaningless.” That message stirred us
to search for a better solution and a way to implement full
lexical scoping. By October 2001 we had an initial imple-
mentation of full lexical scoping and described it to the list.
The idea was to access each upvalue through an indirection
that pointed to the stack while the variable was in scope;
at the end of the scope a special virtual machine instruc-
tion “closed” the upvalue, moving the variable’s value to a
heap-allocated space and correcting the indirection to point
there. Open closures (those with upvalues still pointing to
the stack) were kept in a list to allow their correction and
14 A year later Java adopted a similar solution to allow inner classes. Instead
of freezing the value of an external variable, Java insists that you can only
access final variables in inner classes, and so ensures that the variable is
frozen.
the reuse of open upvalues. Reuse is essential to get the cor-
rect semantics. If two closures, sharing an external variable,
have their own upvalues, then at the end of the scope each
closure will have its own copy of the variable, but the cor-
rect semantics dictates that they should share the variable.
To ensure reuse, the algorithm that created closures worked
as follows: for each external variable used by the closure, it
first searched the list of open closures. If it found an upvalue
pointing to that external variable, it reused that upvalue; oth-
erwise, it created a new upvalue.
Edgar Toering, an active member of the Lua community,
misunderstood our description of lexical scoping. It turned
out that the way he understood it was better than our orig-
inal idea: instead of keeping a list of open closures, keep a
list of open upvalues. Because the number of local variables
used by closures is usually smaller than the number of clo-
sures using them (the first is statically limited by the program
text), his solution was more efficient than ours. It was also
easier to adapt to coroutines (which were being implemented
at around the same time), because we could keep a separate
list of upvalues for each stack. We added full lexical scoping
to Lua 5.0 using this algorithm because it met all our require-
ments: it could be implemented with a one-pass compiler; it
imposed no burden on functions that did not access exter-
nal local variables, because they continued to manipulate all
their local variables in the stack; and the cost to access an
external local variable was only one extra indirection [31].
6.7 Coroutines
For a long time we searched for some kind of first-class
continuations for Lua. This search was motivated by the
existence of first-class continuations in Scheme (always a
source of inspiration to us) and by demands from game
programmers for some mechanism for “soft” multithreading
(usually described as “some way to suspend a character and
continue it later”).
In 2000, Maria Julia de Lima implemented full first-class
continuations on top of Lua 4.0 alpha, as part of her Ph.D.
work [35]. She used a simple approach because, like lexi-
cal scoping, smarter techniques to implement continuations
were too complex compared to the overall simplicity of Lua.
The result was satisfactory for her experiments, but too slow
to be incorporated in a final product. Nevertheless, her im-
plementation uncovered a problem peculiar to Lua. Since
Lua is an extensible extension language, it is possible (and
common) to call Lua from C and C from Lua. Therefore, at
any given point in the execution of a Lua program, the cur-
rent continuation usually has parts in Lua mixed with parts
in C. Although it is possible to manipulate a Lua continu-
ation (essentially by manipulating the Lua call stack), it is
impossible to manipulate a C continuation within ANSI C.
At that time, we did not understand this problem deeply
enough. In particular, we could not figure out what the ex-
act restrictions related to C calls were. Lima simply forbade
any C calls in her implementation. Again, that solution was
2-15](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-44-320.jpg)
![satisfactory for her experiments, but unacceptable for an of-
ficial Lua version because the ease of mixing Lua code with
C code is one of Lua’s hallmarks.
Unaware of this difficulty, in December 2001 Thatcher
Ulrich announced in the mailing list:
I’ve created a patch for Lua 4.0 that makes calls from
Lua to Lua non-recursive (i.e., ‘stackless’). This al-
lows the implementation of a ‘sleep()’ call, which ex-
its from the host program [...], and leaves the Lua
state in a condition where the script can be resumed
later via a call to a new API function, lua_resume.
In other words, he proposed an asymmetric coroutine mech-
anism, based on two primitives: yield (which he called sleep)
and resume. His patch followed the high-level description
given in the mailing list by Bret Mogilefsky on the changes
made to Lua 2.5 and 3.1 to add cooperative multitasking in
Grim Fandango. (Bret could not provide details, which were
proprietary.)
Shortly after this announcement, during the Lua Library
Design Workshop held at Harvard in February 2002, there
was some discussion about first-class continuations in Lua.
Some people claimed that, if first-class continuations were
deemed too complex, we could implement one-shot contin-
uations. Others argued that it would be better to implement
symmetric coroutines. But we could not find a proper imple-
mentation of any of these mechanisms that could solve the
difficulty related to C calls.
It took us some time to realize why it was hard to im-
plement symmetric coroutines in Lua, and also to under-
stand how Ulrich’s proposal, based on asymmetric corou-
tines, avoided our difficulties. Both one-shot continuations
and symmetric coroutines involve the manipulation of full
continuations. So, as long as these continuations include any
C part, it is impossible to capture them (except by using fa-
cilities outside ANSI C). In contrast, an asymmetric corou-
tine mechanism based on yield and resume manipulates par-
tial continuations: yield captures the continuation up to the
corresponding resume [19]. With asymmetric coroutines, the
current continuation can include C parts, as long as they
are outside the partial continuation being captured. In other
words, the only restriction is that we cannot yield across a
C call.
After that realization, and based on Ulrich’s proof-of-
concept implementation, we were able to implement asym-
metrical coroutines in Lua 5.0. The main change was that the
interpreter loop, which executes the instructions for the vir-
tual machine, ceased to be recursive. In previous versions,
when the interpreter loop executed a CALL instruction, it
called itself recursively to execute the called function. Since
Lua 5.0, the interpreter behaves more like a real CPU: when
it executes a CALL instruction, it pushes some context infor-
mation onto a call stack and proceeds to execute the called
function, restoring the context when that function returns.
After that change, the implementation of coroutines became
straightforward.
Unlike most implementations of asymmetrical corou-
tines, in Lua coroutines are what we call stackfull [19]. With
them, we can implement symmetrical coroutines and even
the call/1cc operator (call with current one-shot continua-
tion) proposed for Scheme [11]. However, the use of C func-
tions is severely restricted within these implementations.
We hope that the introduction of coroutines in Lua 5.0
marks a revival of coroutines as powerful control struc-
tures [18].
6.8 Extensible semantics
As mentioned in §5.2, we introduced extensible semantics
in Lua 2.1 in the form of fallbacks as a general mechanism
to allow the programmer to intervene whenever Lua did not
know how to proceed. Fallbacks thus provided a restricted
form of resumable exception handling. In particular, by us-
ing fallbacks, we could make a value respond to operations
not originally meant for it or make a value of one type be-
have like a value of another type. For instance, we could
make userdata and tables respond to arithmetic operations,
userdata behave as tables, strings behave as functions, etc.
Moreover, we could make a table respond to keys that were
absent in it, which is fundamental for implementing inheri-
tance. With fallbacks for table indexing and a little syntactic
sugar for defining and calling methods, object-oriented pro-
gramming with inheritance became possible in Lua.
Although objects, classes, and inheritance were not core
concepts in Lua, they could be implemented directly in Lua,
in many flavors, according to the needs of the application. In
other words, Lua provided mechanisms, not policy — a tenet
that we have tried to follow closely ever since.
The simplest kind of inheritance is inheritance by del-
egation, which was introduced by Self and adopted in
other prototype-based languages such as NewtonScript and
JavaScript. The code below shows an implementation of in-
heritance by delegation in Lua 2.1.
function Index(a,i)
if i == "parent" then
return nil
end
local p = a.parent
if type(p) == "table" then
return p[i]
else
return nil
end
end
setfallback("index", Index)
When a table was accessed for an absent field (be it an
attribute or a method), the index fallback was triggered.
Inheritance was implemented by setting the index fallback
to follow a chain of “parents” upwards, possibly triggering
2-16](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-45-320.jpg)
![the index fallback again, until a table had the required field
or the chain ended.
After setting that index fallback, the code below printed
‘red’ even though ‘b’ did not have a ‘color’ field:
a=Window{x=100, y=200, color="red"}
b=Window{x=300, y=400, parent=a}
print(b.color)
There was nothing magical or hard-coded about delega-
tion through a “parent” field. Programmers had complete
freedom: they could use a different name for the field con-
taining the parent, they could implement multiple inheri-
tance by trying a list of parents, etc. Our decision not to
hard-code any of those possible behaviors led to one of the
main design concepts of Lua: meta-mechanisms. Instead of
littering the language with lots of features, we provided ways
for users to program the features themselves, in the way they
wanted them, and only for those features they needed.
Fallbacks greatly increased the expressiveness of Lua.
However, fallbacks were global handlers: there was only one
function for each event that could occur. As a consequence,
it was difficult to mix different inheritance mechanisms in
the same program, because there was only one hook for
implementing inheritance (the index fallback). While this
might not be a problem for a program written by a single
group on top of its own object system, it became a problem
when one group tried to use code from other groups, because
their visions of the object system might not be consistent
with each other. Hooks for different mechanisms could be
chained, but chaining was slow, complicated, error-prone,
and not very polite. Fallback chaining did not encourage
code sharing and reuse; in practice almost nobody did it.
This made it very hard to use third-party libraries.
Lua 2.1 allowed userdata to be tagged. In Lua 3.0 we
extended tags to all values and replaced fallbacks with tag
methods. Tag methods were fallbacks that operated only on
values with a given tag. This made it possible to implement
independent notions of inheritance, for instance. No chain-
ing was needed because tag methods for one tag did not af-
fect tag methods for another tag.
The tag method scheme worked very well and lasted
until Lua 5.0, when we replaced tags and tag methods by
metatables and metamethods. Metatables are just ordinary
Lua tables and so can be manipulated within Lua without
the need for special functions. Like tags, metatables can
be used to represent user-defined types with userdata and
tables: all objects of the same “type” should share the same
metatable. Unlike tags, metatables and their contents are
naturally collected when no references remain to them. (In
contrast, tags and their tag methods had to live until the
end of the program.) The introduction of metatables also
simplified the implementation: while tag methods had their
own private representation inside Lua’s core, metatables use
mainly the standard table machinery.
The code below shows the implementation of inheritance
in Lua 5.0. The index metamethod replaces the index tag
method and is represented by the ‘__index’ field in the
metatable. The code makes ‘b’ inherit from ‘a’ by setting
a metatable for ‘b’ whose ‘__index’ field points to ‘a’.
(In general, index metamethods are functions, but we have
allowed them to be tables to support simple inheritance by
delegation directly.)
a=Window{x=100, y=200, color="red"}
b=Window{x=300, y=400}
setmetatable(b,{ __index = a })
print(b.color) --> red
6.9 C API
Lua is provided as a library of C functions and macros that
allow the host program to communicate with Lua. This API
between Lua and C is one of the main components of Lua; it
is what makes Lua an embeddable language.
Like the rest of the language, the API has gone through
many changes during Lua’s evolution. Unlike the rest of the
language, however, the API design received little outside in-
fluence, mainly because there has been little research activity
in this area.
The API has always been bi-directional because, since
Lua 1.0, we have considered calling Lua from C and call-
ing C from Lua equally important. Being able to call Lua
from C is what makes Lua an extension language, that is, a
language for extending applications through configuration,
macros, and other end-user customizations. Being able to
call C from Lua makes Lua an extensible language, because
we can use C functions to extend Lua with new facilities.
(That is why we say that Lua is an extensible extension lan-
guage [30].) Common to both these aspects are two mis-
matches between C and Lua to which the API must adjust:
static typing in C versus dynamic typing in Lua and manual
memory management in C versus automatic garbage collec-
tion in Lua.
Currently, the C API solves both difficulties by using an
abstract stack15
to exchange data between Lua and C. Every
C function called by Lua gets a new stack frame that initially
contains the function arguments. If the C function wants to
return values to Lua, it pushes those values onto the stack
just before returning.
Each stack slot can hold a Lua value of any type. For each
Lua type that has a corresponding representation in C (e.g.,
strings and numbers), there are two API functions: an injec-
tion function, which pushes onto the stack a Lua value cor-
responding to the given C value; and a projection function,
which returns a C value corresponding to the Lua value at
a given stack position. Lua values that have no correspond-
ing representation in C (e.g., tables and functions) can be
manipulated via the API by using their stack positions.
15 Throughout this section, ‘stack’ always means this abstract stack. Lua
never accesses the C stack.
2-17](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-46-320.jpg)
![Practically all API functions get their operands from the
stack and push their results onto the stack. Since the stack
can hold values of any Lua type, these API functions operate
with any Lua type, thus solving the typing mismatch. To
prevent the collection of Lua values in use by C code, the
values in the stack are never collected. When a C function
returns, its Lua stack frame vanishes, automatically releasing
all Lua values that the C function was using. These values
will eventually be collected if no further references to them
exist. This solves the memory management mismatch.
It took us a long time to arrive at the current API. To
discuss how the API evolved, we use as illustration the
C equivalent of the following Lua function:
function foo(t)
return t.x
end
In words, this function receives a single parameter, which
should be a table, and returns the value stored at the ‘x’ field
in that table. Despite its simplicity, this example illustrates
three important issues in the API: how to get parameters,
how to index tables, and how to return results.
In Lua 1.0, we would write foo in C as follows:
void foo_l (void) {
lua_Object t = lua_getparam(1);
lua_Object r = lua_getfield(t, "x");
lua_pushobject(r);
}
Note that the required value is stored at the string index "x"
because ‘t.x’ is syntactic sugar for ‘t["x"]’. Note also that
all components of the API start with ‘lua_’ (or ‘LUA_’) to
avoid name clashes with other C libraries.
To export this C function to Lua with the name ‘foo’ we
would do
lua_register("foo", foo_l);
After that, foo could be called from Lua code just like any
other Lua function:
t = {x = 200}
print(foo(t)) --> 200
A key component of the API was the type lua_Object,
defined as follows:
typedef struct Object *lua_Object;
In words, lua_Object was an abstract type that represented
Lua values in C opaquely. Arguments given to C functions
were accessed by calling lua_getparam, which returned a
lua_Object. In the example, we call lua_getparam once
to get the table, which is supposed to be the first argument to
foo. (Extra arguments are silently ignored.) Once the table
is available in C (as a lua_Object), we get the value of
its "x" field by calling lua_getfield. This value is also
represented in C as a lua_Object, which is finally sent back
to Lua by pushing it onto the stack with lua_pushobject.
The stack was another key component of the API. It
was used to pass values from C to Lua. There was one
push function for each Lua type with a direct representation
in C: lua_pushnumber for numbers, lua_pushstring for
strings, and lua_pushnil, for the special value nil. There
was also lua_pushobject, which allowed C to pass back
to Lua an arbitrary Lua value. When a C function returned,
all values in the stack were returned to Lua as the results of
the C function (functions in Lua can return multiple values).
Conceptually, a lua_Object was a union type, since it
could refer to any Lua value. Several scripting languages,
including Perl, Python, and Ruby, still use a union type
to represent their values in C. The main drawback of this
representation is that it is hard to design a garbage collector
for the language. Without extra information, the garbage
collector cannot know whether a value has a reference to it
stored as a union in the C code. Without this knowledge,
the collector may collect the value, making the union a
dangling pointer. Even when this union is a local variable in
a C function, this C function can call Lua again and trigger
garbage collection.
Ruby solves this problem by inspecting the C stack, a task
that cannot be done in a portable way. Perl and Python solve
this problem by providing explicit reference-count functions
for these union values. Once you increment the reference
count of a value, the garbage collector will not collect that
value until you decrement the count to zero. However, it is
not easy for the programmer to keep these reference counts
right. Not only is it easy to make a mistake, but it is dif-
ficult to find the error later (as anyone who has ever de-
bugged memory leaks and dangling pointers can attest). Fur-
thermore, reference counting cannot deal with cyclic data
structures that become garbage.
Lua never provided such reference-count functions. Be-
fore Lua 2.1, the best you could do to ensure that an unan-
chored lua_Object was not collected was to avoid calling
Lua whenever you had a reference to such a lua_Object.
(As long as you could ensure that the value referred to by
the union was also stored in a Lua variable, you were safe.)
Lua 2.1 brought an important change: it kept track of all
lua_Object values passed to C, ensuring that they were not
collected while the C function was active. When the C func-
tion returned to Lua, then (and only then) all references to
these lua_Object values were released, so that they could
be collected.16
More specifically, in Lua 2.1 a lua_Object ceased to
be a pointer to Lua’s internal data structures and became an
index into an internal array that stored all values that had to
be given to C:
typedef unsigned int lua_Object;
This change made the use of lua_Object reliable: while a
value was in that array, it would not be collected by Lua.
16 A similar method is used by JNI to handle “local references”.
2-18](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-47-320.jpg)
![However, Lua 3 behavior had a major drawback: it com-
bined into a single primitive (lua_pushuserdata) two ba-
sic operations: userdata searching and userdata creation.
For instance, it was impossible to check whether a given
C pointer had a corresponding userdata without creating that
userdata. Also, it was impossible to create a new userdata re-
gardless of its C pointer. If Lua already had a userdata with
that value, no new userdata would be created.
Lua 4 mitigated that drawback by introducing a new func-
tion, lua_newuserdata. Unlike lua_pushuserdata, this
function always created a new userdata. Moreover, what was
more important at that time, those userdata were able to store
arbitrary C data, instead of pointers only. The user would tell
lua_newuserdata the amount memory to be allocated and
lua_newuserdata returned a pointer to the allocated area.
By having Lua allocate memory for the user, several com-
mon tasks related to userdata were simplified. For instance,
C code did not need to handle memory-allocation errors, be-
cause they were handled by Lua. More important, C code
did not need to handle memory deallocation: memory used
by such userdata was released by Lua automatically, when
the userdata was collected.
However, Lua 4 still did not offer a nice solution to the
search problem (i.e., finding a userdata given its C pointer).
So, it kept the lua_pushuserdata operation with its old be-
havior, resulting in a hybrid system. It was only in Lua 5 that
we removed lua_pushuserdata and dissociated userdata
creation and searching. Actually, Lua 5 removed the search-
ing facility altogether. Lua 5 also introduced light userdata,
which store plain C pointer values, exactly like regular user-
data in Lua 1. A program can use a weak table to associate
C pointers (represented as light userdata) to its correspond-
ing “heavy” userdata in Lua.
As is usual in the evolution of Lua, userdata in Lua 5
is more flexible than it was in Lua 4; it is also simpler to
explain and simpler to implement. For simple uses, which
only require storing a C structure, userdata in Lua 5 is trivial
to use. For more complex needs, such as those that require
mapping a C pointer back to a Lua userdata, Lua 5 offers
the mechanisms (light userdata and weak tables) for users to
implement strategies suited to their applications.
6.11 Reflectivity
Since its very first version Lua has supported some reflective
facilities. A major reason for this support was the proposed
use of Lua as a configuration language to replace SOL. As
described in §4, our idea was that the programmer could
use the language itself to write type-checking routines, if
needed.
For instance, if a user wrote something like
T = @track{ y=9, x=10, id="1992-34" }
we wanted to be able to check that the track did have a y
field and that this field was a number. We also wanted to be
able to check that the track did not have extraneous fields
(possibly to catch typing mistakes). For these two tasks, we
needed access to the type of a Lua value and a mechanism to
traverse a table and visit all its pairs.
Lua 1.0 provided the needed functionality with only two
functions, which still exist: type and next. The type func-
tion returns a string describing the type of any given value
("number", "nil", "table", etc.). The next function re-
ceives a table and a key and returns a “next” key in the ta-
ble (in an arbitrary order). The call next(t,nil) returns a
“first” key. With next we can traverse a table and process all
its pairs. For instance, the following code prints all pairs in a
table t:19
k = next(t,nil)
while k do
print(k,t[k])
k = next(t,k)
end
Both these functions have a simple implementation: type
checks the internal tag of the given value and returns the
corresponding string; next finds the given key in the table
and then goes to the next key, following the internal table
representation.
In languages like Java and Smalltalk, reflection must
reify concepts like classes, methods, and instance variables.
Moreover, that reification demands new concepts like meta-
classes (the class of a reified class). Lua needs nothing like
that. In Lua, most facilities provided by the Java reflective
package come for free: classes and modules are tables, meth-
ods are functions. So, Lua does not need any special mech-
anism to reify them; they are plain program values. Simi-
larly, Lua does not need special mechanisms to build method
calls at run time (because functions are first-class values and
Lua’s parameter-passing mechanism naturally supports call-
ing a function with a variable number of arguments), and it
does not need special mechanisms to access a global vari-
able or an instance variable given its name (because they are
regular table fields).20
7. Retrospect
In this section we give a brief critique of Lua’s evolutionary
process, discussing what has worked well, what we regret,
and what we do not really regret but could have done differ-
ently.
One thing that has worked really well was the early de-
cision (made in Lua 1.0) to have tables as the sole data-
structuring mechanism in Lua. Tables have proved to be
powerful and efficient. The central role of tables in the lan-
guage and in its implementation is one of the main character-
19 Although this code still works, the current idiom is ‘for k,v in
pairs(t) do print(k,v) end’.
20 Before Lua 4.0, global variables were stored in a special data structure
inside the core, and we provided a nextvar function to traverse it. Since
Lua 4.0, global variables are stored in a regular Lua table and nextvar is
no longer needed.
2-22](https://image.slidesharecdn.com/30192-250226174832-68abd05c/85/Proceedings-of-the-third-ACM-SIGPLAN-conference-on-History-of-programming-languages-Acm-Association-For-Computing-Machinery-51-320.jpg)
Proceedings of the third ACM SIGPLAN conference on History of programming languages Acm - Association For Computing Machinery
- 1. Get the full ebook with Bonus Features for a Better Reading Experience on ebookgate.com Proceedings of the third ACM SIGPLAN conference on History of programming languages Acm - Association For Computing Machinery https://ebookgate.com/product/proceedings-of-the-third-acm- sigplan-conference-on-history-of-programming-languages-acm- association-for-computing-machinery/ OR CLICK HERE DOWLOAD NOW Download more ebook instantly today at https://ebookgate.com
- 2. www.acm.org/sigplan/hopl III Sponsored by ACM SIGPLAN in cooperation with SIGSOFT June 9-10, 2007 San Diego, CA History of Programming Languages Conference HOPL
- 3. Proceedings The Third ACM SIGPLAN History of Programming Languages Conference (HOPL-III) San Diego, California, USA 9-10 June 2007 in cooperation with ACM SIGSOFT (co-located with FCRC 2007, 9-16 June 2007) Copyright © 2007 Association for Computing Machinery, Inc. (ACM) FM-1
- 4. History of Programming Languages Conference: HOPL-III Co-Chairs Introduction In 1978, at the first HOPL conference, Jean Sammet wrote: I'm happy to have this opportunity to open this Conference on the History of Programming Languages. It's something that I personally have been thinking about for almost seven years although the actual work on this has only been going on for the past one-and-a-half years. Most of you have not had the opportunity to look at the Preprints, and perhaps you haven't had too much time to even look at the program. For that reason I want to make sure that you understand something about what is intended, and frankly what is not intended be done in this conference. We view this as a start: perhaps the first of a number of conferences which will be held about various aspects of the computing field in general, software in particular, and even more specifically, programming languages. We hope that this conference and the Preprints and the final proceedings will stimulate many older people, many younger people, and many of those in the in-between category in the field to do work in history. This conference, I repeat again, is certainly not the last set of words to be held on this subject. It's only a beginning. I want to emphasize also that it's not a conference on the entire history of programming languages, nor even on the entire history of the languages that we selected. As many of you have seen from some of the earlier publicity that we put out, we're trying to consider and report on the technical factors which influenced the development of languages which satisfied a number of criteria. First, they were created and in use by 1967; they remain in use in 1977, which is when we made the decisions; and they've had considerable influence on the field of computing. The criteria for choosing a language include the following factors, although not every factor applied to each language: we considered usage, influence on language design, overall impact on the environment, novelty, and uniqueness. Particularly because of the cut-off date of 1967, some languages, which are in common use today, are not included. We definitely wanted a perspective of 10 years before we started worrying about the early history of the language. HOPL-I was a start and it did stimulate (some older and younger) people to continue the work of documenting the history of computing in general, and programming languages, in particular. HOPL-II followed in 1993. It extended the notion of history from HOPL-I to include the evolution of languages, language paradigms, and language constructs. It preserved the 10-year perspective. It also repeated the HOPL-I multi-year preparation of submissions, reviews, and re- reviews with teams of reviewers and experts, to come up with the best possible history papers from the people who were directly involved with the creation of their languages. Fifteen years later, HOPL-III is another step in the documentation of the history of FM-2
- 5. our field. Work began three years ago in 2004 to create a Program Committee, to establish paper solicitation criteria (see appendix to this proceedings), and to encourage submissions. As with its predecessors, the goal of HOPL-III was to produce an accurate historical record of programming language design and development. To achieve this goal, the Program Committee worked closely with prospective authors and outside experts to help ensure that all the papers were of high quality. As with HOPL-I and II, there were multiple rounds of reviewing to ensure that all the selected papers met requirements for both technical accuracy and historical completeness. The criteria for the programming languages considered appropriate for HOPL-III were: 1. The programming language came into existence before 1996, that is, it was designed and described at least 11 years before HOPL-III (2007). 2. The programming language has been widely used since 1998 either (i) commercially or (ii) within a specific domain. In either case, “widely used” implies use beyond its creators. 3. There also are some research languages which had great influence on widely used languages that followed them. As long as the research language was used by more than its own inventors, these will be considered to be appropriate for discussion at HOPL-III. The twelve papers in this proceedings represent original historical perspectives on programming languages that span at least five different programming paradigms and communities: object-oriented, functional, reactive, parallel, and scripting. At the time of the conference, the programming languages community continues to create broader mini-histories of each of those paradigms at http://en.wikipedia.org/wiki/HOPL A conference of this scope and level of preparation could not have happened without the time and assistance of many, many people. First we must thank our colleagues on the program committee Fran Allen, IBM Research (Emerita) Thomas J. (Tim) Bergin, American University (Emeritus) Andrew Black, Portland State University Koen Claessen, Chalmers University of Technology Kathleen Fisher, AT&T Research Susan L. Graham, University of California, Berkeley Julia Lawall, DIKU Doug Lea, SUNY Oswego Peter Lee, Carnegie Mellon University Michael S. Mahoney, Princeton University FM-3
- 6. Guy Steele, Sun Microsystems Benjamin Zorn, Microsoft Research and the authors of all the submitted papers. We must also thank the language experts who helped with the extensive paper reviews: Chris Espinoza, Gilad Bracha, Herb Sutter, Andrew Watson, Ulf Wiger, Vivek Sarkar, Norman Ramsey, Greg Nelson, Craig Chambers, and Kathy Yellick. We also thank the set of experts who helped seed the Wikipedia discussions of the language paradigms: Vijay Saraswat, Bard Bloom, Dipayan Gangopadhyay, and Guido van Rossum. Finally, we would like to thank the staff at ACM Headquarters, the SIGPLAN Executive Committee, the SIGSOFT Executive Committee, and Diana Priore, all of whom made this complex conference possible, and Joshua Hailpern, who designed both the HOPL CACM advertisement and the Proceedings and Final Program cover art. We also wish to acknowledge the generous financial support for HOPL-III that has been provided by: • An anonymous donor for multimedia capture/post-processing • Microsoft Research for manuscript copy-editing and proceedings preparation • IBM Research for subsidizing student registration and in support of program committee operations, the final program, and the HOPL-III website • ACM SIGPLAN Executive Committee This has been an ambitious and lengthy project for us; we are glad to see it successfully completed. We hope you enjoy both the conference presentations and the papers in these proceedings – a (partial) record of the past 15 years of our programming languages community. Barbara Ryder, Rutgers University Brent Hailpern, IBM Research HOPL-III Conference/Program Committee co-Chairs FM-4
- 7. HOPL-III Agenda: Saturday, 9 June 2007 08:45 - 09:00 Introduction 09:00 - 10:00 Keynote by Guy Steele (Sun Microsystems) and Richard P. Gabriel (IBM Research) 10:00 - 10:30 Break 10:30 - 12:20 "AppleScript" by William R. Cook (University of Texas at Austin) "The evolution of Lua" by Roberto Ierusalimschy (PUC-Rio), Luiz Henrique de Figueiredo (IMPA), and Waldemar Celes (PUC-Rio) 12:20 - 13:30 Lunch 13:30 - 15:20 "A history of Modula-2 and Oberon" by Niklaus Wirth (ETH Zurich) "Evolving a language in and for the real world: C++ 1991– 2006" by Bjarne Stroustrup (Texas A&M University and AT&T Labs - Research) 15:20 - 16:00 Break 16:00 - 17:50 "Statecharts in the making: a personal account" by David Harel (Weizmann Institute of Science) "A history of Erlang" by Joe Armstrong (Ericsson AB) FM-5
- 8. HOPL-III Agenda: Sunday, 10 June 2007 08:00 - 08:15 Introduction 08:15 - 10:05 "The rise and fall of High Performance Fortran: an historical object lesson" by Ken Kennedy (Rice University), Charles Koelbel (Rice University), Hans Zima (Institute of Scientific Computing, Unversity of Vienna and Jet Propulsion Laboratory, California Institute of Technology) "The design and development of ZPL" by Lawrence Snyder (University of Washington) 10:05 - 10:30 Break 10:30 - 12:20 "Self" by David Ungar (IBM Research), Randall B. Smith (Sun Microsystems) "The when, why and why not of the BETA programming language" by Bent Bruun Kristensen (University of Southern Denmark), Ole Lehrmann Madsen (University of Aarhus), Birger Møller-Pedersen (University of Oslo) 12:20 - 13:30 Lunch 13:30 - 15:20 "The development of the Emerald programming language" by Andrew P. Black (Portland State University), Norman C. Hutchinson (University of British Columbia), Eric Jul (University of Copenhagen) and Henry M. Levy (University of Washington) "A History of Haskell: being lazy with class" by Paul Hudak (Yale University), John Hughes (Chalmers University), Simon Peyton Jones (Microsoft Research), and Philip Wadler (University of Edinburgh) 15:20 - 15:40 Break 15:40 - 17:00 Panel: Programming Language Paradigms: Past, Present, and Future Kathleen Fisher (AT&T Labs - Research), chair Bertrand Meyer (ETH Zurich) Olin Shivers (Georgia Institute of Technology) Larry Wall Kathy Yelick (Universithy of California, Berkeley) FM-6
- 9. AppleScript William R. Cook University of Texas at Austin wcook@cs.utexas.edu With contributions from Warren Harris, Kurt Piersol, Dave Curbow, Donn Denman, Edmund Lai, Ron Lichty, Larry Tesler, Donald Olson, Mitchell Gass and Eric House Abstract AppleScript is a scripting language and environment for the Mac OS. Originally conceived in 1989, AppleScript allows end users to automate complex tasks and customize Mac OS applications. To automate tasks, AppleScript provides standard programming language features (control flow, vari- ables, data structures) and sends Apple Events to invoke ap- plication behavior. Apple Events are a variation on standard remote procedure calls in which messages can identify their arguments by queries that are interpreted by the remote ap- plication. This approach avoids the need for remote object pointers or proxies, and reduces the number of communi- cation round trips, which are expensive in high latency en- vironments like the early Macintosh OS. To customize an application that uses AppleScript’s Open Scripting Architec- ture, users attach scripts to application objects; these scripts can then intercept and modify application behavior. AppleScript was designed for casual users: AppleScript syntax resembles natural language, and scripts can be cre- ated easily by recording manual operations on a graphical interface. AppleScript also supported internationalization in allowing script to be presented in multiple dialects, including English, Japanese, or French. Although the naturalistic syn- tax is easy to read, it can make scripts much more difficult to write. Early adoption was hindered by the difficulty of mod- ifying applications to support Apple Events and the Open Scripting Architecture. Yet AppleScript is now widely used and is an essential differentiator of the Mac OS. Apple- Script’s communication model is a precursor to web ser- vices, and the idea of embedded scripting has been widely adopted. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Copyright c ° ACM [to be supplied]...$5.00 Categories and Subject Descriptors D.3 [Programming Languages] General Terms Languages, Design, Human Factors Keywords AppleScript, Scripting, History 1. Introduction The development of AppleScript was a long and complex process that spanned multiple teams, experienced several false starts and changes of plan, and required coordination between different projects and companies. It is difficult for any one person, or even a small group of people, to present a comprehensive history of such a project, especially without official support from the company for which the work was done. The email record of the team’s communications have been lost, and the author no longer has access to internal specifications and product plans. Nevertheless, I believe that the development of Apple- Script is a story worth telling, and I have been encouraged to attempt it despite the inherent difficulty of the task. I can of- fer only my own subjective views on the project, as someone who was intimately involved with all its aspects. I apologize in advance for errors and distortions that I will inevitably introduce into the story, in spite of my best efforts to be ac- curate. I first heard the idea of AppleScript over lunch with Kurt Piersol in February of 1991. The meeting was arranged by our mutual friend James Redfern. I knew James from Brown, where he was finishing his undergraduate degree after some time off, and I was working on my PhD. James and I both moved to California at about the same time, in 1988. We spent a lot of time together and I had heard a little about what he was up to, but he claimed it was secret. James arranged the meeting because Kurt was looking for someone to lead the AppleScript effort, and I was looking for something new to do. For the previous two and a half years I had been work- ing at HP Labs. I was a member of the Abel group, which included Walt Hill, Warren Harris, Peter Canning, and Wal- ter Olthoff. John Mitchell consulted with us from Stanford. The group was managed by Alan Snyder, whose formaliza- Permission to make digital/hard copy of part of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date of appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Permission may be requested from the Publications Dept., ACM, Inc., 2 Penn Plaza, New York, NY 11201-0701, USA, fax:+1(212) 869-0481, ©2007 ACM 978-1-59593-766-7/2007/06-ART1 $5.00 DOI 10.145/1238844.1238845 http://doi.acm.org/10.1145/1238844.1238845 Permission to make digital/hard copy of part of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date of appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Permission may be requested from the Publications Dept., ACM, Inc., 2 Penn Plaza, New York, NY 11201-0701, USA, fax:+1(212) 869-0481, permissions@acm.org ©2007 ACM 978-1-59593-766-7/2007/06-ART1 $5.00 DOI 10.1145/1238844.1238845 http://doi.acm.org/10.1145/1238844.1238845 1-1
- 10. tion of object concepts [35] was one basis for the develop- ment of CORBA. At HP Labs I finished writing my PhD the- sis, A Denotational Semantics of Inheritance [14, 19], which the Abel group used as the foundation for a number of pa- pers. We published papers on inheritance and subtyping [18], object-oriented abstraction [7, 16], mixins [5], F-Bounded polymorphism [6], and a fundamental flaw in the Eiffel type system [15]. I continued some of the work, analyzing the Smalltalk collection hierarchy [17], after I left HP. Near the end of 1990 HP Labs was undergoing a reorgani- zation and I was not sure I would fit into the new plan. In ad- dition, I was interested in making a change. I had developed several fairly large systems as an undergraduate, including a text editor and a graphical software design tool [39, 20] that were used by hundreds of other students for many years. As a graduate student I had focused on theory, but I wanted to learn the process of commercial software development. Ap- ple seemed like a reasonable place to try, so I agreed to talk with Kurt. 1.1 AppleScript Vision—Over Lunch Kurt and I hit it off immediately at lunch. Kurt Piersol is a large, friendly, eloquent man who was looking for people to work on the AppleScript project. Kurt graduated with a B.S. from the University of Louisville’s Speed Scientific School. He then worked at Xerox, building systems in Smalltalk- 80 and productizing research systems from Xerox PARC. Kurt was hired to work on AppleScript in 1989. He was originally asked to work on a development environment for the new language, but eventually took on the role of steering the project. Kurt and I discussed the advantages or disadvantages of command-line versus graphical user interfaces. With command-line interfaces, commonly used on Unix, users frequently write scripts that automate repeated sequences of program executions. The ability to pipe the output of one program into another program is a simple but powerful form of inter-application communication that allows small pro- grams to be integrated to perform larger tasks. For example, one can write a script that sends a customized version of a text file to a list of users. The sed stream editor can cre- ate the customized text file, which is then piped into the mail command for delivery. This new script can be saved as a mail-merge command, so that it is available for manual execution or invocation from other scripts. One appealing aspect of this model is its compositionality: users can create new commands that are invoked in the same way as built-in commands. This approach works well when atomic com- mands all operate on a common data structure, in this case text streams. It was not obvious that it would work for more complex structured data, like images, databases, or office documents, or for long-running programs that interact with users. With a graphical user interface (GUI) important func- tions, including the mail-merge command described above, are usually built into a larger product, e.g. a word processor. A GUI application offers pre-packaged integrated function- ality, so users need not combine basic commands to perform common tasks. Although careful design of graphical inter- faces eliminates the need for automation for basic use, there are still many tasks that users perform repeatedly within a graphical interface. The designers of the graphical interface cannot include commands to cover all these situations — if they did, then some users would execute these new com- mands repeatedly. No finite set of commands can ever satisfy all situations. Most users are happy with GUI applications and do not need a command-line interface or a scripting language. But there are clearly some limitations to the GUI approach on which Macintosh OS was based. Power users and system integrators were frustrated by the inability to build custom solutions by assembling multiple applications and special- izing them to particular tasks. Allowing users to automate the tasks that are relevant to them relieves pressure on the graphical user interface to include more and more special- ized features. The vision for AppleScript was to provide a kind of command-line interface to augment the power of GUI ap- plications and to bring this power to casual users. 1.2 Automation and Customization Kurt and I talked about two ways in which scripts and GUI applications could interact: for automation and for cus- tomization. Automation means that a script directs an ap- plication to perform a sequence of actions—the actions are performed “automatically” rather than manually. With au- tomation, the script is in control and one or more applica- tions respond to script requests. Customization occurs when a script is invoked from within an application—the script can perform “custom” actions that replace or augment the normal application behavior. With customization, the appli- cation manages and invokes scripts that users have attached to application objects. Automation is useful even without customization, but customization requires automation to be useful. We discussed whether there was sufficient benefit in pro- viding a standard platform for scripting, when custom so- lutions for each application might be better. Some applica- tions already had their own macro capability or a proprietary scripting language. However, this approach requires users to learn a different scripting language to automate each appli- cation. These languages typically include some variation on the basic elements of any programming language, including variables, loops, conditionals, data types, procedures, and exceptions. In addition, they may include special forms or constructs specific to the application in question. For exam- ple, a spreadsheet language can refer to cells, sheets, equa- tions and evaluation. One benefit of a standard scripting platform is that appli- cations can then be integrated with each other. This capa- 1-2
- 11. bility is important because users typically work with mul- tiple applications at the same time. In 1990, the user’s op- tions for integrating applications on the Macintosh OS were limited to shared files or copy/paste with the clipboard. If a repeated task involves multiple applications, there is lit- tle hope that one application will implement a single com- mand to perform the action. Integrating graphical applica- tions can be done at several levels: visual integration, behav- ioral integration, or data integration. Visual integration in- volves embedding one graphical component inside another; examples include a running Java applet inside a web page, or a specialized organization chart component displayed inside a word-processing document. Behavioral integration occurs when two components communicate with each other; exam- ples include workflow or invocation of a spell-check com- ponent from within a word processor. Data integration oc- curs whenever one application reads a file (or clipboard data) written by another application. A given system can include aspects of all three forms of integration. I agreed with Kurt that the most important need at that time was for behavioral integration. To compete with custom application-specific scripting languages, AppleScript would have to allow application-specific behaviors to be incorpo- rated into the language in as natural a way as possible, while preserving the benefits of a common underlying language. The core of the language should support the standard fea- tures of a programming language, including variables, pro- cedures, and basic data types. An application then provides a vocabulary of specific terminology that apply to the domain: a photo-processing application would manipulate images, pixels and colors, while a spreadsheet application would ma- nipulate cells, sheets, and graphs. The idea of AppleScript was to implement the “computer science boilerplate” once, while seamlessly integrating the vocabulary of the applica- tion domain so that users of the language can manipulate domain objects naturally. We discussed the vision of Apple- Script as a pervasive architecture for inter-application com- munication, so that it is easy to integrate multiple applica- tions with a script, or invoke the functionality of one appli- cation from a script in another application. We hoped that scripting would create a “network effect”, by which each new scriptable application improves the value of scripting for all other applications. 1.3 AppleScript Begins Soon after, Kurt offered me a job and I accepted quickly. This event illustrates one of the recurring characteristics of AppleScript: the basic idea is so compelling that it is enthusi- astically embraced by almost every software developer who is exposed to it. What was not immediately obvious was how difficult the vision was to achieve—not for strictly technical reasons, but because AppleScript required a fundamental refactoring, or at least augmentation, of almost the entire Macintosh code base. The demonstrable benefits of AppleScript’s vision has led developers to persevere in this massive task for the last twenty years; yet the work is truly Sisyphean, in that the slow incremental progress has been interrupted by major steps backward, first when the hardware was changed from the Motorola 68000 chip to the IBM PowerPC, and again when the operating system was reimplemented for Mac OS X. At this point it is impossible to identify one individual as the originator of the AppleScript vision. The basic idea is simple and has probably been independently discovered many times. The AppleScript team was successful in elab- orating the original vision into a practical system used by millions of people around the world. 2. Background When I started working at Apple in April 1991 I had never used a Macintosh computer. My first task was to understand the background and context in which AppleScript had to be built. The main influences I studied were the Macintosh op- erating system, HyperCard, and Apple Events. HyperCard was a good source of inspiration because it was a flexible application development environment with a scripting lan- guage embedded within it. A previous team had designed and implemented Apple Events to serve as the underlying mechanism for inter-application communication. The Apple Events Manager had to be shipped early so that it could be included in the Macintosh System 7 OS planned for summer 1991. When I started at Apple, the Apple Event Manager was in final beta testing. The fact that AppleScript and Apple Events were not designed together proved to be a continuing source of difficulties. Macintosh systems at that time had 4 to 8 megabytes of random-access memory (RAM) and a 40- to 60-megabyte hard drive. They had 25-50 MHz Motorola 68000 series processors. The entire company was internally testing Sys- tem 7.0, a major revision of the Macintosh OS. Applications on the Macintosh OS were designed around a main event processing loop, which handled lower-level keyboard and mouse events from the operating system [12]. The OS allowed an application to post a low-level event to another application, providing a simple form of inter- application communication. In this way one application could drive another application, by sending synthetic mouse and keyboard events that select menus or data and enter text into an application. This technique was used in two utility applications, MacroMaker and QuicKeys, which recorded and played back low-level user interface events. It was also used in the Macintosh help system, which could post low- level events to show the user how to use the system. Scripts that send low-level events are fragile, because they can fail if the position or size of user interface elements changes between the time the script is recorded and when it is run. They are also limited in the actions they can perform; low- 1-3
- 12. level events can be used to change the format of the current cell in a spreadsheet, but cannot read the contents of a cell. In the following section I describe these systems as they were described to me at the start of the AppleScript project, in April 1991. 2.1 HyperCard HyperCard [27, 30], originally released in 1987, was the most direct influence on AppleScript. HyperCard is a com- bination of a simple database, a collection of user interface widgets, and an English-like scripting language. These el- ements are organized around the metaphor of information on a collection of index cards. A collection of such cards is called a stack. A card could contain text, pictures, but- tons, and other graphical objects. Each object has many properties, including location on the card, size, font, style, etc. Cards with similar structure could use a common back- ground; a background defines the structure, but not the con- tent, of multiple cards. For example, a stack for household information might contain recipe cards and contact cards. The recipe cards use a recipe background that includes text fields for the recipe title, ingredients, and steps. The contact cards use a contact background with appropriate fields, in- cluding a photo. HyperCard scripts are written in HyperTalk, an English- like scripting language [28]. The language is for the most part a standard structured, imperative programming lan- guage. However, it introduced a unique approach to data structures: the stack, cards, objects and properties are used to store data. These structures are organized in a contain- ment hierarchy: stacks contain cards, cards contain objects, and properties exist on stacks, cards, and objects. This pre- defined structure makes it easy to build simple stacks, but more difficult to create custom data structures. Scripts in a stack can refer to the objects, collections of objects, and their properties by using chunk expressions. A chunk expression is best understood as a kind of query. For example, the following chunk expression refers to the text style property of a word element in a field of the current card: the textStyle of word 2 of card field ”Description” A chunk expression can refer to properties and elements of objects. A property is a single-valued attribute, for example textStyle . Elements are collections of objects identified by a type, for example word and card field . Element access may be followed by a name, index or range to select element(s) from the collection. Properties access distributes over col- lections; the following expression represents a a collection of 10 text style properties: the textStyle of character 1 to 10 of card field ”Description” HyperCard has a built-in set of property and collection names. Each object has a script containing procedures defined for that object. If the procedure name is an event name, then the procedure is a handler for that event— it is called when the event occurs for that object. For example, a button script may have handlers for mouseDown, mouseUp and mouseMove events. The following handler shows the next card when a button is released. on mouseUp go to next card end mouseUp Actions can be performed on chunk expressions to mod- ify the stack, its cards, the objects, or their properties. For example, clicking a button may run a script that moves to the next card, adds/removes cards, or modifies the contents of one or more cards. HyperCard has a set of predefined ac- tions, including set, go, add, close, etc. For example, the text style can be updated to a predefined constant bold: set the textStyle of character 1 to 10 of card field ”Description” to bold HyperCard 2.0 was released in 1990. HyperCard was very influential and widely used. Developers could easily create some applications in HyperCard, but to create more complex applications, they had to switch to more difficult general-purpose development tools. The need for unification of these approaches was recognized early at Apple, leading to the formation of a research and development project to build a new development platform for the Mac, discussed in the next section. Looking forward, the rapid develop- ment capabilities pioneered by HyperCard were added to more sophisticated general-purpose development environ- ments. This gradually reduced the need for systems like Hy- perCard, which was discontinued in 2004. 2.2 Family Farm Many of the ideas that eventually emerged in AppleScript were initially explored as part of a research project code- named Family Farm, which was undertaken in the Advanced Technology Group (ATG) at Apple, starting in 1989. The re- search team was led by Larry Tesler and included Mike Farr, Mitchell Gass, Mike Gough, Jed Harris, Al Hoffman, Ruben Kleiman, Edmund Lai, and Frank Ludolph. Larry received a B.S. in Mathematics from Stanford University in 1965. In 1963, he founded and ran IPC, one of the first software de- velopment firms in Palo Alto, CA. From 1968-73, he did research at the Stanford A.I. Lab on cognitive modeling and natural language understanding, where he designed and de- veloped PUB, one of the first markup languages with embed- ded tags and scripting. From 1973-80, he was a researcher at Xerox PARC, working on object-oriented languages, user interfaces, and desktop publishing. He joined Apple in 1980 1-4
- 13. to work on the Lisa. In 1986, he was named director of Ad- vanced Development, and in 1987, the first VP of Advanced Technology, a new group focused on research. The original goal of Family Farm was to create a new integrated development environment for the Macintosh OS. Family Farm included work on a new system-level pro- gramming language, an interprocess communication model, a user-level scripting language, and object component mod- els, in addition to other areas. The first AppleScript specification, which was the foun- dation for later development, was written by the Family Farm team. This document defined the basic approach to generalizing HyperTalk chunk expressions to create Apple- Script object-specifiers and Apple Events (described in de- tail below). I believe credit for these ideas must be shared equally by the Family Farm team, which generated many ideas, and the teams which turned these ideas into usable systems. As a research project that grew quickly, the organization put in place for Family Farm turned out not to be sufficient to build and deliver commercial products. The team was in the research group, not the system software group; No changes to the Macintosh system could be shipped to cus- tomers without approval of the system software group. And if Family Farm did get approval, they would have to follow the strict software development, scheduling, and quality con- trol processes enforced by the system software group. Over time it became clear that the Family Farm team was also too small to achieve its vision. After about a year and a half of work, the Family Farm project was disbanded, and new teams were created to design and implement some of the concepts investigated by Family Farm. One of the main themes to emerge from Family Farm was a focus on techniques for integrating applications. As men- tioned in Section 1.2, integrating graphical applications can be done at several levels: visual embedding, behavioral co- ordination, or data exchange. The spin-off projects were Ap- ple Events, AppleScript, and OpenDoc. The Apple Events project, formed in mid-1990 from a subset of the Family Farm team, developed the underlying communication model on which later projects were based. Later projects involved larger teams that pulled key members from outside Fam- ily Farm. AppleScript was next, then OpenDoc, both within the Developer Tools Group at Apple. AppleScript focused on data and behavioral integration. The OpenDoc project, which is not discussed in detail here, focused on visual inte- gration by embedding components. Family Farm’s transition from research to product development was a difficult one; in the end the primary product transferred from Family Farm to its descendants was an inspiring vision. 2.3 Apple Event Manager The Apple Event Manager provides an inter-application communication platform for the Macintosh. It was designed with scripting in mind—however, the design was completed before development of AppleScript began. When I started at Apple in April 1991, my first job was to do a complete code review of Apple Events, which was nearing the end of its beta testing period. I sat in a conference room with Ed Lai for over a week reading the code line by line and also jumping around to check assumptions and review interre- lationships. Ed was the primary developer of Apple Events code. The system was written in Pascal, as was most of the Macintosh system software of that era. The Apple Events team was part of the Developer Tools group and was orig- inally managed by Larry Tesler (who was also still VP in ATG), but was later taken over by Kurt Piersol. In designing Apple Events, Kurt, Ed and the team had to confront a serious limitation of the Macintosh OS: in 1990, the Macintosh OS could switch processes no more than 60 times a second. If a process performed only a few instructions before requesting a switch, the machine would idle until 1/60th of a second had elapsed. A fine-grained communication model, at the level of individual procedure or method calls between remote objects, would be far too slow: while a script within a single application could easily call thousands of methods in a fraction of a second, it would take several seconds to perform the same script if every method call required a remote message and process switch. As a result of this limitation of the OS, traditional remote procedure calls (RPC) could not be used. Fine-grained RPC was used in CORBA and COM, which were being developed at the same time. The Macintosh OS needed a communication model that allowed objects in remote applications to be manipulated without requiring many process round-trips. The Apple Events communication model uses a generalization of Hy- perCard’s chunk expressions. Just as a HyperCard command contains a verb and one or more chunk expressions in its predefined internal language, an Apple Event contains a verb and a set of chunk expressions that refer to objects and prop- erties in the target application. The generalized chunk ex- pressions are called object specifiers. Apple Events address the process-switching bottleneck by making it natural to pack more behavior into a single message, thereby reducing the need for communication round-trips between processes. An Apple Event is in effect a small query/update program that is formulated in one application and then sent to another application for interpretation. Kurt Piersol and Mike Farr debated whether there should be few commands that could operate on many objects, or a large number of specific commands as in traditional command-line interfaces. For example, on Unix there are different commands to delete print jobs (lprm), directories (rmdir), and processes (kill). The analogy with verbs and nouns in English helped Kurt win the argument for few commands (verbs) that operate on general objects (nouns). For example, in AppleScript there is a single delete com- 1-5
- 14. Document Paragraph Word Character Size Style Size Style Size Style Name Property Element A B A contains B Key: Root Figure 1. The properties and elements in a simple object model. mand that can delete paragraphs or characters from a word- processing document, or files from the file system. Having a small number of generic verbs, including set, copy, delete, insert, open and close, proved to be more extensible. 2.3.1 Object Specifiers Object specifiers are symbolic references to objects in an application. One application can create object specifiers that refer to objects in another application, called the target. The specifiers can then be included in an Apple Event and sent to the target application. These symbolic references are interpreted by the target application to locate the actual remote objects. The target application then performs the action specified by the verb in the Apple Event upon the objects. For example, a single Apple Event can be used to copy a paragraph from one location to another in a document; the source and target location are included in the event as object specifiers. The content of the paragraph remains local in the target application. Traditional RPC models would require the client to retrieve the paragraph contents, and then send it back to the target as a separate insertion operation. Object specifiers provide a view of application data that includes elements and properties. An element name repre- sents a collection of values; a property has a single value. The value of a property may be either a basic value, for ex- ample an integer or a string, or another object. Elements are always objects. A simple object model is illustrated in Figure 1. A document has multiple paragraph elements and a name property. A paragraph has style and size properties and contains word and character elements. The distinction between properties and elements is re- lated to the concept of cardinality in entity-relationship mod- eling [9] and UML class diagrams [32]. Cardinality indicates the maximum number of objects that may be involved in a relationship. The most important distinction is between single-valued (cardinality of 1) relationships and multi- valued (cardinality greater than 1). The entity-relationship model also includes attributes, which identify scalar, prim- itive data values. An AppleScript property is used for both attributes and single-valued relationships. Elements are used to describe multivalued relationships. The name identifying a set of elements is called a class name, identifying a specific kind of contained object and/or its role. For example, a Family object might have elements parents and children , which are elements that refer to sets of Person objects. Object specifiers allow application- specific names for elements and properties, which generalize the fixed set of predefined names available in HyperCard. Object specifiers also generalize HyperCard chunk ex- pressions in other ways. One extension was the addition of conditions to select elements of a collection based on their properties. This extension made object specifiers a form of query language with significant expressive power. Supporting Apple Events was frequently quite difficult. To create a new scriptable application, the software architect must design a scripting interface in addition to the traditional GUI interface. If the existing application architecture sepa- rates views (graphical presentations) from models (underly- ing information representation) [34], then adding scripting is usually possible. In this case the internal methods on the model may be exposed to scripts. There are two reasons why direct access to an existing object model may not be suffi- cient: 1. Users often want to control the user interface of an appli- cation, not just internal data of the application. Thus the scripting interface should provide access to both the view objects and the model objects. 2. The interfaces of the internal object model may not be suitable as an external scripting interface. In this case the scripting interface is usually implemented to provide another abstract view of the internal model. Even with a good architecture, it can be difficult to retrofit an existing GUI application to include a second interface for scripting. If the application does not use a model-view architecture, then adding scripting is much more difficult. The Apple Events team created a support library to assist application writers in interpreting object specifiers. It inter- prets nesting and conditions in the object specifiers, while using application callbacks to perform primitive property and element operations. 2.3.2 Apple Events Implementation Object specifiers are represented in Apple Events as nested record structures, called descriptors [11]. Descriptors use a self-describing, tagged tree data structured designed to be easily transported or stored in a flattened binary format. De- scriptors can either contain primitive data, a list of descrip- tors, or a labeled product of descriptors. Primitive data types include numbers (small and large integers and floats), pic- tures, styled and unstyled text, process IDs, files, and aliases. All the structures, types, and record fields are identified by 1-6
- 15. four-byte type codes. These codes are chosen to be human- readable to facilitate low-level debugging. Each kind of object specifier is a record with fields for the class, property name, index, and container. The container is either another object specifier record or null, representing the default or root container of the application. Events may be sent synchronously or asynchronously. The default behavior of Apple Events is stateless—the server does not maintain state for each client session. However, Apple Events sup- ports a simple form of transaction: multiple events can be tagged with a transaction ID, which requires an application to perform the events atomically or else signal an error. Apple Events was first released with Macintosh System 7 in 1991. The entire Apple Events system was designed, implemented, and shipped in the Macintosh OS before any products using it were built. It was a complex problem: ap- plications could not be built until the infrastructure existed, but the infrastructure could not be validated until many ap- plications had used it. In the end, the Apple Events team adopted a “build it and they will come” approach. They de- signed the system as well as they could to meet predicted needs. Only a few small sample applications were developed to validate the model. In addition, the operating system team defined four standard events with a single simple parame- ter: open application, open documents, print documents, and quit. These first basic events did not use object specifiers; the open and print events used a vector of path names as argu- ments. Later projects, including AppleScript, had to work around many of the Apple Events design choices that were made essentially within a vacuum. For example, Apple Events included some complex optimized message that were never used because they were too unwieldy. For example, if an array of values all has a common prefix, this prefix can be defined once and omitted in each element of the array. This was originally motivated by a desire to omit repetitive type information. This optimization is not used by AppleScript because it is difficult to detect when it could be used, and the reduction in message length provided by the optimization does not significantly affect performance. 3. The AppleScript Language My primary task was to lead the design and implementation of the AppleScript language. After I decided to join Apple I mentioned the opportunity to Warren Harris. I enjoyed working with Warren at HP and thought he would be a great addition to the AppleScript effort. Warren has a BS and MS inree EE from the University of Kansas. At HP Warren was still working on his “Abel Project Posthumous Report”, which contained all the ideas we had discussed, but had not time to complete, while working together at HP Labs [25]. Warren talked to Kurt and eventually decided to join the AppleScript team as a software engineer. He quickly became the co-architect and primary implementor of the language. 3.1 Requirements AppleScript is intended to be used by all users of the Mac- intosh OS. This does not imply that all users would use Ap- pleScript to the same degree or in the same way—there is clearly a wide range of sophistication in users of the Mac- intosh OS, and many of them have no interest in, or need for, learning even the simplest form of programming lan- guage. However, these users could invoke scripts created by other users, so there were important issues of packaging of scripts, in addition to developing them. More sophisticated users might be able to record or modify a script even if they could not write it. Finally, it should be possible for non- computer specialists to write scripts after some study. The language was primarily aimed at casual program- mers, a group consisting of programmers from all experi- ence levels. What distinguishes casual programming is that it is infrequent and in service of some larger goal—the pro- grammer is trying to get something else done, not create a program. Even an experienced software developer can be a casual programmer in AppleScript. The team recognized that scripting is a form of program- ming and requires more study and thought than using a graphical interface. The team felt that the language should be easy enough to learn and use to make it accessible to non-programmers, and that such users would learn enough programming as they went along to do the things that they needed to do. Programs were planned to be a few hundred lines long at most, written by one programmer and maintained by a series of programmers over widely spaced intervals. Scripts embedded inside an application were also expected to be small. In this case, the application provides the structure to interconnect scripts; one script may send messages to an application object that contains another script. Because scripts are small, compilation speed was not a significant concern. Readability was important because it was one way to check that scripts did not contain malicious code. In the early ’90s computer memory was still expensive enough that code size was a significant issue, so the Ap- pleScript compiler and execution engine needed to be as small as possible. Portability was not an issue, since the lan- guage was specifically designed for the Macintosh OS. Per- formance of scripts within an application was not a signifi- cant concern, because complex operations like image filter- ing or database lookup would be performed by applications running native code, not by scripts. Due to the high latency of process switches needed for communication, optimization of communication costs was the priority. 3.2 Application Terminologies An application terminology is a dictionary that defines the names of all the events, properties, and elements supported by an application. A terminology can define names as plural 1-7
- 16. reference ::= propertyName | ‘beginning’ | ‘end’ | ‘before’ term | ‘after’ term | ‘some’ singularClass | ‘ first ’ singularClass | ‘ last ’ singularClass | term (‘ st ’ | ‘nd’ | ‘rd’ | ‘th’) anyClass | ‘middle’ anyClass | plural [‘ from’ term toOrThrough term] | anyClass term [toOrThrough term] | singularClass ‘before’ term | singularClass ‘after’ term | term (‘of’ | ‘in’ | ‘’ s’) term | term (‘whose’ | ‘where’ | ‘that’) term plural ::= pluralClass | ‘every’ anyClass toOrThrough ::= ‘to’ | ‘thru’ | ‘through’ call ::= message ‘(’ expr∗ ‘)’ | message [‘in’ | ‘of ’] [term] arguments | term name arguments message ::= name | terminologyMessage arguments ::= ( preposition expression | flag | record)∗ flag ::= (‘with’ | ‘without’) [name]+ record ::= ‘given’ (name ‘:’ expr)∗ preposition ::= ‘to’ | ‘from’ | ‘thru’ | ‘through’ | ‘by’ | ‘on’ | ‘into’ | terminologyPreposition Figure 2. AppleScript grammar for object references and message sends. or masculine/feminine, and this information can be used by the custom parser for a dialect. One of the main functions of AppleScript is to send and receive Apple Events (defined in Section 2.3). Sending an Apple Event is analogous to an object-oriented message send, while handling an Apple Event is done by defining a method whose name is an Apple Event. One view of this arrangement is that each application is an object that re- sponds to a variety of messages containing complex argu- ment types. AppleScript encourages a higher-level view in which each application manages a set of objects to which messages may be sent. AppleScript also provides special syntax for manipulat- ing Apple Event object specifiers, which are called “object references” in AppleScript documentation. When an oper- ation is performed on an object reference, an Apple Event is created containing the appropriate verb and object spec- ifier. Thus AppleScript needed to provide a concise syntax for expressing Apple Events. These AppleScript expressions create object specifiers: the first word of paragraph 22 name of every figure of document ‘‘taxes ’’ the modification date of every file whose size > 1024 The first example is a reference to a particular word of a paragraph. The second refers to the collection of names associated with all figures in a document named “taxes”. The last one refers to the modification dates of large files. For example, the object reference name of window 1 identifies the name of the first window in an application. Object references are like first-class pointers that can be dereferenced or updated. An object reference is automati- cally dereferenced when a primitive value is needed: print the name of window 1 The primitive value associated with an object reference can be updated: set the name of window 1 to ‘‘Taxes’’ These examples illustrate that object specifiers can act as both l-values and r-values. Figure 2 includes the part of the AppleScript grammar relating to object specifiers. The grammar makes use of four nonterminals that represent symbols from the application terminology: for property, singularClass , pluralClass , and anyClass. As mentioned in Section 2.3.1, a terminology has properties and elements, which are identified by a class name. Property names are identified by lexical analysis and passed to the parser. For class names the terminology can in- clude both plural and singular forms or a generic “any” form. For example, name is a property, window is a singular class, and windows is a plural class. The grammar then accepts windows from 1 to 10 and every window from 1 to 10, Figure 2 also summarizes the syntax of messages. Argu- ments can be given by position or name after the given key- word. In addition, a set of standard prepositions can be used as argument labels. This allows messages of the form: copy paragraph 1 to end of document The first parameter is paragraph 1, and the second argu- ment is a prepositional argument named to with value end of document. One of the most difficult aspects of the language design arose from the fundamental ambiguity of object references: an object reference is itself a first-class value, but it also de- notes a particular object (or set of objects) within an applica- tion. In a sense an object reference is like a symbolic pointer, which can be dereferenced to obtain its value; the referenced value can also be updated or assigned through the object reference. The difficulties arose because of a desire to hide the distinction between a reference and its value. The solu- tion to this problem was to dereference them automatically when necessary, and require special syntax to create an ob- ject reference instead of accessing its value. The expression a reference to o creates a first-class reference to an object 1-8
- 17. described by o. Although the automatic dereferencing han- dles most cases, a script can explicitly dereference r using the expression value of r. The examples above can be ex- pressed using a reference value: set x to a reference to the name of window 1 The variable x can then be used in place of the original reference. The following statements illustrate the effect of operations involving x: print the value of x print x set the value of x to ‘‘ Taxes’’ set x to ‘‘ Taxes’’ The first and second statements both print the name of the window. In the first statement the dereference is explicit, but in the second it happens implicitly because print expects a string, not a reference. The third statement changes the name of the window, while the last one changes the values of the variable x but leaves the window unchanged. An object reference can be used as a base for further object specifications. set x to a reference to window 1 print the name of x Figure 3 and 4 describe the custom terminology dictio- nary for iChat, a chat, or instant messaging, program for the Macintosh. It illustrates the use of classes, elements, proper- ties, and custom events. Terminologies for large applications are quite extensive: the Microsoft Word 2004 AppleScript Reference is 529 pages long, and the Adobe Photoshop CS2 AppleScript Scripting Reference is 251 pages. They each have dozens of classes and hundreds of properties. In addition to the terminology interpreted by individ- ual applications, AppleScript has its own terminology to identify applications on multiple machines. The expression application ‘‘ name’’ identifies an application. The expres- sion application ‘‘ appName’’ of machine ‘‘machineName’’ refers to an application running on a specific machine. A block of commands can be targeted at an application using the tell statement: tell application ‘‘ Excel ’’ on machine x put 3.14 into cell 1 of row 2 of window 1 end This is an example of a static tell command, because the name of the target application is known statically, at compile time. The target of the tell statement can also be a dynamic value rather than an application literal. In this case the terminology is not loaded from the application. To communicate with a dynamic application using a statically specified terminology, a dynamic tell can be nested inside a static tell ; the outer one sets the static terminology, while the inner one defines the dynamic target. This brings up the possibility that applications may receive messages that they Class application: iChat application Plural form: applications Elements: account, service , window, document Properties: idle time integer Time in seconds that I have been idle. image picture My image as it appears in all services. status message string My status message, visible to other people while I am online. status string My status on all services: away/offline/available. Class service: An instant-messaging service Plural form: services Elements: account Properties: status string The status of the service.: disconnecting/connected/connecting/disconnected. id string The unique id of the service. name string The name of the service. image picture The image for the service. Class account: An account on a service Plural form: accounts Properties: status string away/offline/available/idle/unknown. id string The account’s service and handle. For ex- ample: AIM:JohnDoe007. handle string The account’s online name. name string The account’s name as it appears in the buddy list. status message The account’s status message. capabilities list The account’s messaging capabilities. image picture The account’s custom image. idle time integer The time in seconds the account has been idle. Figure 3. iChat Suite: Classes in the iChat scripting termi- nology [13]. 1-9
- 18. Events log in service Log in a service with an account. If the account password is not in the keychain the user will be prompted to enter one. log out service Logs out of a service, or all services if none is specified. send message to account Send account a text message or video invitation. Figure 4. iChat Suite: Events in the iChat scripting termi- nology [13]. do not understand. In such situations, the application should return an error. Integration of multiple applications opens the possibility that a single command may involve object references from multiple applications. The target of a message is determined by examining the arguments of the message. If all the argu- ments are references to the same application, then that appli- cation is the target. But if the arguments contain references to different applications, one of the applications must be cho- sen as the target. Since applications can interpret only their own object specifiers, the other object references must be evaluated to primitive values, which can then be sent to the target application. copy the name of the first window of application ”Excel” to the end of the first paragraph of app ”Scriptable Text Editor” This example illustrates copying between applications without using the global clipboard. AppleScript picks the target for an event by examining the first object reference in the argument list. If the argument list contains references to other applications, the values of the references are retrieved and passed in place of the references in the argument list. Standard events and reference structures are defined in the Apple Event Registry. The Registry is divided into suites that apply to domains of application. Suites contain spec- ifications for classes and their properties, and events. Cur- rently there are suites for core operations, text manipulation, databases, graphics, collaboration, and word services (spell- checking, etc.). Jon Pugh, with a BSCS from Western Washington Uni- versity in 1983, was in charge of the Apple Events registry. He also helped out with quality assurance and evangelism. Since then he has worked on numerous scriptable applica- tions, and created “Jon’s Commands,” a shareware library of AppleScript extensions. Terminologies also provide natural-language names for the four-letter codes used within an Apple Event. This meta- data is stored in an application as a resource. As discussed in Section 3.4 below, the terminology resources in an applica- tion are used when parsing scripts targeting that application. 3.3 Programming Language Features AppleScript’s programming language features include vari- ables and assignment, control flow, and basic data struc- tures. Control flow constructs include conditionals, a vari- ety of looping constructs, subroutines, and exceptions. Sub- routines allow positional, prepositional, and keyword para- meters. Data structures include records, lists, and objects. Destructuring bind, also known as pattern matching, can be used to break apart basic data structures. Lists and records are mutable. AppleScript also supports objects and a simple transaction mechanism. AppleScript has a simple object model. A script object contains properties and methods. Methods are dynamically dispatched, so script objects support a simple form of object- oriented programming. The following simple script declara- tion binds the name Counter to a new script object represent- ing a counter: script Counter property count : 0 to increment set count to count + 1 return count end increment end script A script declaration can be placed inside a method to create multiple instances. Such a method is called a factory method, and is similar to a constructor method in Java. Since script can access any lexically enclosing variables, all the objects created by a factory have access to the state of the object that constructed them. The resulting pattern of object references resembles the class/metaclass system in Smalltalk [23], although in much simpler form. AppleScript’s object model is a prototype model similar to that employed by Self [37], as opposed to the container- based inheritance model of HyperTalk. Script objects sup- port single inheritance by delegating unhandled commands to the value in their parent property [36]. JavaScript later adopted a model similar to AppleScript. The top level of every script is an implicit object decla- ration. Top-level properties are persistent, in that changes to properties are saved when the application running the script quits. A standalone script can be stored in a script file and executed by opening it in the Finder. Such a script can di- rect other applications to perform useful functions, but may also call other script files. Thus, script files provide a simple mechanism for modularity. AppleScript provides no explicit support for threading or synchronization. However, the application hosting a script can invoke scripts from multiple threads: the execution engine was thread-safe when run by the non-preemptive scheduling in the original Macintosh OS. It is not safe when 1-10
- 19. English the first character of every word whose style is bold Japanese French le premier caractère de tous les mots dont style est gras Professional { words | style == bold }.character[1] Figure 5. Illustration of dialects. run on multiple preemptive threads on Mac OS X. Script objects can also be sent to another machine, making mobile code possible. 3.4 Parsing and Internationalization The AppleScript parser integrates the terminology of ap- plications with its built-in language constructs. For ex- ample, when targeting the Microsoft ExcelTM application, spreadsheet terms are known by the parser—nouns like cell and formula, and verbs like recalculate . The statement tell application ‘‘ Excel ’’ introduces a block in which the Excel terminology is available. The terminology can contain names that are formed from multiple words; this means that the lexical analyzer must be able to recognize multiple words as a single logical identifier. As a result, lexical analysis depends upon the state of the parser: on en- tering a tell block, the lexical analysis tables are modified with new token definitions. The tables are reset when the parser reaches the end of the block. This approach increases flexibility but makes parsing more difficult. I believe the added complexity in lexing/parsing makes it more difficult for users to write scripts. Apple also required that AppleScript, like most of its other products, support localization, so that scripts could be read and written in languages other than English. Scripts are stored in a language-independent internal representation. A dialect defines a presentation for the internal language. Di- alects contain lexing and parsing tables, and printing rou- tines. A script can be presented using any dialect—so a script written using the English dialect can be viewed in Japanese. Examples are given in Figure 5. For complete localization, the application terminologies must also include entries for multiple languages. Apple developed dialects for Japanese and French. A “professional” dialect, which resembles Java, was created but not released. There are numerous difficulties in parsing a programming language that resembles a natural language. For example, Japanese does not have explicit separation between words. This is not a problem for language keywords and names from the terminology, but special conventions were required to recognize user-defined identifiers. Other languages have complex conjugation and agreement rules that are difficult to implement. Nonetheless, the internal representation of Ap- pleScript and the terminology resources contain information to support these features. The AppleScript parser was created using Yacc [29], a popular LALR parser generator. Poor error messages are a common problem with LALR parsing [1]. I wrote a tool that produces somewhat better error messages by including a simplified version of the follow set at the point where the error occurred. The follow set was simplified by replacing some common sets of symbols (like binary operators) with a generic name, so that the error message would be “ex- pected binary operator” instead of a list of every binary op- erator symbol. Despite these improvements, obscure error messages continue to be one of the biggest impediments to using AppleScript. 3.5 AppleScript Implementation During the design of AppleScript in mid-1991, we consid- ered building AppleScript on top of an existing language or runtime. We evaluated Macintosh Common Lisp (MCL), Franz Lisp, and Smalltalk systems from ParcPlace and Dig- italk. These were all good systems, but were not suitable as a foundation for AppleScript for the same reason: there was not sufficient separation between the development environ- ment and the runtime environment. Separating development from execution is useful because it a allows compiled script to be executed in a limited runtime environment with low overhead. The full environment would be needed only when compiling or debugging a script. Instead, we developed our own runtime and compiler. The runtime includes a garbage collector and byte-code inter- preter. The compiler and runtime were loaded separately to minimize memory footprint. One AppleScript T-shirt had the slogan “We don’t patch out the universe”. Many projects at Apple were implemented by “patching”: installing new functions in place of kernel operating system functions. The operating system had no protection mechanisms, so any function could be patched. Patches typically had to be installed in a particular order, or else they would not function. In addition, a bug in a patch could cause havoc for a wide range of applications. AppleScript did not patch any operating system func- tions. Instead the system was carefully packaged as a thread- safe QuickTime component. QuickTime components are a lightweight dynamic library mechanism introduced by the QuickTime team. Only one copy of the AppleScript com- piler and runtime was loaded and shared by all applica- tions on a machine. The careful packaging is one of the rea- sons AppleScript was able to continue running unchanged through numerous operating system upgrades, and even onto the PowerPC. 1-11
- 20. Integers Lists Strings Compile GetSource Display Data CoerceFromDesc CoerceToDesc Execute Storage Format Load Store ExecuteEvent Text Objects Commands Script Text Data Key: Interface … … Scripts Figure 6. Overview of the Open Scripting API. The AppleScript runtime is implemented in C++. The en- tire system, including dialects, is 118K lines of code, includ- ing comments and header files. Compiling the entire Apple- Script runtime took over an hour on the machines used by the development team. After the alpha milestone, the devel- opment team was not allowed to produce official builds for the testing team. Instead, the code had to be checked in and built by a separate group on a clean development environ- ment. This process typically took 8 to 12 hours, because the process was not fully automated, so there was sometimes a significant lag between identifying a bug and delivering a fix to the quality assurance team. This was a significant source of frustration for the overall team. 4. Script Management Open Scripting Architecture (OSA) allows any application to manipulate and execute scripts [11]. The Open Scripting API is centered around the notion of a script, as shown in Figure 6. A script is either a data value or a program. Many of the routines in the API are for translating between scripts and various external formats. Compile parses script text and creates a script object, while GetSource translates a script object back into human-readable script text. Display trans- lates a value into a printed representation. When applied to a string, e.g. ‘‘ Gustav’’, GetSource returns a program lit- eral ‘‘ Gustav’’, while Display just returns the text Gustav. CoerceFromDesc and CoerceToDesc convert AppleScript values to and from Apple Event descriptors. Load and Store convert to/from compact binary byte-streams that can be in- cluded in a file. The Execute function runs a script in a context. A context is a script that contains bindings for global variables. At its simplest, the script management API supports the construction of a basic script editor that can save scripts as stand-alone script applications. The OSA API does not include support for debugging, although this was frequently discussed by the team. How- ever, other companies have worked around this problem and created effective debugging tools (Section 6.3). 4.1 Embedding The script management API also supports attaching scripts to objects in an existing application. Such scripts can be trig- gered during normal use of the application. This usage is supported by the ExecuteEvent function, which takes as in- put a script and an Apple Event. The event is interpreted as a method call to the script. The corresponding method decla- ration in the script is invoked. In this way an application can pass Apple Events to scripts that are attached to application objects. Embedded scripts allow default application behavior to be customized, extended, or even replaced. For example, the Finder can run a script whenever a file is added to a folder, or an email package can run a script when new mail is re- ceived. Scripts can also be attached to new or existing menu items to add new functionality to an application. By em- bedding a universal scripting languages, application devel- opers do not need to build proprietary scripting languages, and users do not need to learn multiple languages. Users can also access multiple applications from a single script. AppleScript demonstrated the idea that a single scripting language could be used for all applications, while allowing application-specific behaviors to be incorporated so that the language was specialized for use in each application. Embedding can also be used to create entire applications. In this case there is no predefined application structure to which scripts are attached. Instead, the user builds the ap- plication objects — for data and user interfaces, and then at- taches scripts to them. Several application development tools based on AppleScript are described in Section 6. 4.2 Multiple Scripting Languages Halfway through the development of AppleScript, Apple management decided to allow third-party scripting lan- guages to be used in addition to AppleScript. A new API for managing scripts and scripting language runtime engines had to be designed and implemented. These changes con- tributed to delays in shipping AppleScript. However, they also led to a more robust architecture for embedding. In February of 1992, just before the first AppleScript al- pha release, Dave Winer convinced Apple management that having one scripting language would not be good for the Macintosh. At that time, Dave Winer was an experienced Macintosh developer, having created one of the first outliner applications, ThinkTank. In the early 1990s, Dave created an alternative scripting system, called Frontier. Before I joined the project, Apple had discussed the possibility of buying Frontier and using it instead of creating its own language. For some reason the deal fell through, but Dave continued developing Frontier. Apple does not like to take business away from developers, so when Dave complained that the impending release of AppleScript was interfering with his product, Apple decided the AppleScript should be opened up to multiple scripting languages. The AppleScript team mod- 1-12
- 21. ified the OSA APIs so that they could be implemented by multiple scripting systems, not just AppleScript. As a result, OSA is a generic interface between clients of scripting ser- vices and scripting systems that support a scripting language. Each script is tagged with the scripting system that created it, so clients can handle multiple kinds of script without know- ing which scripting system they belong to. Dave Winer’s Frontier is a complete scripting and ap- plication development environment that eventually became available as an Open Scripting component. Dave went on to participate in the design of web services and SOAP [4]. Tcl, JavaScript, Python and Perl have also been packaged as Open Scripting components. 4.3 Recording Events as Scripts The AppleScript infrastructure supports recording of events in recordable applications, which publish events in response to user actions. Donn Denman, a senior software engineer on the AppleScript team with a BS in Computer Science and Math from Antioch College, designed and implemented much of the infrastructure for recording. At Apple he worked on Basic interpreters. He was involved in some of the early discussions of AppleScript, worked on Apple Events and ap- plication terminologies in AppleScript. In addition, Donn created MacroMaker, a low-level event record and play- back system for Macintosh OS System 5. Working on Mar- coMaker gave Donn a lot of experience in how recording should work. Recording allows automatic generation of scripts for re- peated playback of what would otherwise be repetitive tasks. Recorded scripts can be subsequently generalized by users for more flexibility. This approach to scripting alleviates the “staring at a blank page” syndrome that can be so crippling to new scripters. Recording is also useful for learning the terminology of basic operations of an application, because it helps users to connect actions in the graphical interface with their symbolic expression in script. Recording high-level events is different from recording low-level events of the graphical user interface. Low-level events include mouse and keyboard events. Low-level events can also express user interface actions, e.g. “perform Open menu item in the File menu”, although the response to this event is usually to display a dialog box, not to open a par- ticular file. Additional low-level events are required to ma- nipulate dialog boxes by clicking on interface elements and buttons. Low-level events do not necessarily have the same effect if played back on a different machine, or when dif- ferent windows are open. High-level events are more robust because they express the intent of an action more directly. For example, a high-level event can indicate which file to open. Recording is supported by a special mode in the Apple Event manager, based on the idea that a user’s actions in manipulating a GUI interface can be described by a corre- sponding Apple Event. For example, if the user selects the File Open menu, then finds and selects a file named “Re- sumé” in a folder named “Personal”, the corresponding Ap- ple Event would be a FileOpen event containing the path “Personal:Resumé”. To be recordable, an application must post Apple Events that describe the actions a user performs with the GUI. Recordable applications can be difficult to build, since they must post an Apple Event describing each operation performed by a user. The AppleScript team promoted an ar- chitecture that turned this difficulty into a feature. We advo- cated that applications should be factored into two parts, a GUI and a back end, where the only communication from the GUI to the back end is via Apple Events. With this ar- chitecture, all the core functionality of the application must be exposed via Apple Events, so the application is inher- ently scriptable. The GUI’s job becomes one of translating low-level user input events (keystrokes and mouse move- ments) into high-level Apple Events. An application built in this way is inherently recordable; the Apple Event manager simply records the Apple Events that pass from the GUI to the back end. If an application is already scriptable, it can be made recordable by arranging for the user interface to communicate with the application model only through Ap- ple Events. The reality of recording is more complex, however. If there is a type Apple Event to add characters into a docu- ment, the GUI must forward each character immediately to the back end so that the user will see the result of typing. During recording, if the user types “Hello” the actions will record an undesirable script: type ‘‘ H” type ‘‘ e” type ‘‘ l” type ‘‘ l” type ‘‘ o” It would be better to record type ‘‘ Hello”. To get this effect, the GUI developer could buffer the typing and send a single event. But then the user will not see the effect of typing immediately. AppleEvents has the ability to specify certain events as record-only, meaning that it is a summary of a user’s actions and should not be executed. Creating such summaries makes developing a recordable application quite difficult. In 2006 twenty-five recordable applications were listed on Apple’s website and in the AppleScript Sourcebook [8], one of several repositories of information about AppleScript. Some, but fewer than half, of the major productivity appli- cations are recordable. Recordable applications include Mi- crosoft Word and Excel, Netscape Navigator, Quark Express (via a plugin) and CorelDRAW. One of the inherent difficulties of recording is the am- biguity of object specification. As the language of events becomes more rich, there may be many ways to describe a given user action. Each version might be appropriate for a 1-13
- 22. given situation, but the system cannot pick the correct action without knowing the intent of the user. For example, when closing a window, is the user closing the front window or the window specifically named “Example”? This is a well- known problem in research on programming by example, where multiple examples of a given action can be used to dis- ambiguate the user’s intent. Allen Cypher did fundamental research on this problem while at Apple. He built a prototype system called Eager that anticipated user actions by watch- ing previous actions [21, 22]. AppleScript does not have built-in support for analyzing multiple examples. There are also ambiguities when recording and embedding are com- bined: if a recorded event causes a script to execute, should the original event or the events generated by the script be recorded? Application designers must decide what is most appropriate for a given situation. Cypher worked with Dave Curbow in writing guidelines to help developers make these difficult choices [26]. Recording can also be used for other purposes. For ex- ample, a help system can include step-by-step instructions defined by a script. The steps can be played as normal scripts, or the user can be given the option of performing the steps manually under the supervision of the help system. By recording the user’s actions, the help system can pro- vide feedback or warnings when the user’s actions do not correspond to the script. 5. Development Process Unlike most programming languages, AppleScript was de- signed within a commercial software development project. The team members are listed in Figure 7. AppleScript was designed by neither an individual nor a committee; the team used a collaborative design process, with significant user testing and evaluation. The project leaders guided the process and made final decisions: there was lively debate within the group about how things should work. The ex- tended team included project management, user interface design and testing, documentation, and product marketing. The AppleScript project had a strong quality assurance (QA) team. They created a large test suite which was run against nightly builds. From a management viewpoint, the QA group also had significant control over the project, be- cause they were required to give final approval of a release. The project was code-named “Gustav” after Donn’s mas- sive Rottweiler dog. The dog slimed everything it came in contact with, and was the impetus behind a T-shirt that read “Script Happens”. The project tag line was “Pure Guava” because Gary Bond was designing a t-shirt that said “Apple- Script: Pure Gold” and Warren Harris got him to change it to Pure Guava after the Ween album he was in love with at the time. AppleScript and the associated tools were designed and implemented between 1990 and 1993. Figure 8 gives a time- line of the development process. The line labeled “changes” Jens Alfke Developer Greg Anderson Developer, Scriptable Finder Mike Askins Engineering Project Man- ager Gary Bond QA Scott Bongiorno QA, User Testing B. Bruce Brinson Developer Kevin Calhoun Manager Jennifer Chaffee User Interface Design Dan Clifford Developer William Cook Architect, Developer, Manager Sean Cotter Documentation Dave Curbow User Interface Design Donn Denman Developer Sue Dumont Developer, QA Mike Farr Marketing Mitch Gass Documentation Laura Clark Hamersley Marketing Warren Harris Architect, Developer Eric House QA, Developer Ron Karr QA, Apple Events Devel- oper Edmund Lai Developer, Apple Events Ron Lichty Manager, Finder Bennet Marks Developer Mark Minshull Manager Kazuhisa Ohta Developer, Dialects Donald Olson QA, Manager Chuck Piercey Marketing Kurt Piersol Architect James Redfern QA, Developer Brett Sher Developer, QA Laile Di Silvestro QA, Developer Sal Soghoian Product Manager Francis Stanbach Developer, Scriptable Finder Kazuhiko Tateda Japanese Dialect Larry Tesler Manager, VP Mark Thomas Evangelist Susan Watkins Marketing Figure 7. AppleScript and related project team members. 1-14
- 23. 0 20 40 60 80 100 120 140 160 180 200 6/1/91 7/1/91 7/31/91 8/30/91 9/29/91 10/29/91 11/28/91 12/28/91 1/27/92 2/26/92 3/27/92 4/26/92 5/26/92 6/25/92 7/25/92 8/24/92 9/23/92 10/23/92 11/22/92 12/22/92 1/21/93 2/20/93 3/22/93 4/21/93 5/21/93 6/20/93 7/20/93 8/19/93 9/18/93 10/18/93 11/17/93 12/17/93 1/16/94 2/15/94 f1 Final f3 b1 Beta b2 b3 b5 b4 AppleScript 1.0 a1 a17 a10 a3 AppleScript 1.1 Alpha b2 f1 b5 b3 Develoment Beta Alpha Dev. a6 a4 a1 Final 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 changes Number of file changes Number of release candidate builds Figure 8. Development statistics: number of file changes and candidate builds. shows the cumulative number of files changed during de- velopment (the scale is on the left). The second line shows the cumulative number of candidate release builds. The fi- nal candidate builds were created by Apple source control group from a clean set of sources and then given to the test- ing team. By placing source code control between develop- ment and testing, Apple ensured that each build could be recreated on a clean development environment with archived sources. Note that the number of file changes in the alpha or beta phases starts off slowly, then increases until just before the next milestone, when changes are no longer allowed un- less absolutely necessary. The AppleScript Beta was delivered in September 1992. In April 1993 the AppleScript 1.0 Developer’s Toolkit shipped, including interface declaration files (header files), sample code, sample applications and the Scripting Lan- guage Guide. The first end-user version, AppleScript 1.1, was released in September 1993 and included with System 7 Pro. In De- cember 1993, the 1.1 Developer’s Toolkit and Scripting Kit versions both released. In 1994, AppleScript was included as part of System 7.5. In January 1993, Apple management decided that the next version of AppleScript had to have more features than AppleScript 1.1, but that the development must be done with half the number of people. Since this was not likely to lead to a successful development process, Warren and I decided to leave Apple. Without leadership, the AppleScript group was disbanded. Many of the team members, including Jens Alfke, Donn Denman, and Donald Olson, joined Kurt Piersol on the OpenDoc team, which was working on visual integration of applications. AppleScript was integrated into the OpenDoc framework. 5.1 Documentation Internal documentation was ad hoc. The team made exten- sive use of an early collaborative document managemen- t/writing tool called Instant Update, that was used in a wiki- like fashion, a living document constantly updated with the current design. Instant Update provides a shared space of multiple documents that were viewed and edited simultane- ously by any number of users. Each user’s text was color- coded and time-stamped. I have not been able to recover a copy of this collection of documents. No formal semantics was created for the language, de- spite the fact that my PhD research and work at HP Labs was focused entirely on formal semantics of programming languages. One reason was that only one person on the team was familiar with formal semantics techniques, so writing a formal semantics would not be an effective means of com- munication. In addition, there wasn’t much point in develop- ing a formal semantics for the well-known features (objects, 1-15
- 24. inheritance, lexical closures, etc.), because the goal for this aspect of the language was to apply well-known constructs, not define new ones. There was no standard formal seman- tic framework for the novel aspects of AppleScript, espe- cially the notion of references for access to external objects residing in an application. The project did not have the lux- ury of undertaking extensive research into semantic founda- tions; its charter was to develop and ship a practical language in a short amount of time. Sketches of a formal semantics were developed, but the primary guidance for language de- sign came from solving practical problems and from user studies, rather than from a priori formal analysis. The public documentation was developed by professional writers who worked closely with the team throughout the project. The primary document is Inside Macintosh: Inter- application Communication, which includes details on the Apple Event Manager and Scripting Components [11]. The AppleScript language is also thoroughly documented [2], and numerous third-party books have been written about it, for examples see [31, 24]. Mitch Gass and Sean Cotter documented Apple Events and AppleScript for external use. Mitch has a bachelor’s degrees in comparative literature and computer science, and worked at Tandem and Amiga before joining Apple. Mitch worked during the entire project to provide that documentation, and in the process managed to be a significant communication point for the entire team. 5.2 User Testing Following Apple’s standard practice, we user-tested the lan- guage in a variety of ways. We identified novice users and asked them, “What do you think this script does?” The fol- lowing questions illustrate the kinds of questions asked dur- ing user testing. Part I. Please answer the following multiple choice ques- tions about AppleScript. 3. Given the handler: on doit from x to y with z return (x ∗ y) + z end doit What does the following statement evaluate to? doit with 3 from 8 to 5 a) 29 b) 43 c) error d) other: Part II. Please state which of the following AppleScript statements you prefer. 8. a) put ‘‘a ’’, {‘‘b ’’, ‘‘ c’’} into x b) put {‘‘a ’’, {‘‘b ’’, ‘‘ c’’}} into x 9. a) window named ‘‘fred’’ b) window ‘‘fred ’’ 10. a) window 1 b) window #1 11. a) word 34 b) word #34 12. a) ‘‘ apple ’’ < ‘‘betty ’’ b) ‘‘ apple ’’ comes before ‘‘betty ’’ Part III. This section shows sequences of commands and then asks questions about various variables after they are executed. 15. Given the commands: put {1, 2, 3} into x put x into y put 4 into item 1 of x What is x? a) {1, 2, 3} b) {4, 2, 3} c) error d) other: What is y? a) {1, 2, 3} b) {4, 2, 3} c) error d) other: Part IV. In this section, assume that all AppleScript state- ments refer to window 1, which contains the following text: this is a test of the emergency broadcast system 18. What does the following statement evaluate to? count every line of window 1 a) 2 b) 4, 5 c) 9 d) 14, 33 e) 47 f) error g) other: 1-16
- 25. What does the following statement evaluate to? count each line of window 1 a) 2 b) 4, 5 c) 9 d) 14, 33 e) 47 f) error g) other: 21. What does the following statement evaluate to? every line of window 1 whose first character = ‘‘x ’’ a) {} b) error c) other: One result of user testing concerned the choice of verb for assignment commands. The average user thought that after the command put x into y the variable x no longer retained its old value. The language was changed to use copy x into y instead. We also conducted interviews and a round-table discussion about what kind of functionality users would like to see in the system. In the summer of 1992, Apple briefed 50 key developers and collected reactions. The user interface team conducted controlled experiments of the usability of the language in early 1993, but since these took place during the beta-testing period, they were too late in the product development cycle to have fundamental impact. 6. AppleScript Applications and Tools Much of the practical power of AppleScript comes from the applications and tools that work with scripts and han- dle events. From the viewpoint of AppleScript, applications are large, well-designed and internally consistent libraries of specialized functionality and algorithms. So, when used with a database application, AppleScript can perform data- oriented operations. When used with a text layout applica- tion, AppleScript can automate typesetting processes. When used with a photo editing application, AppleScript can per- form complex image manipulation. Since new libraries can be created to cover any applica- tion domain, only the most basic data types were supported in AppleScript directly. For example, string handling was minimal in AppleScript. AppleScript’s capabilities were ini- tially limited by the availability of scriptable applications. Success of the project required that many applications and diverse parts of the operating system be updated to support scripting. A second benefit of pervasive scripting is that it can be used to provide a uniform interface to the operating sys- tem. With Unix, access to information in a machine is idio- syncratic, in the sense that one program was used to list print jobs, another to list users, another for files, and another for hardware configuration. I envisioned a way in which all these different kinds of information could be referenced uni- formly. A uniform naming model allows every piece of infor- mation anywhere in the system, be it an application or the operating system, to be accessed and updated uniformly. Application-specific terminologies allow applications to be accessed uniformly; an operating system terminology would provide access to printer queues, processor attributes, or net- work configurations. Thus, the language must support mul- tiple terminologies simultaneously so that a single script can access objects from multiple applications and the operating system at the same time. 6.1 Scriptable Finder Having a scriptable Finder was a critical requirement for AppleScript, since the Finder provides access to most system resources. However, it was difficult to coordinate schedules and priorities because the Finder and AppleScript teams were in different divisions within Apple. The Finder team was also pulled in many directions at once. As a result, Finder was not fully scriptable when Apple- Script shipped in 1992. The Finder team created a separate library, called the “Finder scripting extension”, to provide some additional Finder script commands. The Finder had been rewritten in C++ from the ground up for System 7 to be extensible. But extensions relied on internal C++ dispatch tables, so the Finder was not dynamically extensible: it had to be recompiled for each extension. The Finder extension mechanism had been designed so that Finder functionality could grow incrementally. It was the mechanism for adding large quantities of new functionality to support a specific project. It was not until 1997 that a scriptable Finder was released. A year later the Finder supported embedding, which greatly increased its power. Embedding allowed scripts to be trig- gered from within the Finder in response to events, for ex- ample opening a folder or emptying the trash. 6.2 Publishing Workflow Automation of publishing workflows is a good illustration of AppleScript and scriptable applications. Consider the au- tomation of a catalog publishing system. An office-products company keeps all its product information in a FileMaker ProTM database that includes descriptions, prices, special of- fer information, and a product code. The product code iden- tifies a picture of the product in a KudosTM image database. The final catalog is a QuarkXPressTM document that is ready for printing. Previously, the catalog was produced manually, 1-17
- 26. a task that took a team of twenty up to a month for a single catalog. An AppleScript script automates the entire process. The script reads the price and descriptive text from the FileMaker Pro database and inserts it into appropriate QuarkXPress fields. The script applies special formatting: it deletes the decimal point in the prices and superscripts the cents (e.g. 3499 ). To make the text fit precisely in the width of the en- closing box, the script computes a fractional expansion fac- tor for the text by dividing the width of the box by the width of the text (this task was previously done with a calculator). It adjusts colors and sets the first line of the description in boldface type. Finally, it adds special markers like “Buy 2 get 1 free” and “Sale price $1799 ” where specified by the database. Once this process is automated, one person can produce the entire catalog in under a day, a tiny fraction of the time taken by the manual process. It also reduced errors during copying and formatting. Of course, creating and maintaining the scripts takes time, but the overall time is significantly reduced over the long run. 6.3 Scripting Tools AppleScript included a simple and elegant script editor cre- ated by Jens Alfke, who had graduated from Caltech and worked with Kurt Piersol at Xerox Parc on Smallktalk-80 applications. Jens was one of the key developers on the Ap- pleScript team; he focused on tools, consistency of the APIs and usability of the overall system. Soon after AppleScript was released, more powerful script development environments were created outside Ap- ple. They addressed one of the major weaknesses of Apple- Script: lack of support for debugging. One developer outside Apple who took on this challenge is Cal Simone, who has also been an unofficial evangelist for AppleScript since its inception. Cal created Scripter, which allows users to single- step through a script. It works by breaking a script up into individual lines that are compiled and executed separately. The enclosing tell statements are preserved around each line as it is executed. Scripter also allows inspection of local variables and execution of immediate commands within the context of the suspended script. Script Debugger uses a dif- ferent technique: it adds a special Apple Event between each line of a script. The Apple Event is caught by the debugger and the processing of the script is suspended. The current values of variables can then be inspected. To continue the script, the debugger simply returns from the event. AppleScript also enables creation of sophisticated inter- face builders. The interface elements post messages when a user interacts with them. The user arranges the elements into windows, menus, and dialogs. Scripts may be attached to any object in the interface to intercept the messages be- ing sent by the interface elements and provide sophisticated behavior and linking between the elements. Early applica- tion builders included FrontmostTM , a window and dialog builder, and AgentBuilderTM , which specialized in commu- nication front-ends. Version 2.2 of HyperCard, released in 1992, included support for OSA, so that AppleScript or any OSA language could be used in place of HyperTalk. Two major application builders have emerged recently. FaceSpan, originally released in 1994, has grown into a full- featured application development tool. FaceSpan includes an integrated script debugger. Apple released AppleScript Studio in 2002 as part of its XCode development platform. A complete application can be developed with a wide range of standard user interface elements to which scripts can be attached. AppleScript Studio won Macworld Best of Show Awards at the 2001 Seybold Conference in San Francisco. In 2005 Apple released Automator, a tool for creating sequences of actions that define workflows. Automator se- quences are not stored or executed as AppleScripts, but can contain AppleScripts as primitive actions. The most inter- esting thing about Automator is that each action has an input and an output, much like a command in a Unix pipe. The resulting model is quite intuitive and easy to use for simple automation tasks. Although Apple Events are normally handled by appli- cations, it is also possible to install system event handlers. When an Apple Event is delivered to an application, the ap- plication may handle the event or indicate that it was not han- dled. When an application does not handle an event, the Ap- ple Event manager searches for a system event handler. Sys- tem event handlers are packaged in script extensions (also known as OSAX) and are installed on the system via Script- ing Additions that are loaded when the system starts up. 6.4 Scriptable Applications Eventually, a wide range of scriptable applications became available: there are currently 160 scriptable applications listed on the Apple web site and the AppleScript source- book [8]. Every kind of application is present, including word processors, databases, file compression utilities, and development tools. Many of the most popular applications are scriptable, including Microsoft Office, Adobe Photo- shop, Quark Expression, FileMaker, and Illustrator. In ad- dition, most components of the Mac OS are scriptable, including the Finder, QuickTime Player, Address Book, iTunes, Mail, Safari Browser, AppleWorks, DVD Player, Help Viewer, iCal, iChat, iSync, iPhoto, and control panels. Other systems also benefitted from the infrastructure cre- ated by AppleScript. The Macintosh AVTM speech recogni- tion system uses AppleScript, so any scriptable application can be driven using speech. 7. Evolution After version 1.1, the evolution of AppleScript was driven primarily by changes in the Macintosh OS. Since Apple- Script was first released, the operating system has undergone two major shifts, first when Apple moved from the Motorola 1-18
- 27. 68000 to the PowerPC chip, and then when it moved from the Classic Macintosh OS to the Unix-based OS X. Few changes were made to the language itself, while scriptable applications and operating system components experienced rapid expansion and evolution. A detailed history with dis- cussion of new features, bugs, and fixes can be found in the AppleScript Sourcebook [8], which we summarize here. The first upgrade to AppleScript, version 1.1.2, was cre- ated for Macintosh OS 8.0, introduced in July 1997. De- spite the rigorous source code configuration process (see Section 5), Apple could not figure out how to compile the system and contracted with Warren Harris to help with the job. A number of bugs were fixed and some small enhance- ments were made to conform to Macintosh OS 8.0 standards. At the same time several system applications and extensions were changed in ways that could break old scripts. The most important improvement was a new scriptable Finder, which eliminated the need for a Finder scripting extension. In 1997 AppleScript was at the top of the list of features to eliminate in order to save money. Cal Simone, mentioned in Section 6.3, successfully rallied customers to rescue Ap- pleScript. In October 1998 Apple released AppleScript 1.3 with UNICODE support recompiled as a native PowerPC exten- sion; however, the Apple Events Manager was still emulated as Motorola 68000 code. The dialect feature was no longer supported; English became the single standard dialect. This version came much closer to realizing the vision of uni- form access to all system resources from scripts. At least 30 different system components, including File Sharing, Ap- ple Video Player and Users & Groups, were now scriptable. New scriptable applications appeared as well, including Mi- crosoft Internet Explorer and Outlook Express. The PowerPC version of AppleScript received an Eddy Award from MacWorld as “Technology of the Year” for 1998 and was also demonstrated in Steve Jobs’ Seybold 1998 address. In 2006, MacWorld placed AppleScript as #17 on its list of the 30 most significant Mac products ever. AppleScript was a long-term investment in fundamental in- frastructure that took many years to pay dividends. The most significant language changes involved the tell statement. For example, the machine class used to identify remote applications was extended to accept URLs (see Sec- tion 3.2), allowing AppleScript control of remote applica- tions via TCP/IP. When Mac OS X was released in March 2001, it included AppleScript 1.6. In porting applications and system compo- nents to OS X, Apple sometimes sacrificed scripting support. As a result, there was a significant reduction in the number of scriptable applications after the release of OS X. Full script- ability is being restored slowly in later releases. In October 2006, Google reported an estimated 8,570,000 hits for the word “AppleScript”. 8. Evaluation AppleScript was developed by a small group with a short schedule, a tight budget and a big job. There was neither time nor money to fully research design choices. AppleScript and Apple Events introduced a new approach to remote communication in high-latency environments [33]. Object references are symbolic paths, or queries, that iden- tify one or more objects in an application. When a command is applied to an object reference, both the command and the object reference are sent (as an Apple Event containing an object specifier) to the application hosting the target object. The application interprets the object specifier and then per- forms the action on the specified objects. In summary, AppleScript views an application as a form of object-oriented database. The application publishes a specialized terminology containing verbs and nouns that describe the logical structure and behavior of its objects. Names in the terminology are composed using a standard query language to create programs that are executed by the remote application. The execution model does not involve remote object references and proxies as in CORBA. Rather than send each field access and method individually to the remote application and creating proxies to represent inter- mediate values, AppleScript sends the entire command to the remote application for execution. From a pure object- oriented viewpoint, the entire application is the only real ob- ject; the “objects” within it are identified only by symbolic references, or queries. After completing AppleScript, I learned about COM and was impressed with its approach to distributed object- oriented programming. Its consistent use of interfaces en- ables interoperability between different systems and lan- guages. Although interface negotiation is complex, invoking a method through an interface is highly optimized. This ap- proach allows fine-grained objects that are tightly coupled through shared binary interfaces. For many years I believed that COM and CORBA would beat the AppleScript com- munication model in the long run. However, recent develop- ments have made me realize that this may not be the case. AppleScript uses a large-granularity messaging model that has many similarities to the web service standards that began to emerge in 1999 [10]. Both are loosely coupled and support large-granularity communication. Apple Events data descriptors are similar to XML in that they describe arbi- trary labeled tree structures without fixed semantics. Apple- Script terminologies are similar to web service description language (WSDL) files. It is perhaps not an accident that Dave Winer, who worked extensively with AppleScript and Apple Events, is also one of the original developers of web service models. There may be useful lessons to be learned for web services, given that AppleScript represents a sig- nificant body of experience with large-granularity messag- ing. One difference is that AppleScript includes a standard query model for identifying remote objects. A similar ap- 1-19
- 28. proach could be useful for web services. As I write in 2006, I suspect that COM and CORBA will be overwhelmed by web services, although the outcome is far from certain now. AppleScript is also similar to traditional database inter- faces like ODBC [38]. In AppleScript the query model is integrated directly into the language, rather than being exe- cuted as strings as in ODBC. A similar approach has been adopted by Microsoft for describing queries in .NET lan- guages [3]. User tests revealed that casual users don’t easily under- stand the idea of references, or having multiple references to the same value. It is easier to understand a model in which values are copied or moved, rather than assigning references. The feedback from user tests in early 1993 was too late in the development cycle to address this issue with anything more than a cosmetic change, to use copy instead of set for as- signment. Writing scriptable applications is difficult. Just as user interface design requires judgment and training, creating a good scripting interface requires a lot of knowledge and careful design. It is too difficult for application developers to create terminologies that work well in the naturalistic grammar. They must pay careful attention to the linguistic properties of the names they choose. The experiment in designing a language that resembled natural languages (English and Japanese) was not success- ful. It was assumed that scripts should be presented in “nat- ural language” so that average people could read and write them. This lead to the invention of multi-token keywords and the ability to disambiguate tokens without spaces for Japanese Kanji. In the end the syntactic variations and flex- ibility did more to confuse programmers than to help them out. It is not clear whether it is easier for novice users to work with a scripting language that resembles natural lan- guage, with all its special cases and idiosyncrasies. The main problem is that AppleScript only appears to be a natural language: in fact, it is an artificial language, like any other programming language. Recording was successful, but even small changes to the script may introduce subtle syntactic er- rors that baffle users. It is easy to read AppleScript, but quite hard to write it. When writing programs or scripts, users prefer a more conventional programming language structure. Later ver- sions of AppleScript dropped support for dialects. In hind- sight, we believe that AppleScript should have adopted the Professional Dialect that was developed but never shipped. Finally, readability was no substitute for an effective se- curity mechanism. Most people just run scripts—they don’t read or write them. 9. Conclusion AppleScript is widely used today and is a core technol- ogy of Mac OS X. Many applications, including Quark Ex- press, Microsoft Office, and FileMaker, support scripting. Small scripts are used to automate repetitive tasks. Larger scripts have been developed for database publishing, docu- ment preparation, and even web applications. There are many interesting lessons to be learned from AppleScript. On a technical level, its model of pluggable embedded scripting languages has become commonplace. The communication mechanism of Apple Events, which is certainly inferior to RPC mechanisms for single-machine or in-process interactions, may turn out to be a good model for large-granularity communication models such as web services. Many of the current problems in AppleScript can be traced to the use of syntax based on natural language; however, the ability to create pluggable dialects may provide a solution in the future, by creating a new syntax based on conventional programming languages. Acknowledgments Thanks to Jens Alfke, Paul Berkowitz, Bill Cheeseman, Chris Espinosa, Michael Farr, Steve Goldband Tom Ham- mer, David Hoerl, Alexander Kellett, Wayne Malkin, Matt Neuburg, Chuck Piercey, Hamish Sanderson, and Stephen Weyl, for discussions about this paper. Special thanks to Andrew Black and Kathleen Fisher for their guidance, en- couragement, flexibility, and careful reading of my work in progress. References [1] Alfred V. Aho and Jeffrey D. Ullman. Principles of Compiler Design (Addison-Wesley series in computer science and information processing). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1977. [2] Apple Computer Inc. AppleScript Language Guide. Addison- Wesley, 1993. [3] Gavin Bierman, Erik Meijer, and Wolfram Schulte. The essence of data access in cω. In European Conference on Object-Oriented Programming. Springer Verlag, 2005. [4] Don Box, David EhneBuske, Gopal Kakivaya, Andrew Layman, Noah Mendelsohn, Henrik Frystyk Nielson, Satish Thatte, and Dave Winer. Simple object access protocol 1.1. http://www.w3.org/TR/SOAP. [5] Gilad Bracha and William Cook. Mixin-based inheritance. In Proc. of ACM Conf. on Object-Oriented Programming, Systems, Languages and Applications, pages 303–311, 1990. [6] Peter Canning, William Cook, Walt Hill, John Mitchell, and Walter Olthoff. F-bounded polymorphism for object- oriented programming. In Proc. of Conf. on Functional Programming Languages and Computer Architecture, pages 273–280, 1989. [7] Peter Canning, William Cook, Walt Hill, and Walter Olthoff. Interfaces for strongly-typed object-oriented programming. In Proc. of ACM Conf. on Object-Oriented Programming, Systems, Languages and Applications, pages 457–467, 1989. [8] Bill Cheeseman. Applescript sourcebook. http://www. AppleScriptSourcebook.com. 1-20
- 29. [9] Peter P. Chen. The entity-relationship model — toward a unified view of data. ACM Transactions on Database Systems (TODS), 1(1):9–36, 1976. [10] Roberto Chinnici, Martin Gudgin, Jean-Jacques Moreau, and Sanjiva Weerawarana. Web Services Description Language Version 1.2, July 2002. W3C Working Draft 9. [11] Apple Computer. Inside Macintosh: Interraplication Com- munication. Addison-Wesley, 1991. [12] Apple Computer. Inside Macintosh: Macintosh Toolbox Essentials. Addison-Wesley, 1991. [13] Apple Computer. ichat 2.0 dictionary. FooDoo Lounge web site by Richard Morton, 2002-2005. [14] William Cook. A Denotational Semantics of Inheritance. PhD thesis, Brown University, 1989. [15] William Cook. A proposal for making Eiffel type-safe. In Proc. European Conf. on Object-Oriented Programming, pages 57–70. British Computing Society Workshop Series, 1989. Also in The Computer Journal, 32(4):305–311, 1989. [16] William Cook. Object-oriented programming versus abstract data types. In Proc. of the REX Workshop/School on the Foundations of Object-Oriented Languages, volume 173 of Lecture Notes in Computer Science. Springer-Verlag, 1990. [17] William Cook. Interfaces and specifications for the Smalltalk collection classes. In Proc. of ACM Conf. on Object-Oriented Programming, Systems, Languages and Applications, 1992. [18] William Cook, Walt Hill, and Peter Canning. Inheritance is not subtyping. In Proc. of the ACM Symp. on Principles of Programming Languages, pages 125–135, 1990. [19] William Cook and Jens Palsberg. A denotational semantics of inheritance and its correctness. In Proc. of ACM Conf. on Object-Oriented Programming, Systems, Languages and Applications, pages 433–444, 1989. [20] William R. Cook and Victor Law. An algorithm editor for structured design (abstract). In Proc. of the ACM Computer Science Conference, 1983. [21] Allen Cypher. Eager: programming repetitive tasks by example. In CHI ’91: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 33–39, New York, NY, USA, 1991. ACM Press. [22] Allen Cypher, editor. Watch What I Do – Programming by Demonstration. MIT Press, Cambridge, MA, USA, 1993. Full text available at web.media.mit.edu/ lieber/PBE/. [23] A. Goldberg and D. Robson. Smalltalk-80: the Language and Its Implementation. Addison-Wesley, 1983. [24] A. Goldstein. AppleScript: The Missing Manual. O’Reilly, 2005. [25] Warren Harris. Abel posthumous report. HP Labs, 1993. [26] Apple Computer Inc. Scripting interface guidelines. Techni- cal Report TN2106, Apple Computer Inc. [27] Apple Computer Inc. HyperCard User’s Guide. Addison Wesley, 1987. [28] Apple Computer Inc. HyperCard Script Language Guide: The HyperTalk Language. Addison Wesley, 1988. [29] Steven C. Johnson. Yacc: Yet another compiler compiler. In UNIX Programmer’s Manual, volume 2, pages 353–387. Holt, Rinehart, and Winston, New York, NY, USA, 1979. [30] S. Michel. HyperCard: The Complete Reference. Osborne Mc-GrawHill, 1989. [31] M. Neuburg. AppleScript : The Definitive Guide. O’Reilly, 2003. [32] Object Management Group. OMG Unified Modeling Language Specification Version 1.5, March 2003. [33] David A. Patterson. Latency lags bandwith. Commun. ACM, 47(10):71–75, 2004. [34] Trygve Reenskaug. Models — views — controllers. Technical report, Xerox PARC, December 1979. [35] Alan Snyder. The essence of objects: Concepts and terms. IEEE Softw., 10(1):31–42, 1993. [36] Lynn Andrea Stein, Henry Lieberman, and David Ungar. A shared view of sharing: The treaty of orlando. In Won Kim and Frederick H. Lochovsky, editors, Object-Oriented Concepts, Databases and Applications, pages 31–48. ACM Press and Addison-Wesley, 1989. [37] D. Ungar and R.B. Smith. Self: The power of simplicity. In Proc. of ACM Conf. on Object-Oriented Programming, Systems, Languages and Applications, pages 227–242, 1987. [38] Murali Venkatrao and Michael Pizzo. SQL/CLI – a new binding style for SQL. SIGMOD Record, 24(4):72–77, 1995. [39] Steven R. Wood. Z — the 95% program editor. In Proceedings of the ACM SIGPLAN SIGOA symposium on Text manipulation, pages 1–7, New York, NY, USA, 1981. ACM Press. 1-21
- 30. The Evolution of Lua Roberto Ierusalimschy Department of Computer Science, PUC-Rio, Rio de Janeiro, Brazil roberto@inf.puc-rio.br Luiz Henrique de Figueiredo IMPA–Instituto Nacional de Matemática Pura e Aplicada, Brazil lhf@impa.br Waldemar Celes Department of Computer Science, PUC-Rio, Rio de Janeiro, Brazil celes@inf.puc-rio.br Abstract We report on the birth and evolution of Lua and discuss how it moved from a simple configuration language to a versatile, widely used language that supports extensible semantics, anonymous functions, full lexical scoping, proper tail calls, and coroutines. Categories and Subject Descriptors K.2 [HISTORY OF COMPUTING]: Software; D.3 [PROGRAMMING LAN- GUAGES] 1. Introduction Lua is a scripting language born in 1993 at PUC-Rio, the Pontifical Catholic University of Rio de Janeiro in Brazil. Since then, Lua has evolved to become widely used in all kinds of industrial applications, such as robotics, literate programming, distributed business, image processing, exten- sible text editors, Ethernet switches, bioinformatics, finite- element packages, web development, and more [2]. In par- ticular, Lua is one of the leading scripting languages in game development. Lua has gone far beyond our most optimistic expecta- tions. Indeed, while almost all programming languages come from North America and Western Europe (with the notable exception of Ruby, from Japan) [4], Lua is the only language created in a developing country to have achieved global rel- evance. From the start, Lua was designed to be simple, small, portable, fast, and easily embedded into applications. These design principles are still in force, and we believe that they account for Lua’s success in industry. The main characteris- tic of Lua, and a vivid expression of its simplicity, is that it offers a single kind of data structure, the table, which is the Lua term for an associative array [9]. Although most script- ing languages offer associative arrays, in no other language do associative arrays play such a central role. Lua tables provide simple and efficient implementations for modules, prototype-based objects, class-based objects, records, arrays, sets, bags, lists, and many other data structures [28]. In this paper, we report on the birth and evolution of Lua. We discuss how Lua moved from a simple configuration language to a powerful (but still simple) language that sup- ports extensible semantics, anonymous functions, full lexical scoping, proper tail calls, and coroutines. In §2 we give an overview of the main concepts in Lua, which we use in the other sections to discuss how Lua has evolved. In §3 we re- late the prehistory of Lua, that is, the setting that led to its creation. In §4 we relate how Lua was born, what its original design goals were, and what features its first version had. A discussion of how and why Lua has evolved is given in §5. A detailed discussion of the evolution of selected features is given in §6. The paper ends in §7 with a retrospective of the evolution of Lua and in §8 with a brief discussion of the reasons for Lua’s success, especially in games. 2. Overview In this section we give a brief overview of the Lua language and introduce the concepts discussed in §5 and §6. For a complete definition of Lua, see its reference manual [32]. For a detailed introduction to Lua, see Roberto’s book [28]. For concreteness, we shall describe Lua 5.1, which is the current version at the time of this writing (April 2007), but most of this section applies unchanged to previous versions. Syntactically, Lua is reminiscent of Modula and uses familiar keywords. To give a taste of Lua’s syntax, the code below shows two implementations of the factorial function, one recursive and another iterative. Anyone with a basic knowledge of programming can probably understand these examples without explanation. function factorial(n) function factorial(n) if n == 0 then local a = 1 return 1 for i = 1,n do else a = a*i return n*factorial(n-1) end end return a end end Permission to make digital/hard copy of part of this work for personal or classroom use is granted without fee provided that the copies are not made or distributed for profit or commercial advantage, the copyright notice, the title of the publication, and its date of appear, and notice is given that copying is by permission of the ACM, Inc. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Permission may be requested from the Publications Dept., ACM, Inc., 2 Penn Plaza, New York, NY 11201-0701, USA, fax:+1(212) 869-0481, permissions@acm.org ©2007 ACM 978-1-59593-766-7/2007/06-ART2 $5.00 DOI 10.1145/1238844.1238846 http://doi.acm.org/10.1145/1238844.1238846 2-1
- 31. Semantically, Lua has many similarities with Scheme, even though these similarities are not immediately clear be- cause the two languages are syntactically very different. The influence of Scheme on Lua has gradually increased during Lua’s evolution: initially, Scheme was just a language in the background, but later it became increasingly important as a source of inspiration, especially with the introduction of anonymous functions and full lexical scoping. Like Scheme, Lua is dynamically typed: variables do not have types; only values have types. As in Scheme, a variable in Lua never contains a structured value, only a reference to one. As in Scheme, a function name has no special status in Lua: it is just a regular variable that happens to refer to a function value. Actually, the syntax for function definition ‘function foo() · · · end’ used above is just syntactic sugar for the assignment of an anonymous function to a variable: ‘foo = function () · · · end’. Like Scheme, Lua has first-class functions with lexical scoping. Actually, all values in Lua are first-class values: they can be assigned to global and local variables, stored in tables, passed as arguments to functions, and returned from functions. One important semantic difference between Lua and Scheme — and probably the main distinguishing feature of Lua — is that Lua offers tables as its sole data-structuring mechanism. Lua tables are associative arrays [9], but with some important features. Like all values in Lua, tables are first-class values: they are not bound to specific variable names, as they are in Awk and Perl. A table can have any value as key and can store any value. Tables allow sim- ple and efficient implementation of records (by using field names as keys), sets (by using set elements as keys), generic linked structures, and many other data structures. Moreover, we can use a table to implement an array by using natural numbers as indices. A careful implementation [31] ensures that such a table uses the same amount of memory that an array would (because it is represented internally as an actual array) and performs better than arrays in similar languages, as independent benchmarks show [1]. Lua offers an expressive syntax for creating tables in the form of constructors. The simplest constructor is the expression ‘{}’, which creates a new, empty table. There are also constructors to create lists (or arrays), such as {"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"} and to create records, such as {lat= -22.90, long= -43.23, city= "Rio de Janeiro"} These two forms can be freely mixed. Tables are indexed using square brackets, as in ‘t[2]’, with ‘t.x’ as sugar for ‘t["x"]’. The combination of table constructors and functions turns Lua into a powerful general-purpose procedural data- description language. For instance, a bibliographic database in a format similar to the one used in BibTEX [34] can be written as a series of table constructors such as this: article{"spe96", authors = {"Roberto Ierusalimschy", "Luiz Henrique de Figueiredo", "Waldemar Celes"}, title = "Lua: an Extensible Extension Language", journal = "Software: Practice & Experience", year = 1996, } Although such a database seems to be an inert data file, it is actually a valid Lua program: when the database is loaded into Lua, each item in it invokes a function, because ‘article{· · ·}’ is syntactic sugar for ‘article({· · ·})’, that is, a function call with a table as its single argument. It is in this sense that such files are called procedural data files. We say that Lua is an extensible extension language [30]. It is an extension language because it helps to extend ap- plications through configuration, macros, and other end-user customizations. Lua is designed to be embedded into a host application so that users can control how the application be- haves by writing Lua programs that access application ser- vices and manipulate application data. It is extensible be- cause it offers userdata values to hold application data and extensible semantics mechanisms to manipulate these values in natural ways. Lua is provided as a small core that can be extended with user functions written in both Lua and C. In particular, input and output, string manipulation, mathema- tical functions, and interfaces to the operating system are all provided as external libraries. Other distinguishing features of Lua come from its im- plementation: Portability: Lua is easy to build because it is implemented in strict ANSI C.1 It compiles out-of-the-box on most platforms (Linux, Unix, Windows, Mac OS X, etc.), and runs with at most a few small adjustments in virtually all platforms we know of, including mobile devices (e.g., handheld computers and cell phones) and embedded mi- croprocessors (e.g., ARM and Rabbit). To ensure porta- bility, we strive for warning-free compilations under as many compilers as possible. Ease of embedding: Lua has been designed to be easily embedded into applications. An important part of Lua is a well-defined application programming interface (API) that allows full communication between Lua code and external code. In particular, it is easy to extend Lua by exporting C functions from the host application. The API allows Lua to interface not only with C and C++, but also with other languages, such as Fortran, Java, Smalltalk, Ada, C# (.Net), and even with other scripting languages (e.g., Perl and Ruby). 1 Actually, Lua is implemented in “clean C”, that is, the intersection of C and C++. Lua compiles unmodified as a C++ library. 2-2
- 32. Small size: Adding Lua to an application does not bloat it. The whole Lua distribution, including source code, doc- umentation, and binaries for some platforms, has always fit comfortably on a floppy disk. The tarball for Lua 5.1, which contains source code, documentation, and exam- ples, takes 208K compressed and 835K uncompressed. The source contains around 17,000 lines of C. Under Linux, the Lua interpreter built with all standard Lua li- braries takes 143K. The corresponding numbers for most other scripting languages are more than an order of mag- nitude larger, partially because Lua is primarily meant to be embedded into applications and so its official distri- bution includes only a few libraries. Other scripting lan- guages are meant to be used standalone and include many libraries. Efficiency: Independent benchmarks [1] show Lua to be one of the fastest languages in the realm of interpreted scripting languages. This allows application developers to write a substantial fraction of the whole application in Lua. For instance, over 40% of Adobe Lightroom is written in Lua (that represents around 100,000 lines of Lua code). Although these are features of a specific implementation, they are possible only due to the design of Lua. In particular, Lua’s simplicity is a key factor in allowing a small, efficient implementation [31]. 3. Prehistory Lua was born in 1993 inside Tecgraf, the Computer Graph- ics Technology Group of PUC-Rio in Brazil. The cre- ators of Lua were Roberto Ierusalimschy, Luiz Henrique de Figueiredo, and Waldemar Celes. Roberto was an assis- tant professor at the Department of Computer Science of PUC-Rio. Luiz Henrique was a post-doctoral fellow, first at IMPA and later at Tecgraf. Waldemar was a Ph.D. student in Computer Science at PUC-Rio. All three were members of Tecgraf, working on different projects there before getting together to work on Lua. They had different, but related, backgrounds: Roberto was a computer scientist interested mainly in programming languages; Luiz Henrique was a mathematician interested in software tools and computer graphics; Waldemar was an engineer interested in appli- cations of computer graphics. (In 2001, Waldemar joined Roberto as faculty at PUC-Rio and Luiz Henrique became a researcher at IMPA.) Tecgraf is a large research and development laboratory with several industrial partners. During the first ten years after its creation in May 1987, Tecgraf focused mainly on building basic software tools to enable it to produce the inter- active graphical programs needed by its clients. Accordingly, the first Tecgraf products were drivers for graphical termi- nals, plotters, and printers; graphical libraries; and graphical interface toolkits. From 1977 until 1992, Brazil had a pol- icy of strong trade barriers (called a “market reserve”) for computer hardware and software motivated by a national- istic feeling that Brazil could and should produce its own hardware and software. In that atmosphere, Tecgraf’s clients could not afford, either politically or financially, to buy cus- tomized software from abroad: by the market reserve rules, they would have to go through a complicated bureaucratic process to prove that their needs could not be met by Brazil- ian companies. Added to the natural geographical isolation of Brazil from other research and development centers, those reasons led Tecgraf to implement from scratch the basic tools it needed. One of Tecgraf’s largest partners was (and still is) Petro- bras, the Brazilian oil company. Several Tecgraf products were interactive graphical programs for engineering appli- cations at Petrobras. By 1993, Tecgraf had developed little languages for two of those applications: a data-entry appli- cation and a configurable report generator for lithology pro- files. These languages, called DEL and SOL, were the an- cestors of Lua. We describe them briefly here to show where Lua came from. 3.1 DEL The engineers at Petrobras needed to prepare input data files for numerical simulators several times a day. This process was boring and error-prone because the simulation programs were legacy code that needed strictly formatted input files — typically bare columns of numbers, with no indication of what each number meant, a format inherited from the days of punched cards. In early 1992, Petrobras asked Tecgraf to create at least a dozen graphical front-ends for this kind of data entry. The numbers would be input interactively, just by clicking on the relevant parts of a diagram describing the simulation — a much easier and more meaningful task for the engineers than editing columns of numbers. The data file, in the correct format for the simulator, would be generated automatically. Besides simplifying the creation of data files, such front-ends provided the opportunity to add data validation and also to compute derived quantities from the input data, thus reducing the amount of data needed from the user and increasing the reliability of the whole process. To simplify the development of those front-ends, a team led by Luiz Henrique de Figueiredo and Luiz Cristovão Gomes Coelho decided to code all front-ends in a uni- form way, and so designed DEL (“data-entry language”), a simple declarative language to describe each data-entry task [17]. DEL was what is now called a domain-specific lan- guage [43], but was then simply called a little language [10]. A typical DEL program defined several “entities”. Each entity could have several fields, which were named and typed. For implementing data validation, DEL had predi- cate statements that imposed restrictions on the values of entities. DEL also included statements to specify how data was to be input and output. An entity in DEL was essen- tially what is called a structure or record in conventional 2-3
- 33. programming languages. The important difference — and what made DEL suitable for the data-entry problem — is that entity names also appeared in a separate graphics metafile, which contained the associated diagram over which the en- gineer did the data entry. A single interactive graphical inter- preter called ED (an acronym for ‘entrada de dados’, which means ‘data entry’ in Portuguese) was written to interpret DEL programs. All those data-entry front-ends requested by Petrobras were implemented as DEL programs that ran under this single graphical application. DEL was a success both among the developers at Tec- graf and among the users at Petrobras. At Tecgraf, DEL simplified the development of those front-ends, as originally intended. At Petrobras, DEL allowed users to tailor data- entry applications to their needs. Soon users began to de- mand more power from DEL, such as boolean expressions for controlling whether an entity was active for input or not, and DEL became heavier. When users began to ask for con- trol flow, with conditionals and loops, it was clear that ED needed a real programming language instead of DEL. 3.2 SOL At about the same time that DEL was created, a team lead by Roberto Ierusalimschy and Waldemar Celes started working on PGM, a configurable report generator for lithology pro- files, also for Petrobras. The reports generated by PGM con- sisted of several columns (called “tracks”) and were highly configurable: users could create and position the tracks, and could choose colors, fonts, and labels; each track could have a grid, which also had its set of options (log/linear, verti- cal and horizontal ticks, etc.); each curve had its own scale, which had to be changed automatically in case of overflow; etc. All this configuration was to be done by the end-users, typically geologists and engineers from Petrobras working in oil plants and off-shore platforms. The configurations had to be stored in files, for reuse. The team decided that the best way to configure PGM was through a specialized description language called SOL, an acronym for Simple Object Lan- guage. Because PGM had to deal with many different objects, each with many different attributes, the SOL team decided not to fix those objects and attributes into the language. In- stead, SOL allowed type declarations, as in the code below: type @track{ x:number, y:number=23, id=0 } type @line{ t:@track=@track{x=8}, z:number* } T = @track{ y=9, x=10, id="1992-34" } L = @line{ t=@track{x=T.y, y=T.x}, z=[2,3,4] } This code defines two types, track and line, and creates two objects, a track T and a line L. The track type contains two numeric attributes, x and y, and an untyped attribute, id; attributes y and id have default values. The line type con- tains a track t and a list of numbers z. The track t has as default value a track with x=8, y=23, and id=0. The syntax of SOL was strongly influenced by BibTEX [34] and UIL, a language for describing user interfaces in Motif [39]. The main task of the SOL interpreter was to read a report description, check whether the given objects and attributes were correctly typed, and then present the information to the main program (PGM). To allow the communication between the main program and the SOL interpreter, the latter was implemented as a C library that was linked to the main program. The main program could access all configuration information through an API in this library. In particular, the main program could register a callback function for each type, which the SOL interpreter would call to create an object of that type. 4. Birth The SOL team finished an initial implementation of SOL in March 1993, but they never delivered it. PGM would soon require support for procedural programming to allow the creation of more sophisticated layouts, and SOL would have to be extended. At the same time, as mentioned before, ED users had requested more power from DEL. ED also needed further descriptive facilities for programming its user interface. Around mid-1993, Roberto, Luiz Henrique, and Waldemar got together to discuss DEL and SOL, and con- cluded that the two languages could be replaced by a single, more powerful language, which they decided to design and implement. Thus the Lua team was born; it has not changed since. Given the requirements of ED and PGM, we decided that we needed a real programming language, with assignments, control structures, subroutines, etc. The language should also offer data-description facilities, such as those offered by SOL. Moreover, because many potential users of the language were not professional programmers, the language should avoid cryptic syntax and semantics. The implemen- tation of the new language should be highly portable, be- cause Tecgraf’s clients had a very diverse collection of com- puter platforms. Finally, since we expected that other Tec- graf products would also need to embed a scripting lan- guage, the new language should follow the example of SOL and be provided as a library with a C API. At that point, we could have adopted an existing scripting language instead of creating a new one. In 1993, the only real contender was Tcl [40], which had been explicitly designed to be embedded into applications. However, Tcl had unfa- miliar syntax, did not offer good support for data description, and ran only on Unix platforms. We did not consider LISP or Scheme because of their unfriendly syntax. Python was still in its infancy. In the free, do-it-yourself atmosphere that then reigned in Tecgraf, it was quite natural that we should try to develop our own scripting language. So, we started working on a new language that we hoped would be simpler to use than existing languages. Our original design decisions were: keep the language simple and small, and keep the im- 2-4
- 34. plementation simple and portable. Because the new language was partially inspired by SOL (sun in Portuguese), a friend at Tecgraf (Carlos Henrique Levy) suggested the name ‘Lua’ (moon in Portuguese), and Lua was born. (DEL did not in- fluence Lua as a language. The main influence of DEL on the birth of Lua was rather the realization that large parts of complex applications could be written using embeddable scripting languages.) We wanted a light full language with data-description fa- cilities. So we took SOL’s syntax for record and list con- struction (but not type declaration), and unified their imple- mentation using tables: records use strings (the field names) as indices; lists use natural numbers. An assignment such as T = @track{ y=9, x=10, id="1992-34" } which was valid in SOL, remained valid in Lua, but with a different meaning: it created an object (that is, a table) with the given fields, and then called the function track on this table to validate the object or perhaps to provide default values to some of its fields. The final value of the expression was that table. Except for its procedural data-description constructs, Lua introduced no new concepts: Lua was created for production use, not as an academic language designed to support re- search in programming languages. So, we simply borrowed (even unconsciously) things that we had seen or read about in other languages. We did not reread old papers to remem- ber details of existing languages. We just started from what we knew about other languages and reshaped that according to our tastes and needs. We quickly settled on a small set of control structures, with syntax mostly borrowed from Modula (while, if, and repeat until). From CLU we took multiple assignment and multiple returns from function calls. We regarded mul- tiple returns as a simpler alternative to reference parameters used in Pascal and Modula and to in-out parameters used in Ada; we also wanted to avoid explicit pointers (used in C). From C++ we took the neat idea of allowing a local vari- able to be declared only where we need it. From SNOBOL and Awk we took associative arrays, which we called tables; however, tables were to be objects in Lua, not attached to variables as in Awk. One of the few (and rather minor) innovations in Lua was the syntax for string concatenation. The natural ‘+’ operator would be ambiguous, because we wanted automatic coer- cion of strings to numbers in arithmetic operations. So, we invented the syntax ‘..’ (two dots) for string concatenation. A polemic point was the use of semicolons. We thought that requiring semicolons could be a little confusing for en- gineers with a Fortran background, but not allowing them could confuse those with a C or Pascal background. In typi- cal committee fashion, we settled on optional semicolons. Initially, Lua had seven types: numbers (implemented solely as reals), strings, tables, nil, userdata (pointers to C objects), Lua functions, and C functions. To keep the lan- guage small, we did not initially include a boolean type: as in Lisp, nil represented false and any other value repre- sented true. Over 13 years of continuous evolution, the only changes in Lua types were the unification of Lua functions and C functions into a single function type in Lua 3.0 (1997) and the introduction of booleans and threads in Lua 5.0 (2003) (see §6.1). For simplicity, we chose to use dynamic typing instead of static typing. For applications that needed type checking, we provided basic reflective facilities, such as run-time type information and traversal of the global en- vironment, as built-in functions (see §6.11). By July 1993, Waldemar had finished the first implemen- tation of Lua as a course project supervised by Roberto. The implementation followed a tenet that is now central to Extreme Programming: “the simplest thing that could pos- sibly work” [7]. The lexical scanner was written with lex and the parser with yacc, the classic Unix tools for imple- menting languages. The parser translated Lua programs into instructions for a stack-based virtual machine, which were then executed by a simple interpreter. The C API made it easy to add new functions to Lua, and so this first version provided only a tiny library of five built-in functions (next, nextvar, print, tonumber, type) and three small exter- nal libraries (input and output, mathematical functions, and string manipulation). Despite this simple implementation — or possibly be- cause of it — Lua surpassed our expectations. Both PGM and ED used Lua successfully (PGM is still in use today; ED was replaced by EDG [12], which was mostly written in Lua). Lua was an immediate success in Tecgraf and soon other projects started using it. This initial use of Lua at Tec- graf was reported in a brief talk at the VII Brazilian Sympo- sium on Software Engineering, in October 1993 [29]. The remainder of this paper relates our journey in im- proving Lua. 5. History Figure 1 shows a timeline of the releases of Lua. As can be seen, the time interval between versions has been gradually increasing since Lua 3.0. This reflects our perception that 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 1.0 1.1 2.1 2.2 2.4 2.5 3.0 3.1 3.2 4.0 5.0 5.1 Figure 1. The releases of Lua. 2-5
- 35. 1.0 1.1 2.1 2.2 2.4 2.5 3.0 3.1 3.2 4.0 5.0 5.1 constructors • • • • • • • • • • • • garbage collection • • • • • • • • • • • • extensible semantics ◦ ◦ • • • • • • • • • • support for OOP ◦ ◦ • • • • • • • • • • long strings ◦ ◦ ◦ • • • • • • • • • debug API ◦ ◦ ◦ • • • • • • • • • external compiler ◦ ◦ ◦ ◦ • • • • • • • • vararg functions ◦ ◦ ◦ ◦ ◦ • • • • • • • pattern matching ◦ ◦ ◦ ◦ ◦ • • • • • • • conditional compilation ◦ ◦ ◦ ◦ ◦ ◦ • • • ◦ ◦ ◦ anonymous functions, closures ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • • • • debug library ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • • • multi-state API ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • • for statement ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • • long comments ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • full lexical scoping ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • booleans ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • coroutines ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • • incremental garbage collection ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • module system ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ • 1.0 1.1 2.1 2.2 2.4 2.5 3.0 3.1 3.2 4.0 5.0 5.1 libraries 4 4 4 4 4 4 4 4 5 6 8 9 built-in functions 5 7 11 11 13 14 25 27 35 0 0 0 API functions 30 30 30 30 32 32 33 47 41 60 76 79 vm type (stack × register) S S S S S S S S S S R R vm instructions 64 65 69 67 67 68 69 128 64 49 35 38 keywords 16 16 16 16 16 16 16 16 16 18 21 21 other tokens 21 21 23 23 23 23 24 25 25 25 24 26 Table 1. The evolution of features in Lua. Lua was becoming a mature product and needed stability for the benefit of its growing community. Nevertheless, the need for stability has not hindered progress. Major new versions of Lua, such as Lua 4.0 and Lua 5.0, have been released since then. The long times between versions also reflects our release model. Unlike other open-source projects, our alpha versions are quite stable and beta versions are essentially final, except for uncovered bugs.2 This release model has proved to be good for Lua stability. Several products have been shipped with alpha or beta versions of Lua and worked fine. How- ever, this release model did not give users much chance to experiment with new versions; it also deprived us of timely feedback on proposed changes. So, during the development of Lua 5.0 we started to release “work” versions, which are just snapshots of the current development of Lua. This move brought our current release model closer to the “Release Early, Release Often” motto of the open-source community. 2 The number of bugs found after final versions were released has been consistently small: only 10 in Lua 4.0, 17 in Lua 5.0, and 10 in Lua 5.1 so far, none of them critical bugs. In the remainder of this section we discuss some mile- stones in the evolution of Lua. Details on the evolution of several specific features are given in §6. Table 1 summarizes this evolution. It also contains statistics about the size of Lua, which we now discuss briefly. The number of standard libraries has been kept small be- cause we expect that most Lua functions will be provided by the host application or by third-party libraries. Until Lua 3.1, the only standard libraries were for input and output, string manipulation, mathematical functions, and a special library of built-in functions, which did not use the C API but directly accessed the internal data structures. Since then, we have added libraries for debugging (Lua 3.2), interfacing with the operating system (Lua 4.0), tables and coroutines (Lua 5.0), and modules (Lua 5.1). The size of C API changed significantly when it was re- designed in Lua 4.0. Since then, it has moved slowly toward completeness. As a consequence, there are no longer any built-in functions: all standard libraries are implemented on top the C API, without accessing the internals of Lua. The virtual machine, which executes Lua programs, was stack-based until Lua 4.0. In Lua 3.1 we added variants 2-6
- 36. for many instructions, to try to improve performance. How- ever, this turned out to be too complicated for little per- formance gain and we removed those variants in Lua 3.2. Since Lua 5.0, the virtual machine is register-based [31]. This change gave the code generator more opportunities for optimization and reduced the number of instructions of typi- cal Lua programs. (Instruction dispatch is a significant frac- tion of the time spent in the virtual machine [13].) As far as we know, the virtual machine of Lua 5.0 was the first register-based virtual machine to have wide use. 5.1 Lua 1 The initial implementation of Lua was a success in Tec- graf and Lua attracted users from other Tecgraf projects. New users create new demands. Several users wanted to use Lua as the support language for graphics metafiles, which abounded in Tecgraf. Compared with other programmable metafiles, Lua metafiles have the advantage of being based on a truly procedural language: it is natural to model com- plex objects by combining procedural code fragments with declarative statements. In contrast, for instance, VRML [8] must use another language (Javascript) to model procedural objects. The use of Lua for this kind of data description, especially large graphics metafiles, posed challenges that were unusual for typical scripting languages. For instance, it was not un- common for a diagram used in the data-entry program ED to have several thousand parts described by a single Lua ta- ble constructor with several thousand items. That meant that Lua had to cope with huge programs and huge expressions. Because Lua precompiled all programs to bytecode for a vir- tual machine on the fly, it also meant that the Lua compiler had to run fast, even for large programs. By replacing the lex-generated scanner used in the first version by a hand-written one, we almost doubled the speed of the Lua compiler on typical metafiles. We also modified Lua’s virtual machine to handle a long constructor by adding key-value pairs to the table in batches, not individually as in the original virtual machine. These changes solved the initial demands for better performance. Since then, we have always tried to reduce the time spent on precompilation. In July 1994, we released a new version of Lua with those optimizations. This release coincided with the publication of the first paper describing Lua, its design, and its implementa- tion [15]. We named the new version ‘Lua 1.1’. The previous version, which was never publicly released, was then named ‘Lua 1.0’. (A snapshot of Lua 1.0 taken in July 1993 was released in October 2003 to celebrate 10 years of Lua.) Lua 1.1 was publicly released as software available in source code by ftp, before the open-source movement got its current momentum. Lua 1.1 had a restrictive user license: it was freely available for academic purposes but commercial uses had to be negotiated. That part of the license did not work: although we had a few initial contacts, no commer- cial uses were ever negotiated. This and the fact that other scripting languages (e.g, Tcl) were free made us realize that restrictions on commercial uses might even discourage aca- demic uses, since some academic projects plan to go to mar- ket eventually. So, when the time came to release the next version (Lua 2.1), we chose to release it as unrestricted free software. Naively, we wrote our own license text as a slight collage and rewording of existing licenses. We thought it was clear that the new license was quite liberal. Later, how- ever, with the spread of open-source licenses, our license text became a source of noise among some users; in particular, it was not clear whether our license was compatible with GPL. In May 2002, after a long discussion in the mailing list, we decided to release future versions of Lua (starting with Lua 5.0) under the well-known and very liberal MIT license [3]. In July 2002, the Free Software Foundation con- firmed that our previous license was compatible with GPL, but we were already committed to adopting the MIT license. Questions about our license have all but vanished since then. 5.2 Lua 2 Despite all the hype surrounding object-oriented program- ming (which in the early 1990s had reached its peak) and the consequent user pressure to add object-oriented features to Lua, we did not want to turn Lua into an object-oriented language because we did not want to fix a programming paradigm for Lua. In particular, we did not think that Lua needed objects and classes as primitive language concepts, especially because they could be implemented with tables if needed (a table can hold both object data and methods, since functions are first-class values). Despite recurring user pres- sure, we have not changed our minds to this day: Lua does not force any object or class model onto the programmer. Several object models have been proposed and implemented by users; it is a frequent topic of discussion in our mailing list. We think this is healthy. On the other hand, we wanted to allow object-oriented programming with Lua. Instead of fixing a model, we de- cided to provide flexible mechanisms that would allow the programmer to build whatever model was suitable to the ap- plication. Lua 2.1, released in February 1995, marked the in- troduction of these extensible semantics mechanisms, which have greatly increased the expressiveness of Lua. Extensible semantics has become a hallmark of Lua. One of the goals of extensible semantics was to allow ta- bles to be used as a basis for objects and classes. For that, we needed to implement inheritance for tables. Another goal was to turn userdata into natural proxies for application data, not merely handles meant to be used solely as arguments to functions. We wanted to be able to index userdata as if they were tables and to call methods on them. This would allow Lua to fulfill one of its main design goals more naturally: to extend applications by providing scriptable access to ap- plication services and data. Instead of adding mechanisms to support all these features directly in the language, we de- cided that it would be conceptually simpler to define a more 2-7
- 37. general fallback mechanism to let the programmer intervene whenever Lua did not know how to proceed. We introduced fallbacks in Lua 2.1 and defined them for the following operations: table indexing, arithmetic oper- ations, string concatenation, order comparisons, and func- tion calls.3 When one of these operations was applied to the “wrong” kind of values, the corresponding fallback was called, allowing the programmer to determine how Lua would proceed. The table indexing fallbacks allowed userdata (and other values) to behave as tables, which was one of our motivations. We also defined a fallback to be called when a key was absent from a table, so that we could support many forms of inheritance (through dele- gation). To complete the support for object-oriented pro- gramming, we added two pieces of syntactic sugar: method definitions of the form ‘function a:foo(· · ·)’ as sugar for ‘function a.foo(self,· · ·)’ and method calls of the form ‘a:foo(· · ·)’ as sugar for ‘a.foo(a,· · ·)’. In §6.8 we discuss fallbacks in detail and how they evolved into their later incarnations: tag methods and metamethods. Since Lua 1.0, we have provided introspective functions for values: type, which queries the type of a Lua value; next, which traverses a table; and nextvar, which traverses the global environment. (As mentioned in §4, this was par- tially motivated by the need to implement SOL-like type checking.) In response to user pressure for full debug fa- cilities, Lua 2.2 (November 1995) introduced a debug API to provide information about running functions. This API gave users the means to write in C their own introspective tools, such as debuggers and profilers. The debug API was initially quite simple: it allowed access to the Lua call stack, to the currently executing line, and provided a function to find the name of a variable holding a given value. Following the M.Sc. work of Tomás Gorham [22], the debug API was improved in Lua 2.4 (May 1996) by functions to access local variables and hooks to be called at line changes and function calls. With the widespread use of Lua at Tecgraf, many large graphics metafiles were being written in Lua as the output of graphical editors. Loading such metafiles was taking in- creasingly longer as they became larger and more complex.4 Since its first version, Lua precompiled all programs to byte- code just before running them. The load time of a large pro- gram could be substantially reduced by saving this bytecode to a file. This would be especially relevant for procedural data files such as graphics metafiles. So, in Lua 2.4, we in- troduced an external compiler, called luac, which precom- piled a Lua program and saved the generated bytecode to a binary file. (Our first paper about Lua [15] had already an- 3 We also introduced fallbacks for handling fatal errors and for monitoring garbage collection, even though they were not part of extensible semantics. 4 Surprisingly, a substantial fraction of the load time was taken in the lexer for converting real numbers from text form to floating-point representation. Real numbers abound in graphics metafiles. ticipated the possibility of an external compiler.) The format of this file was chosen to be easily loaded and reasonably portable. With luac, programmers could avoid parsing and code generation at run time, which in the early days were costly. Besides faster loading, luac also allowed off-line syntax checking and protection from casual user changes. Many products (e.g., The Sims and Adobe Lightroom) dis- tribute Lua scripts in precompiled form. During the implementation of luac, we started to restruc- ture Lua’s core into clearly separated modules. As a conse- quence, it is now quite easy to remove the parsing modules (lexer, parser, and code generator), which currently repre- sent 35% of the core code, leaving just the module that loads precompiled Lua programs, which is merely 3% of the core code. This reduction can be significant when embedding Lua in small devices such as mobile devices, robots and sensors.5 Since its first version, Lua has included a library for string-processing. The facilities provided by this library were minimal until Lua 2.4. However, as Lua matured, it became desirable to do heavier text processing in Lua. We thought that a natural addition to Lua would be pattern matching, in the tradition of Snobol, Icon, Awk, and Perl. However, we did not want to include a third-party pattern- matching engine in Lua because such engines tend to be very large; we also wanted to avoid copyright issues that could be raised by including third-party code in Lua. As a student project supervised by Roberto in the second semester of 1995, Milton Jonathan, Pedro Miller Rabinovitch, Pedro Willemsens, and Vinicius Almendra pro- duced a pattern-matching library for Lua. Experience with that design led us to write our own pattern-matching en- gine for Lua, which we added to Lua 2.5 (November 1996) in two functions: strfind (which originally only found plain substrings) and the new gsub function (a name taken from Awk). The gsub function globally replaced substrings matching a given pattern in a larger string. It accepted either a replacement string or a function that was called each time a match was found and was intended to return the replace- ment string for that match. (That was an innovation at the time.) Aiming at a small implementation, we did not include full regular expressions. Instead, the patterns understood by our engine were based on character classes, repetitions, and captures (but not alternation or grouping). Despite its sim- plicity, this kind of pattern matching is quite powerful and was an important addition to Lua. That year was a turning point in the history of Lua be- cause it gained international exposure. In June 1996 we pub- lished a paper about Lua in Software: Practice & Experi- ence [30] that brought external attention to Lua, at least in 5 Crazy Ivan, a robot that won RoboCup in 2000 and 2001 in Denmark, had a “brain” implemented in Lua. It ran directly on a Motorola Coldfire 5206e processor without any operating system (in other words, Lua was the operating system). Lua was stored on a system ROM and loaded programs at startup from the serial port. 2-8
- 38. academic circles.6 In December 1996, shortly after Lua 2.5 was released, the magazine Dr. Dobb’s Journal featured an article about Lua [16]. Dr. Dobb’s Journal is a popular publication aimed directly at programmers, and that article brought Lua to the attention of the software industry. Among several messages that we received right after that publication was one sent in January 1997 by Bret Mogilefsky, who was the lead programmer of Grim Fandango, an adventure game then under development by LucasArts. Bret told us that he had read about Lua in Dr. Dobb’s and that they planned to re- place their home-brewed scripting language with Lua. Grim Fandango was released in October 1998 and in May 1999 Bret told us that “a tremendous amount of the game was written in Lua” (his emphasis) [38].7 Around that time, Bret attended a roundtable about game scripting at the Game De- velopers’ Conference (GDC, the main event for game pro- grammers) and at the end he related his experience with the successful use of Lua in Grim Fandango. We know of several developers who first learned about Lua at that event. After that, Lua spread by word of mouth among game developers to become a definitely marketable skill in the game industry (see §8). As a consequence of Lua’s international exposure, the number of messages sent to us asking questions about Lua increased substantially. To handle this traffic more effi- ciently, and also to start building a Lua community, so that other people could answer Lua questions, in February 1997 we created a mailing list for discussing Lua. Over 38,000 messages have been posted to this list since then. The use of Lua in many popular games has attracted many people to the list, which now has over 1200 subscribers. We have been fortunate that the Lua list is very friendly and at the same time very technical. The list has become the focal point of the Lua community and has been a source of motivation for improving Lua. All important events occur first in the mailing list: release announcements, feature requests, bug reports, etc. The creation of a comp.lang.lua Usenet newsgroup was discussed twice in the list over all these years, in April 1998 and in July 1999. The conclusion both times was that the traffic in the list did not warrant the creation of a newsgroup. Moreover, most people preferred a mailing list. The creation of a newsgroup seems no longer relevant because there are several web interfaces for reading and searching the complete list archives. 6 In November 1997, that article won the First Prize (technological cate- gory) in the II Compaq Award for Research and Development in Computer Science, a joint venture of Compaq Computer in Brazil, the Brazilian Min- istry of Science and Technology, and the Brazilian Academy of Sciences. 7 Grim Fandango mentioned Lua and PUC-Rio in its final credits. Several people at PUC-Rio first learned about Lua from that credit screen, and were surprised to learn that Brazilian software was part of a hit game. It has always bothered us that Lua is widely known abroad but has remained relatively unknown in Brazil until quite recently. 5.3 Lua 3 The fallback mechanism introduced in Lua 2.1 to support extensible semantics worked quite well but it was a global mechanism: there was only one hook for each event. This made it difficult to share or reuse code because modules that defined fallbacks for the same event could not co-exist eas- ily. Following a suggestion by Stephan Herrmann in Decem- ber 1996, in Lua 3.0 (July 1997) we solved the fallback clash problem by replacing fallbacks with tag methods: the hooks were attached to pairs (event, tag) instead of just to events. Tags had been introduced in Lua 2.1 as integer labels that could be attached to userdata (see §6.10); the intention was that C objects of the same type would be represented in Lua by userdata having the same tag. (However, Lua did not force any interpretation on tags.) In Lua 3.0 we extended tags to all values to support tag methods. The evolution of fallbacks is discussed in §6.8. Lua 3.1 (July 1998) brought functional programming to Lua by introducing anonymous functions and function clo- sures via “upvalues”. (Full lexical scoping had to wait until Lua 5.0; see §6.6.) The introduction of closures was mainly motivated by the existence of higher-order functions, such as gsub, which took functions as arguments. During the work on Lua 3.1, there were discussions in the mailing list about multithreading and cooperative multitasking, mainly moti- vated by the changes Bret Mogilefsky had made to Lua 2.5 and 3.1 alpha for Grim Fandango. No conclusions were reached, but the topic remained popular. Cooperative multi- tasking in Lua was finally provided in Lua 5.0 (April 2003); see §6.7. The C API remained largely unchanged from Lua 1.0 to Lua 3.2; it worked over an implicit Lua state. However, newer applications, such as web services, needed multiple states. To mitigate this problem, Lua 3.1 introduced multiple independent Lua states that could be switched at run time. A fully reentrant API would have to wait until Lua 4.0. In the meantime, two unofficial versions of Lua 3.2 with ex- plicit Lua states appeared: one written in 1998 by Roberto Ierusalimschy and Anna Hester based on Lua 3.2 alpha for CGILua [26], and one written in 1999 by Erik Hougaard based on Lua 3.2 final. Erik’s version was publicly avail- able and was used in the Crazy Ivan robot. The version for CGILua was released only as part of the CGILua distribu- tion; it never existed as an independent package. Lua 3.2 (July 1999) itself was mainly a maintenance re- lease; it brought no novelties except for a debug library that allowed tools to be written in Lua instead of C. Neverthe- less, Lua was quite stable by then and Lua 3.2 had a long life. Because the next version (Lua 4.0) introduced a new, incompatible API, many users just stayed with Lua 3.2 and never migrated to Lua 4.0. For instance, Tecgraf never mi- grated to Lua 4.0, opting to move directly to Lua 5.0; many products at Tecgraf still use Lua 3.2. 2-9
- 39. 5.4 Lua 4 Lua 4.0 was released in November 2000. As mentioned above, the main change in Lua 4.0 was a fully reentrant API, motivated by applications that needed multiple Lua states. Since making the API fully reentrant was already a major change, we took the opportunity and completely redesigned the API around a clear stack metaphor for exchanging val- ues with C (see §6.9). This was first suggested by Reuben Thomas in July 2000. Lua 4.0 also introduced a ‘for’ statement, then a top item in the wish-list of most Lua users and a frequent topic in the mailing list. We had not included a ‘for’ statement earlier because ‘while’ loops were more general. However, users complained that they kept forgetting to update the control variable at the end of ‘while’ loops, thus leading to infinite loops. Also, we could not agree on a good syntax. We considered the Modula ‘for’ too restrictive because it did not cover iterations over the elements of a table or over the lines of a file. A ‘for’ loop in the C tradition did not fit with the rest of Lua. With the introduction of closures and anonymous functions in Lua 3.1, we decided to use higher-order functions for implementing iterations. So, Lua 3.1 provided a higher-order function that iterated over a table by calling a user-supplied function over all pairs in the table. To print all pairs in a table t, one simply said ‘foreach(t,print)’. In Lua 4.0 we finally designed a ‘for’ loop, in two vari- ants: a numeric loop and a table-traversal loop (first sug- gested by Michael Spalinski in October 1997). These two variants covered most common loops; for a really generic loop, there was still the ‘while’ loop. Printing all pairs in a table t could then be done as follows:8 for k,v in t do print(k,v) end The addition of a ‘for’ statement was a simple one but it did change the look of Lua programs. In particular, Roberto had to rewrite many examples in his draft book on Lua programming. Roberto had been writing this book since 1998, but he could never finish it because Lua was a moving target. With the release of Lua 4.0, large parts of the book and almost all its code snippets had to be rewritten. Soon after the release of Lua 4.0, we started working on Lua 4.1. Probably the main issue we faced for Lua 4.1 was whether and how to support multithreading, a big is- sue at that time. With the growing popularity of Java and Pthreads, many programmers began to consider support for multithreading as an essential feature in any programming language. However, for us, supporting multithreading in Lua posed serious questions. First, to implement multithread- ing in C requires primitives that are not part of ANSI C — 8 With the introduction of ‘for’ iterators in Lua 5.0, this syntax was marked as obsolete and later removed in Lua 5.1. although Pthreads was popular, there were (and still there are) many platforms without this library. Second, and more important, we did not (and still do not) believe in the stan- dard multithreading model, which is preemptive concur- rency with shared memory: we still think that no one can write correct programs in a language where ‘a=a+1’ is not deterministic. For Lua 4.1, we tried to solve those difficulties in a typi- cal Lua fashion: we implemented only a basic mechanism of multiple stacks, which we called threads. External libraries could use those Lua threads to implement multithreading, based on a support library such as Pthreads. The same mech- anism could be used to implement coroutines, in the form of non-preemptive, collaborative multithreading. Lua 4.1 alpha was released in July 2001 with support for external multi- threading and coroutines; it also introduced support for weak tables and featured a register-based virtual machine, with which we wanted to experiment. The day after Lua 4.1 alpha was released, John D. Rams- dell started a big discussion in the mailing list about lexi- cal scoping. After several dozen messages, it became clear that Lua needed full lexical scoping, instead of the upvalue mechanism adopted since Lua 3.1. By October 2001 we had come up with an efficient implementation of full lexi- cal scoping, which we released as a work version in Novem- ber 2001. (See §6.6 for a detailed discussion of lexical scop- ing.) That version also introduced a new hybrid representa- tion for tables that let them be implemented as arrays when appropriate (see §6.2 for further details). Because that ver- sion implemented new basic algorithms, we decided to re- lease it as a work version, even though we had already re- leased an alpha version for Lua 4.1. In February 2002 we released a new work version for Lua 4.1, with three relevant novelties: a generic ‘for’ loop based on iterator functions, metatables and metamethods as a replacement for tags and fallbacks9 (see §6.8), and coroutines (see §6.7). After that release, we realized that Lua 4.1 would bring too many major changes — perhaps ‘Lua 5.0’ would be a better name for the next version. 5.5 Lua 5 The final blow to the name ‘Lua 4.1’ came a few days later, during the Lua Library Design Workshop organized by Christian Lindig and Norman Ramsey at Harvard. One of the main conclusions of the workshop was that Lua needed some kind of module system. Although we had always con- sidered that modules could be implemented using tables, not even the standard Lua libraries followed this path. We then decided to take that step for the next version. 9 The use of ordinary Lua tables for implementing extensible semantics had already been suggested by Stephan Herrmann in December 1996, but we forgot all about it until it was suggested again by Edgar Toernig in Octo- ber 2000, as part of a larger proposal, which he called ‘unified methods’. The term ‘metatable’ was suggested by Rici Lake in November 2001. 2-10
- 40. Packaging library functions inside tables had a big practi- cal impact, because it affected any program that used at least one library function. For instance, the old strfind function was now called string.find (field ‘find’ in string library stored in the ‘string’ table); openfile became io.open; sin became math.sin; and so on. To make the transition easier, we provided a compatibility script that defined the old functions in terms of the new ones: strfind = string.find openfile = io.open sin = math.sin ... Nevertheless, packaging libraries in tables was a major change. In June 2002, when we released the next work version incorporating this change, we dropped the name ‘Lua 4.1’ and named it ‘Lua 5.0 work0’. Progress to the final version was steady from then on and Lua 5.0 was re- leased in April 2003. This release froze Lua enough to allow Roberto to finish his book, which was published in Decem- ber 2003 [27]. Soon after the release of Lua 5.0 we started working on Lua 5.1. The initial motivation was the implementation of incremental garbage collection in response to requests from game developers. Lua uses a traditional mark-and- sweep garbage collector, and, until Lua 5.0, garbage col- lection was performed atomically. As a consequence, some applications might experience potentially long pauses dur- ing garbage collection.10 At that time, our main concern was that adding the write barriers needed to implement an incre- mental garbage collector would have a negative impact on Lua performance. To compensate for that we tried to make the collector generational as well. We also wanted to keep the adaptive behavior of the old collector, which adjusted the frequency of collection cycles according to the total memory in use. Moreover, we wanted to keep the collector simple, like the rest of Lua. We worked on the incremental generational garbage col- lector for over a year. But since we did not have access to applications with strong memory requirements (like games), it was difficult for us to test the collector in real scenarios. From March to December 2004 we released several work versions trying to get concrete feedback on the performance of the collector in real applications. We finally received re- ports of bizarre memory-allocation behavior, which we later managed to reproduce but not explain. In January 2005, Mike Pall, an active member of the Lua community, came up with memory-allocation graphs that explained the prob- lem: in some scenarios, there were subtle interactions be- tween the incremental behavior, the generational behavior, and the adaptive behavior, such that the collector “adapted” 10 Erik Hougaard reported that the Crazy Ivan robot would initially drive off course when Lua performed garbage collection (which could take a half second, but that was enough). To stay in course, they had to stop both motors and pause the robot during garbage collection. for less and less frequent collections. Because it was getting too complicated and unpredictable, we gave up the genera- tional aspect and implemented a simpler incremental collec- tor in Lua 5.1. During that time, programmers had been experimenting with the module system introduced in Lua 5.0. New pack- ages started to be produced, and old packages migrated to the new system. Package writers wanted to know the best way to build modules. In July 2005, during the development of Lua 5.1, an international Lua workshop organized by Mark Hamburg was held at Adobe in San Jose. (A similar work- shop organized by Wim Couwenberg and Daniel Silverstone was held in September 2006 at Océ in Venlo.) One of the presentations was about the novelties of Lua 5.1, and there were long discussions about modules and packages. As a re- sult, we made a few small but significant changes in the mod- ule system. Despite our “mechanisms, not policy” guideline for Lua, we defined a set of policies for writing modules and loading packages, and made small changes to support these policies better. Lua 5.1 was released in February 2006. Although the original motivation for Lua 5.1 was incremen- tal garbage collection, the improvement in the module sys- tem was probably the most visible change. On the other hand, that incremental garbage collection remained invisible shows that it succeeded in avoiding long pauses. 6. Feature evolution In this section, we discuss in detail the evolution of some of the features of Lua. 6.1 Types Types in Lua have been fairly stable. For a long time, Lua had only six basic types: nil, number, string, table, function, and userdata. (Actually, until Lua 3.0, C functions and Lua functions had different types internally, but that difference was transparent to callers.) The only real change happened in Lua 5.0, which introduced two new types: threads and booleans. The type thread was introduced to represent coroutines. Like all other Lua values, threads are first-class values. To avoid creating new syntax, all primitive operations on threads are provided by a library. For a long time we resisted introducing boolean values in Lua: nil was false and anything else was true. This state of affairs was simple and seemed sufficient for our purposes. However, nil was also used for absent fields in tables and for undefined variables. In some applications, it is important to allow table fields to be marked as false but still be seen as present; an explicit false value can be used for this. In Lua 5.0 we finally introduced boolean values true and false. Nil is still treated as false. In retrospect, it would probably have been better if nil raised an error in boolean expres- sions, as it does in other expressions. This would be more consistent with its role as proxy for undefined values. How- 2-11
- 41. ever, such a change would probably break many existing programs. LISP has similar problems, with the empty list representing both nil and false. Scheme explicitly represents false and treats the empty list as true, but some implementa- tions of Scheme still treat the empty list as false. 6.2 Tables Lua 1.1 had three syntactical constructs to create tables: ‘@()’, ‘@[]’, and ‘@{}’. The simplest form was ‘@()’, which created an empty table. An optional size could be given at creation time, as an efficiency hint. The form ‘@[]’ was used to create arrays, as in ‘@[2,4,9,16,25]’. In such tables, the keys were implicit natural numbers start- ing at 1. The form ‘@{}’ was used to create records, as in ‘@{name="John",age=35}’. Such tables were sets of key- value pairs in which the keys were explicit strings. A table created with any of those forms could be modified dynam- ically after creation, regardless of how it had been created. Moreover, it was possible to provide user functions when creating lists and records, as in ‘@foo[]’ or ‘@foo{}’. This syntax was inherited from SOL and was the expression of procedural data description, a major feature of Lua (see §2). The semantics was that a table was created and then the function was called with that table as its single argument. The function was allowed to check and modify the table at will, but its return values were ignored: the table was the final value of the expression. In Lua 2.1, the syntax for table creation was unified and simplified: the leading ‘@’ was removed and the only con- structor became ‘{· · ·}’. Lua 2.1 also allowed mixed con- structors, such as grades{8.5, 6.0, 9.2; name="John", major="math"} in which the array part was separated from the record part by a semicolon. Finally, ‘foo{· · ·}’ became sugar for ‘foo({· · ·})’. In other words, table constructors with func- tions became ordinary function calls. As a consequence, the function had to explicitly return the table (or whatever value it chose). Dropping the ‘@’ from constructors was a trivial change, but it actually changed the feel of the language, not merely its looks. Trivial changes that improve the feel of a language are not to be overlooked. This simplification in the syntax and semantics of ta- ble constructors had a side-effect, however. In Lua 1.1, the equality operator was ‘=’. With the unification of table con- structors in Lua 2.1, an expression like ‘{a=3}’ became am- biguous, because it could mean a table with either a pair ("a", 3) or a pair (1, b), where b is the value of the equal- ity ‘a=3’. To solve this ambiguity, in Lua 2.1 we changed the equality operator from ‘=’ to ‘==’. With this change, ‘{a=3}’ meant a table with the pair ("a", 3), while ‘{a==3}’ meant a table with the pair (1, b). These changes made Lua 2.1 incompatible with Lua 1.1 (hence the change in the major version number). Neverthe- less, since at that time virtually all Lua users were from Tec- graf, this was not a fatal move: existing programs were easily converted with the aid of ad-hoc tools that we wrote for this task. The syntax for table constructors has since remained mostly unchanged, except for an addition introduced in Lua 3.1: keys in the record part could be given by any ex- pression, by enclosing the expression inside brackets, as in ‘{[10*x+f(y)]=47}’. In particular, this allowed keys to be arbitrary strings, including reserved words and strings with spaces. Thus, ‘{function=1}’ is not valid (because ‘function’ is a reserved word), but ‘{["function"]=1}’ is valid. Since Lua 5.0, it is also possible to freely intermix the array part and the record part, and there is no need to use semicolons in table constructors. While the syntax of tables has evolved, the semantics of tables in Lua has not changed at all: tables are still asso- ciative arrays and can store arbitrary pairs of values. How- ever, frequently in practice tables are used solely as arrays (that is, with consecutive integer keys) or solely as records (that is, with string keys). Because tables are the only data- structuring mechanism in Lua, we have invested much ef- fort in implementing them efficiently inside Lua’s core. Un- til Lua 4.0, tables were implemented as pure hash tables, with all pairs stored explicitly. In Lua 5.0 we introduced a hybrid representation for tables: every table contains a hash part and an array part, and both parts can be empty. Lua de- tects whether a table is being used as an array and automat- ically stores the values associated to integer indices in the array part, instead of adding them to the hash part [31]. This division occurs only at a low implementation level; access to table fields is transparent, even to the virtual machine. Ta- bles automatically adapt their two parts according to their contents. This hybrid scheme has two advantages. First, access to values with integer keys is faster because no hashing is needed. Second, and more important, the array part takes roughly half the memory it would take if it were stored in the hash part, because the keys are implicit in the array part but explicit in the hash part. As a consequence, if a table is being used as an array, it performs as an array, as long as its integer keys are densely distributed. Moreover, no mem- ory or time penalty is paid for the hash part, because it does not even exist. Conversely, if the table is being used as a record and not as an array, then the array part is likely to be empty. These memory savings are important because it is common for a Lua program to create many small ta- bles (e.g., when tables are used to represent objects). Lua tables also handle sparse arrays gracefully: the statement ‘a={[1000000000]=1}’ creates a table with a single entry in its hash part, not an array with one billion elements. Another reason for investing effort into an efficient im- plementation of tables is that we can use tables for all kinds of tasks. For instance, in Lua 5.0 the standard library func- tions, which had existed since Lua 1.1 as global variables, 2-12
- 42. were moved to fields inside tables (see §5.5). More recently, Lua 5.1 brought a complete package and module system based on tables. Tables play a prominent role in Lua’s core. On two oc- casions we have been able to replace special data structures inside the core with ordinary Lua tables: in Lua 4.0 for repre- senting the global environment (which keeps all global vari- ables) and in Lua 5.0 for implementing extensible seman- tics (see §6.8). Starting with Lua 4.0, global variables are stored in an ordinary Lua table, called the table of globals, a simplification suggested by John Belmonte in April 2000. In Lua 5.0 we replaced tags and tag methods (introduced in Lua 3.0) by metatables and metamethods. Metatables are ordinary Lua tables and metamethods are stored as fields in metatables. Lua 5.0 also introduced environment tables that can be attached to Lua functions; they are the tables where global names in Lua functions are resolved at run time. Lua 5.1 extended environment tables to C functions, userdata, and threads, thus replacing the notion of global en- vironment. These changes simplified both the implementa- tion of Lua and the API for Lua and C programmers, be- cause globals and metamethods can be manipulated within Lua without the need for special functions. 6.3 Strings Strings play a major role in scripting languages and so the facilities to create and manipulate strings are an important part of the usability of such languages. The syntax for literal strings in Lua has had an interesting evolution. Since Lua 1.1, a literal string can be delimited by matching single or double quotes, and can contain C-like escape sequences. The use of both single and double quotes to delimit strings with the same semantics was a bit unusual at the time. (For instance, in the tradition of shell languages, Perl expands variables inside double-quoted strings, but not inside single-quoted strings.) While these dual quotes allow strings to contain one kind of quote without having to escape it, escape sequences are still needed for arbitrary text. Lua 2.2 introduced long strings, a feature not present in classical programming languages, but present in most script- ing languages.11 Long strings can run for several lines and do not interpret escape sequences; they provide a convenient way to include arbitrary text as a string, without having to worry about its contents. However, it is not trivial to de- sign a good syntax for long strings, especially because it is common to use them to include arbitrary program text (which may contain other long strings). This raises the ques- tion of how long strings end and whether they may nest. Until Lua 5.0, long strings were wrapped inside matching ‘[[· · ·]]’ and could contain nested long strings. Unfortu- nately, the closing delimiter ‘]]’ could easily be part of a valid Lua program in an unbalanced way, as in ‘a[b[i]]’, 11 ‘Long string’ is a Lua term. Other languages use terms such as ‘verbatim text’ or ‘heredoc’. or in other contexts, such as ‘<[!CDATA[· · ·]]>’ from XML. So, it was hard to reliably wrap arbitrary text as a long string. Lua 5.1 introduced a new form for long strings: text de- limited by matching ‘[===[· · ·]===]’, where the number of ‘=’ characters is arbitrary (including zero). These new long strings do not nest: a long string ends as soon as a closing de- limiter with the right number of ‘=’ is seen. Nevertheless, it is now easy to wrap arbitrary text, even text containing other long strings or unbalanced ‘]= · · · =]’ sequences: simply use an adequate number of ‘=’ characters. 6.4 Block comments Comments in Lua are signaled by ‘--’ and continue to the end of the line. This is the simplest kind of comment, and is very effective. Several other languages use single-line comments, with different marks. Languages that use ‘--’ for comments include Ada and Haskell. We never felt the need for multi-line comments, or block comments, except as a quick way to disable code. There was always the question of which syntax to use: the famil- iar ‘/* · · · */’ syntax used in C and several other languages does not mesh well with Lua’s single-line comments. There was also the question of whether block comments could nest or not, always a source of noise for users and of complexity for the lexer. Nested block comments happen when program- mers want to ‘comment out’ some block of code, to disable it. Naturally, they expect that comments inside the block of code are handled correctly, which can only happen if block comments can be nested. ANSI C supports block comments but does not allow nesting. C programmers typically disable code by using the C preprocessor idiom ‘#if 0 · · · #endif’. This scheme has the clear advantage that it interacts gracefully with existing comments in the disabled code. With this motivation and in- spiration, we addressed the need for disabling blocks of code in Lua — not the need for block comments — by introducing conditional compilation in Lua 3.0 via pragmas inspired in the C preprocessor. Although conditional compilation could be used for block comments, we do not think that it ever was. During work on Lua 4.0, we decided that the support for conditional compilation was not worth the complexity in the lexer and in its semantics for the user, especially after not having reached any consensus about a full macro facil- ity (see §7). So, in Lua 4.0 we removed support for con- ditional compilation and Lua remained without support for block comments.12 Block comments were finally introduced in Lua 5.0, in the form ‘--[[· · ·]]’. Because they intentionally mimicked the syntax of long strings (see §6.3), it was easy to modify the lexer to support block comments. This similarity also helped users to grasp both concepts and their syntax. Block 12 A further motivation was that by that time we had found a better way to generate and use debug information, and so the pragmas that controlled this were no longer needed. Removing conditional compilation allowed us to get rid of all pragmas. 2-13
- 43. comments can also be used to disable code: the idiom is to surround the code between two lines containing ‘--[[’ and ‘--]]’. The code inside those lines can be re-enabled by simply adding a single ‘-’ at the start of the first line: both lines then become harmless single-line comments. Like long strings, block comments could nest, but they had the same problems as long strings. In particular, valid Lua code containing unbalanced ‘]]’s, such as ‘a[b[i]]’, could not be reliably commented out in Lua 5.0. The new scheme for long strings in Lua 5.1 also applies to block com- ments, in the form of matching ‘--[===[· · ·]===]’, and so provides a simple and robust solution for this problem. 6.5 Functions Functions in Lua have always been first-class values. A func- tion can be created at run time by compiling and executing a string containing its definition.13 Since the introduction of anonymous functions and upvalues in Lua 3.1, programmers are able to create functions at run time without resorting to compilation from text. Functions in Lua, whether written in C or in Lua, have no declaration. At call time they accept a variable number of arguments: excess arguments are discarded and missing arguments are given the value nil. (This coincides with the semantics of multiple assignment.) C functions have always been able to handle a variable number of arguments. Lua 2.5 introduced vararg Lua functions, marked by a parameter list ending in ‘...’ (an experimental feature that became official only in Lua 3.0). When a vararg function was called, the arguments corresponding to the dots were collected into a table named ‘arg’. While this was simple and mostly convenient, there was no way to pass those arguments to another function, except by unpacking this table. Because programmers frequently want to just pass the arguments along to other functions, Lua 5.1 allows ‘...’ to be used in argument lists and on the right-hand side of assignments. This avoids the creation of the ‘arg’ table if it is not needed. The unit of execution of Lua is called a chunk; it is simply a sequence of statements. A chunk in Lua is like the main program in other languages: it can contain both function definitions and executable code. (Actually, a func- tion definition is executable code: an assignment.) At the same time, a chunk closely resembles an ordinary Lua func- tion. For instance, chunks have always had exactly the same kind of bytecode as ordinary Lua functions. However, before Lua 5.0, chunks needed some internal magic to start execut- ing. Chunks began to look like ordinary functions in Lua 2.2, when local variables outside functions were allowed as an undocumented feature (that became official only in Lua 3.1). Lua 2.5 allowed chunks to return values. In Lua 3.0 chunks became functions internally, except that they were executed 13 Some people maintain that the ability to evaluate code from text at run time and within the environment of the running program is what character- izes scripting languages. right after being compiled; they did not exist as functions at the user level. This final step was taken in Lua 5.0, which broke the loading and execution of chunks into two steps, to provide host programmers better control for handling and reporting errors. As a consequence, in Lua 5.0 chunks be- came ordinary anonymous functions with no arguments. In Lua 5.1 chunks became anonymous vararg functions and thus can be passed values at execution time. Those values are accessed via the new ‘...’ mechanism. From a different point of view, chunks are like modules in other languages: they usually provide functions and vari- ables to the global environment. Originally, we did not in- tend Lua to be used for large-scale programming and so we did not feel the need to add an explicit notion of modules to Lua. Moreover, we felt that tables would be sufficient for building modules, if necessary. In Lua 5.0 we made that feel- ing explicit by packaging all standard libraries into tables. This encouraged other people to do the same and made it easier to share libraries. We now feel that Lua can be used for large-scale programming, especially after Lua 5.1 brought a package system and a module system, both based on tables. 6.6 Lexical scoping From an early stage in the development of Lua we started thinking about first-class functions with full lexical scoping. This is an elegant construct that fits well within Lua’s philos- ophy of providing few but powerful constructs. It also makes Lua apt for functional programming. However, we could not figure out a reasonable implementation for full lexical scop- ing. Since the beginning Lua has used a simple array stack to keep activation records (where all local variables and tem- poraries live). This implementation had proved simple and efficient, and we saw no reason to change it. When we allow nested functions with full lexical scoping, a variable used by an inner function may outlive the function that created it, and so we cannot use a stack discipline for such variables. Simple Scheme implementations allocate frames in the heap. Already in 1987, Dybvig [20] described how to use a stack to allocate frames, provided that those frames did not contain variables used by nested functions. His method requires that the compiler know beforehand whether a vari- able appears as a free variable in a nested function. This does not suit the Lua compiler because it generates code to ma- nipulate variables as soon as it parses an expression; at that moment, it cannot know whether any variable is later used free in a nested function. We wanted to keep this design for implementing Lua, because of its simplicity and efficiency, and so could not use Dybvig’s method. For the same rea- son, we could not use advanced compiler techniques, such as data-flow analysis. Currently there are several optimization strategies to avoid using the heap for frames (e.g., [21]), but they all need compilers with intermediate representations, which the Lua compiler does not use. McDermott’s proposal for stack frame allocation [36], which is explicitly addressed to inter- 2-14
- 44. preters, is the only one we know of that does not require in- termediate representation for code generation. Like our cur- rent implementation [31], his proposal puts variables in the stack and moves them to the heap on demand, if they go out of scope while being used by a nested closure. However, his proposal assumes that environments are represented by as- sociation lists. So, after moving an environment to the heap, the interpreter has to correct only the list header, and all ac- cesses to local variables automatically go to the heap. Lua uses real records as activation records, with local-variable access being translated to direct accesses to the stack plus an offset, and so cannot use McDermott’s method. For a long time those difficulties kept us from introducing nested first-class functions with full lexical scoping in Lua. Finally, in Lua 3.1 we settled on a compromise that we called upvalues. In this scheme, an inner function cannot access and modify external variables when it runs, but it can access the values those variables had when the function was cre- ated. Those values are called upvalues. The main advantage of upvalues is that they can be implemented with a simple scheme: all local variables live in the stack; when a function is created, it is wrapped in a closure containing copies of the values of the external variables used by the function. In other words, upvalues are the frozen values of external vari- ables.14 To avoid misunderstandings, we created a new syn- tax for accessing upvalues: ‘%varname’. This syntax made it clear that the code was accessing the frozen value of that variable, not the variable itself. Upvalues proved to be very useful, despite being immutable. When necessary, we could simulate mutable external variables by using a table as the upvalue: although we could not change the table itself, we could change its fields. This feature was especially useful for anonymous functions passed to higher-order functions used for table traversal and pattern matching. In December 2000, Roberto wrote in the first draft of his book [27] that “Lua has a form of proper lexical scop- ing through upvalues.” In July 2001 John D. Ramsdell ar- gued in the mailing list that “a language is either lexically scoped or it is not; adding the adjective ‘proper’ to the phrase ‘lexical scoping’ is meaningless.” That message stirred us to search for a better solution and a way to implement full lexical scoping. By October 2001 we had an initial imple- mentation of full lexical scoping and described it to the list. The idea was to access each upvalue through an indirection that pointed to the stack while the variable was in scope; at the end of the scope a special virtual machine instruc- tion “closed” the upvalue, moving the variable’s value to a heap-allocated space and correcting the indirection to point there. Open closures (those with upvalues still pointing to the stack) were kept in a list to allow their correction and 14 A year later Java adopted a similar solution to allow inner classes. Instead of freezing the value of an external variable, Java insists that you can only access final variables in inner classes, and so ensures that the variable is frozen. the reuse of open upvalues. Reuse is essential to get the cor- rect semantics. If two closures, sharing an external variable, have their own upvalues, then at the end of the scope each closure will have its own copy of the variable, but the cor- rect semantics dictates that they should share the variable. To ensure reuse, the algorithm that created closures worked as follows: for each external variable used by the closure, it first searched the list of open closures. If it found an upvalue pointing to that external variable, it reused that upvalue; oth- erwise, it created a new upvalue. Edgar Toering, an active member of the Lua community, misunderstood our description of lexical scoping. It turned out that the way he understood it was better than our orig- inal idea: instead of keeping a list of open closures, keep a list of open upvalues. Because the number of local variables used by closures is usually smaller than the number of clo- sures using them (the first is statically limited by the program text), his solution was more efficient than ours. It was also easier to adapt to coroutines (which were being implemented at around the same time), because we could keep a separate list of upvalues for each stack. We added full lexical scoping to Lua 5.0 using this algorithm because it met all our require- ments: it could be implemented with a one-pass compiler; it imposed no burden on functions that did not access exter- nal local variables, because they continued to manipulate all their local variables in the stack; and the cost to access an external local variable was only one extra indirection [31]. 6.7 Coroutines For a long time we searched for some kind of first-class continuations for Lua. This search was motivated by the existence of first-class continuations in Scheme (always a source of inspiration to us) and by demands from game programmers for some mechanism for “soft” multithreading (usually described as “some way to suspend a character and continue it later”). In 2000, Maria Julia de Lima implemented full first-class continuations on top of Lua 4.0 alpha, as part of her Ph.D. work [35]. She used a simple approach because, like lexi- cal scoping, smarter techniques to implement continuations were too complex compared to the overall simplicity of Lua. The result was satisfactory for her experiments, but too slow to be incorporated in a final product. Nevertheless, her im- plementation uncovered a problem peculiar to Lua. Since Lua is an extensible extension language, it is possible (and common) to call Lua from C and C from Lua. Therefore, at any given point in the execution of a Lua program, the cur- rent continuation usually has parts in Lua mixed with parts in C. Although it is possible to manipulate a Lua continu- ation (essentially by manipulating the Lua call stack), it is impossible to manipulate a C continuation within ANSI C. At that time, we did not understand this problem deeply enough. In particular, we could not figure out what the ex- act restrictions related to C calls were. Lima simply forbade any C calls in her implementation. Again, that solution was 2-15
- 45. satisfactory for her experiments, but unacceptable for an of- ficial Lua version because the ease of mixing Lua code with C code is one of Lua’s hallmarks. Unaware of this difficulty, in December 2001 Thatcher Ulrich announced in the mailing list: I’ve created a patch for Lua 4.0 that makes calls from Lua to Lua non-recursive (i.e., ‘stackless’). This al- lows the implementation of a ‘sleep()’ call, which ex- its from the host program [...], and leaves the Lua state in a condition where the script can be resumed later via a call to a new API function, lua_resume. In other words, he proposed an asymmetric coroutine mech- anism, based on two primitives: yield (which he called sleep) and resume. His patch followed the high-level description given in the mailing list by Bret Mogilefsky on the changes made to Lua 2.5 and 3.1 to add cooperative multitasking in Grim Fandango. (Bret could not provide details, which were proprietary.) Shortly after this announcement, during the Lua Library Design Workshop held at Harvard in February 2002, there was some discussion about first-class continuations in Lua. Some people claimed that, if first-class continuations were deemed too complex, we could implement one-shot contin- uations. Others argued that it would be better to implement symmetric coroutines. But we could not find a proper imple- mentation of any of these mechanisms that could solve the difficulty related to C calls. It took us some time to realize why it was hard to im- plement symmetric coroutines in Lua, and also to under- stand how Ulrich’s proposal, based on asymmetric corou- tines, avoided our difficulties. Both one-shot continuations and symmetric coroutines involve the manipulation of full continuations. So, as long as these continuations include any C part, it is impossible to capture them (except by using fa- cilities outside ANSI C). In contrast, an asymmetric corou- tine mechanism based on yield and resume manipulates par- tial continuations: yield captures the continuation up to the corresponding resume [19]. With asymmetric coroutines, the current continuation can include C parts, as long as they are outside the partial continuation being captured. In other words, the only restriction is that we cannot yield across a C call. After that realization, and based on Ulrich’s proof-of- concept implementation, we were able to implement asym- metrical coroutines in Lua 5.0. The main change was that the interpreter loop, which executes the instructions for the vir- tual machine, ceased to be recursive. In previous versions, when the interpreter loop executed a CALL instruction, it called itself recursively to execute the called function. Since Lua 5.0, the interpreter behaves more like a real CPU: when it executes a CALL instruction, it pushes some context infor- mation onto a call stack and proceeds to execute the called function, restoring the context when that function returns. After that change, the implementation of coroutines became straightforward. Unlike most implementations of asymmetrical corou- tines, in Lua coroutines are what we call stackfull [19]. With them, we can implement symmetrical coroutines and even the call/1cc operator (call with current one-shot continua- tion) proposed for Scheme [11]. However, the use of C func- tions is severely restricted within these implementations. We hope that the introduction of coroutines in Lua 5.0 marks a revival of coroutines as powerful control struc- tures [18]. 6.8 Extensible semantics As mentioned in §5.2, we introduced extensible semantics in Lua 2.1 in the form of fallbacks as a general mechanism to allow the programmer to intervene whenever Lua did not know how to proceed. Fallbacks thus provided a restricted form of resumable exception handling. In particular, by us- ing fallbacks, we could make a value respond to operations not originally meant for it or make a value of one type be- have like a value of another type. For instance, we could make userdata and tables respond to arithmetic operations, userdata behave as tables, strings behave as functions, etc. Moreover, we could make a table respond to keys that were absent in it, which is fundamental for implementing inheri- tance. With fallbacks for table indexing and a little syntactic sugar for defining and calling methods, object-oriented pro- gramming with inheritance became possible in Lua. Although objects, classes, and inheritance were not core concepts in Lua, they could be implemented directly in Lua, in many flavors, according to the needs of the application. In other words, Lua provided mechanisms, not policy — a tenet that we have tried to follow closely ever since. The simplest kind of inheritance is inheritance by del- egation, which was introduced by Self and adopted in other prototype-based languages such as NewtonScript and JavaScript. The code below shows an implementation of in- heritance by delegation in Lua 2.1. function Index(a,i) if i == "parent" then return nil end local p = a.parent if type(p) == "table" then return p[i] else return nil end end setfallback("index", Index) When a table was accessed for an absent field (be it an attribute or a method), the index fallback was triggered. Inheritance was implemented by setting the index fallback to follow a chain of “parents” upwards, possibly triggering 2-16
- 46. the index fallback again, until a table had the required field or the chain ended. After setting that index fallback, the code below printed ‘red’ even though ‘b’ did not have a ‘color’ field: a=Window{x=100, y=200, color="red"} b=Window{x=300, y=400, parent=a} print(b.color) There was nothing magical or hard-coded about delega- tion through a “parent” field. Programmers had complete freedom: they could use a different name for the field con- taining the parent, they could implement multiple inheri- tance by trying a list of parents, etc. Our decision not to hard-code any of those possible behaviors led to one of the main design concepts of Lua: meta-mechanisms. Instead of littering the language with lots of features, we provided ways for users to program the features themselves, in the way they wanted them, and only for those features they needed. Fallbacks greatly increased the expressiveness of Lua. However, fallbacks were global handlers: there was only one function for each event that could occur. As a consequence, it was difficult to mix different inheritance mechanisms in the same program, because there was only one hook for implementing inheritance (the index fallback). While this might not be a problem for a program written by a single group on top of its own object system, it became a problem when one group tried to use code from other groups, because their visions of the object system might not be consistent with each other. Hooks for different mechanisms could be chained, but chaining was slow, complicated, error-prone, and not very polite. Fallback chaining did not encourage code sharing and reuse; in practice almost nobody did it. This made it very hard to use third-party libraries. Lua 2.1 allowed userdata to be tagged. In Lua 3.0 we extended tags to all values and replaced fallbacks with tag methods. Tag methods were fallbacks that operated only on values with a given tag. This made it possible to implement independent notions of inheritance, for instance. No chain- ing was needed because tag methods for one tag did not af- fect tag methods for another tag. The tag method scheme worked very well and lasted until Lua 5.0, when we replaced tags and tag methods by metatables and metamethods. Metatables are just ordinary Lua tables and so can be manipulated within Lua without the need for special functions. Like tags, metatables can be used to represent user-defined types with userdata and tables: all objects of the same “type” should share the same metatable. Unlike tags, metatables and their contents are naturally collected when no references remain to them. (In contrast, tags and their tag methods had to live until the end of the program.) The introduction of metatables also simplified the implementation: while tag methods had their own private representation inside Lua’s core, metatables use mainly the standard table machinery. The code below shows the implementation of inheritance in Lua 5.0. The index metamethod replaces the index tag method and is represented by the ‘__index’ field in the metatable. The code makes ‘b’ inherit from ‘a’ by setting a metatable for ‘b’ whose ‘__index’ field points to ‘a’. (In general, index metamethods are functions, but we have allowed them to be tables to support simple inheritance by delegation directly.) a=Window{x=100, y=200, color="red"} b=Window{x=300, y=400} setmetatable(b,{ __index = a }) print(b.color) --> red 6.9 C API Lua is provided as a library of C functions and macros that allow the host program to communicate with Lua. This API between Lua and C is one of the main components of Lua; it is what makes Lua an embeddable language. Like the rest of the language, the API has gone through many changes during Lua’s evolution. Unlike the rest of the language, however, the API design received little outside in- fluence, mainly because there has been little research activity in this area. The API has always been bi-directional because, since Lua 1.0, we have considered calling Lua from C and call- ing C from Lua equally important. Being able to call Lua from C is what makes Lua an extension language, that is, a language for extending applications through configuration, macros, and other end-user customizations. Being able to call C from Lua makes Lua an extensible language, because we can use C functions to extend Lua with new facilities. (That is why we say that Lua is an extensible extension lan- guage [30].) Common to both these aspects are two mis- matches between C and Lua to which the API must adjust: static typing in C versus dynamic typing in Lua and manual memory management in C versus automatic garbage collec- tion in Lua. Currently, the C API solves both difficulties by using an abstract stack15 to exchange data between Lua and C. Every C function called by Lua gets a new stack frame that initially contains the function arguments. If the C function wants to return values to Lua, it pushes those values onto the stack just before returning. Each stack slot can hold a Lua value of any type. For each Lua type that has a corresponding representation in C (e.g., strings and numbers), there are two API functions: an injec- tion function, which pushes onto the stack a Lua value cor- responding to the given C value; and a projection function, which returns a C value corresponding to the Lua value at a given stack position. Lua values that have no correspond- ing representation in C (e.g., tables and functions) can be manipulated via the API by using their stack positions. 15 Throughout this section, ‘stack’ always means this abstract stack. Lua never accesses the C stack. 2-17
- 47. Practically all API functions get their operands from the stack and push their results onto the stack. Since the stack can hold values of any Lua type, these API functions operate with any Lua type, thus solving the typing mismatch. To prevent the collection of Lua values in use by C code, the values in the stack are never collected. When a C function returns, its Lua stack frame vanishes, automatically releasing all Lua values that the C function was using. These values will eventually be collected if no further references to them exist. This solves the memory management mismatch. It took us a long time to arrive at the current API. To discuss how the API evolved, we use as illustration the C equivalent of the following Lua function: function foo(t) return t.x end In words, this function receives a single parameter, which should be a table, and returns the value stored at the ‘x’ field in that table. Despite its simplicity, this example illustrates three important issues in the API: how to get parameters, how to index tables, and how to return results. In Lua 1.0, we would write foo in C as follows: void foo_l (void) { lua_Object t = lua_getparam(1); lua_Object r = lua_getfield(t, "x"); lua_pushobject(r); } Note that the required value is stored at the string index "x" because ‘t.x’ is syntactic sugar for ‘t["x"]’. Note also that all components of the API start with ‘lua_’ (or ‘LUA_’) to avoid name clashes with other C libraries. To export this C function to Lua with the name ‘foo’ we would do lua_register("foo", foo_l); After that, foo could be called from Lua code just like any other Lua function: t = {x = 200} print(foo(t)) --> 200 A key component of the API was the type lua_Object, defined as follows: typedef struct Object *lua_Object; In words, lua_Object was an abstract type that represented Lua values in C opaquely. Arguments given to C functions were accessed by calling lua_getparam, which returned a lua_Object. In the example, we call lua_getparam once to get the table, which is supposed to be the first argument to foo. (Extra arguments are silently ignored.) Once the table is available in C (as a lua_Object), we get the value of its "x" field by calling lua_getfield. This value is also represented in C as a lua_Object, which is finally sent back to Lua by pushing it onto the stack with lua_pushobject. The stack was another key component of the API. It was used to pass values from C to Lua. There was one push function for each Lua type with a direct representation in C: lua_pushnumber for numbers, lua_pushstring for strings, and lua_pushnil, for the special value nil. There was also lua_pushobject, which allowed C to pass back to Lua an arbitrary Lua value. When a C function returned, all values in the stack were returned to Lua as the results of the C function (functions in Lua can return multiple values). Conceptually, a lua_Object was a union type, since it could refer to any Lua value. Several scripting languages, including Perl, Python, and Ruby, still use a union type to represent their values in C. The main drawback of this representation is that it is hard to design a garbage collector for the language. Without extra information, the garbage collector cannot know whether a value has a reference to it stored as a union in the C code. Without this knowledge, the collector may collect the value, making the union a dangling pointer. Even when this union is a local variable in a C function, this C function can call Lua again and trigger garbage collection. Ruby solves this problem by inspecting the C stack, a task that cannot be done in a portable way. Perl and Python solve this problem by providing explicit reference-count functions for these union values. Once you increment the reference count of a value, the garbage collector will not collect that value until you decrement the count to zero. However, it is not easy for the programmer to keep these reference counts right. Not only is it easy to make a mistake, but it is dif- ficult to find the error later (as anyone who has ever de- bugged memory leaks and dangling pointers can attest). Fur- thermore, reference counting cannot deal with cyclic data structures that become garbage. Lua never provided such reference-count functions. Be- fore Lua 2.1, the best you could do to ensure that an unan- chored lua_Object was not collected was to avoid calling Lua whenever you had a reference to such a lua_Object. (As long as you could ensure that the value referred to by the union was also stored in a Lua variable, you were safe.) Lua 2.1 brought an important change: it kept track of all lua_Object values passed to C, ensuring that they were not collected while the C function was active. When the C func- tion returned to Lua, then (and only then) all references to these lua_Object values were released, so that they could be collected.16 More specifically, in Lua 2.1 a lua_Object ceased to be a pointer to Lua’s internal data structures and became an index into an internal array that stored all values that had to be given to C: typedef unsigned int lua_Object; This change made the use of lua_Object reliable: while a value was in that array, it would not be collected by Lua. 16 A similar method is used by JNI to handle “local references”. 2-18
- 48. When the C function returned, its whole array was erased, and the values used by the function could be collected if pos- sible. (This change also gave more freedom for implement- ing the garbage collector, because it could move objects if necessary; however, we did not followed this path.) For simple uses, the Lua 2.1 behavior was very practi- cal: it was safe and the C programmer did not have to worry about reference counts. Each lua_Object behaved like a local variable in C: the corresponding Lua value was guar- anteed to be alive during the lifetime of the C function that produced it. For more complex uses, however, this simple scheme had two shortcomings that demanded extra mecha- nisms: sometimes a lua_Object value had to be locked for longer than the lifetime of the C function that produced it; sometimes it had to be locked for a shorter time. The first of those shortcomings had a simple solution: Lua 2.1 introduced a system of references. The function lua_lock got a Lua value from the stack and returned a reference to it. This reference was an integer that could be used any time later to retrieve that value, using the lua_getlocked function. (There was also a lua_unlock function, which destroyed a reference.) With such refer- ences, it was easy to keep Lua values in non-local C vari- ables. The second shortcoming was more subtle. Objects stored in the internal array were released only when the function returned. If a function used too many values, it could over- flow the array or cause an out-of-memory error. For instance, consider the following higher-order iterator function, which repeatedly calls a function and prints the result until the call returns nil: void l_loop (void) { lua_Object f = lua_getparam(1); for (;;) { lua_Object res; lua_callfunction(f); res = lua_getresult(1); if (lua_isnil(res)) break; printf("%sn", lua_getstring(res)); } } The problem with this code was that the string returned by each call could not be collected until the end of the loop (that is, of the whole C function), thus opening the possibility of array overflow or memory exhaustion. This kind of error can be very difficult to track, and so the implementation of Lua 2.1 set a hard limit on the size of the internal array that kept lua_Object values alive. That made the error easier to track because Lua could say “too many objects in a C function” instead of a generic out-of-memory error, but it did not avoid the problem. To address the problem, the API in Lua 2.1 offered two functions, lua_beginblock and lua_endblock, that cre- ated dynamic scopes (“blocks”) for lua_Object values; all values created after a lua_beginblock were removed from the internal array at the corresponding lua_endblock. However, since a block discipline could not be forced onto C programmers, it was all too common to forget to use these blocks. Moreover, such explicit scope control was a little tricky to use. For instance, a naive attempt to correct our previous example by enclosing the for body within a block would fail: we had to call lua_endblock just before the break, too. This difficulty with the scope of Lua objects persisted through several versions and was solved only in Lua 4.0, when we redesigned the whole API. Nevertheless, as we said before, for typical uses the API was very easy to use, and most programmers never faced the kind of situation described here. More important, the API was safe. Erroneous use could produce well-defined errors, but not dangling ref- erences or memory leaks. Lua 2.1 brought other changes to the API. One was the introduction of lua_getsubscript, which allowed the use of any value to index a table. This function had no explicit arguments: it got both the table and the key from the stack. The old lua_getfield was redefined as a macro, for com- patibility: #define lua_getfield(o,f) (lua_pushobject(o), lua_pushstring(f), lua_getsubscript()) (Backward compatibility of the C API is usually imple- mented using macros, whenever feasible.) Despite all those changes, syntactically the API changed little from Lua 1 to Lua 2. For instance, our illustrative func- tion foo could be written in Lua 2 exactly as we wrote it for Lua 1.0. The meaning of lua_Object was quite different, and lua_getfield was implemented on top of new primi- tive operations, but for the average user it was as if nothing had changed. Thereafter, the API remained fairly stable until Lua 4.0. Lua 2.4 expanded the reference mechanism to support weak references. A common design in Lua programs is to have a Lua object (typically a table) acting as a proxy for a C object. Frequently the C object must know who its proxy is and so keeps a reference to the proxy. However, that reference prevents the collection of the proxy object, even when the object becomes inaccessible from Lua. In Lua 2.4, the program could create a weak reference to the proxy; that reference did not prevent the collection of the proxy object. Any attempt to retrieve a collected reference resulted in a special value LUA_NOOBJECT. Lua 4.0 brought two main novelties in the C API: support for multiple Lua states and a virtual stack for exchanging values between C and Lua. Support for multiple, indepen- dent Lua states was achieved by eliminating all global state. Until Lua 3.0, only one Lua state existed and it was imple- mented using many static variables scattered throughout the code. Lua 3.1 introduced multiple independent Lua states; all static variables were collected into a single C struct. An 2-19
- 49. API function was added to allow switching states, but only one Lua state could be active at any moment. All other API functions operated over the active Lua state, which remained implicit and did not appear in the calls. Lua 4.0 introduced explicit Lua states in the API. This created a big incompat- ibility with previous versions.17 All C code that communi- cated with Lua (in particular, all C functions registered to Lua) had to be changed to include an explicit state argument in calls to the C API. Since all C functions had to be rewrit- ten anyway, we took this opportunity and made another ma- jor change in the C – Lua communication in Lua 4.0: we replaced the concept of lua_Object by an explicit virtual stack used for all communication between Lua and C in both directions. The stack could also be used to store temporary values. In Lua 4.0, our foo example could be written as follows: int foo_l (lua_State *L) { lua_pushstring(L, "x"); lua_gettable(L, 1); return 1; } The first difference is the function signature: foo_l now re- ceives a Lua state on which to operate and returns the num- ber of values returned by the function in the stack. In pre- vious versions, all values left in the stack when the func- tion ended were returned to Lua. Now, because the stack is used for all operations, it can contain intermediate values that are not to be returned, and so the function needs to tell Lua how many values in the stack to consider as return val- ues. Another difference is that lua_getparam is no longer needed, because function arguments come in the stack when the function starts and can be directly accessed by their in- dex, like any other stack value. The last difference is the use of lua_gettable, which replaced lua_getsubscript as the means to access table fields. lua_gettable receives the table to be indexed as a stack position (instead of as a Lua object), pops the key from the top of the stack, and pushes the result. Moreover, it leaves the table in the same stack position, because tables are frequently indexed repeatedly. In foo_l, the table used by lua_gettable is at stack position 1, because it is the first argument to that function, and the key is the string "x", which needs to be pushed onto the stack before calling lua_gettable. That call replaces the key in the stack with the corresponding table value. So, after lua_gettable, there are two values in the stack: the table at position 1 and the result of the indexing at position 2, which is the top of the stack. The C function returns 1 to tell Lua to use that top value as the single result returned by the function. To further illustrate the new API, here is an implementa- tion of our loop example in Lua 4.0: 17 We provided a module that emulated the 3.2 API on top of the 4.0 API, but we do not think it was used much. int l_loop (lua_State *L) { for (;;) { lua_pushvalue(L, 1); lua_call(L, 0, 1); if (lua_isnil(L, -1)) break; printf("%sn", lua_tostring(L, -1)); lua_pop(L, 1); } return 0; } To call a Lua function, we push it onto the stack and then push its arguments, if any (none in the example). Then we call lua_call, telling how many arguments to get from the stack (and therefore implicitly also telling where the function is in the stack) and how many results we want from the call. In the example, we have no arguments and expect one result. The lua_call function removes the function and its arguments from the stack and pushes back exactly the requested number of results. The call to lua_pop removes the single result from the stack, leaving the stack at the same level as at the beginning of the loop. For convenience, we can index the stack from the bottom, with positive indices, or from the top, with negative indices. In the example, we use index -1 in lua_isnil and lua_tostring to refer to the top of the stack, which contains the function result. With hindsight, the use of a single stack in the API seems an obvious simplification, but when Lua 4.0 was released many users complained about the complexity of the new API. Although Lua 4.0 had a much cleaner conceptual model for its API, the direct manipulation of the stack requires some thought to get right. Many users were content to use the previous API without any clear conceptual model of what was going on behind the scenes. Simple tasks did not require a conceptual model at all and the previous API worked quite well for them. More complex tasks often broke whatever private models users had, but most users never programmed complex tasks in C. So, the new API was seen as too complex at first. However, such skepticism gradually vanished, as users came to understand and value the new model, which proved to be simpler and much less error- prone. The possibility of multiple states in Lua 4.0 created an un- expected problem for the reference mechanism. Previously, a C library that needed to keep some object fixed could cre- ate a reference to the object and store that reference in a global C variable. In Lua 4.0, if a C library was to work with several states, it had to keep an individual reference for each state and so could not keep the reference in a global C vari- able. To solve this difficulty, Lua 4.0 introduced the registry, which is simply a regular Lua table available to C only. With the registry, a C library that wants to keep a Lua object can choose a unique key and associate the object with this key in the registry. Because each independent Lua state has its own registry, the C library can use the same key in each state to manipulate the corresponding object. 2-20
- 50. We could quite easily implement the original reference mechanism on top of the registry by using integer keys to represent references. To create a new reference, we just find an unused integer key and store the value at that key. Retriev- ing a reference becomes a simple table access. However, we could not implement weak references using the registry. So, Lua 4.0 kept the previous reference mechanism. In Lua 5.0, with the introduction of weak tables in the language, we were finally able to eliminate the reference mechanism from the core and move it to a library. The C API has slowly evolved toward completeness. Since Lua 4.0, all standard library functions can be written using only the C API. Until then, Lua had a number of built- in functions (from 7 in Lua 1.1 to 35 in Lua 3.2), most of which could have been written using the C API but were not because of a perceived need for speed. A few built-in func- tions could not have been written using the C API because the C API was not complete. For instance, until Lua 3.2 it was not possible to iterate over the contents of a table using the C API, although it was possible to do it in Lua using the built-in function next. The C API is not yet complete and not everything that can be done in Lua can be done in C; for instance, the C API lacks functions for performing arith- metic operations on Lua values. We plan to address this issue in the next version. 6.10 Userdata Since its first version, an important feature of Lua has been its ability to manipulate C data, which is provided by a special Lua data type called userdata. This ability is an essential component in the extensibility of Lua. For Lua programs, the userdata type has undergone no changes at all throughout Lua’s evolution: although userdata are first-class values, userdata is an opaque type and its only valid operation in Lua is equality test. Any other operation over userdata (creation, inspection, modification) must be provided by C functions. For C functions, the userdata type has undergone several changes in Lua’s evolution. In Lua 1.0, a userdata value was a simple void* pointer. The main drawback of this simplic- ity was that a C library had no way to check whether a user- data was valid. Although Lua code cannot create userdata values, it can pass userdata created by one library to another library that expects pointers to a different structure. Because C functions had no mechanisms to check this mismatch, the result of this pointer mismatch was usually fatal to the appli- cation. We have always considered it unacceptable for a Lua program to be able to crash the host application. Lua should be a safe language. To overcome the pointer mismatch problem, Lua 2.1 in- troduced the concept of tags (which would become the seed for tag methods in Lua 3.0). A tag was simply an arbitrary in- teger value associated with a userdata. A userdata’s tag could only be set once, when the userdata was created. Provided that each C library used its own exclusive tag, C code could easily ensure that a userdata had the expected type by check- ing its tag. (The problem of how a library writer chose a tag that did not clash with tags from other libraries remained open. It was only solved in Lua 3.0, which provided tag man- agement via lua_newtag.) A bigger problem with Lua 2.1 was the management of C resources. More often than not, a userdata pointed to a dynamically allocated structure in C, which had to be freed when its corresponding userdata was collected in Lua. However, userdata were values, not objects. As such, they were not collected (in the same way that numbers are not collected). To overcome this restriction, a typical design was to use a table as a proxy for the C structure in Lua, storing the actual userdata in a predefined field of the proxy table. When the table was collected, its finalizer would free the corresponding C structure. This simple solution created a subtle problem. Because the userdata was stored in a regular field of the proxy table, a malicious user could tamper with it from within Lua. Specif- ically, a user could make a copy of the userdata and use the copy after the table was collected. By that time, the corre- sponding C structure had been destroyed, making the user- data a dangling pointer, with disastrous results. To improve the control of the life cycle of userdata, Lua 3.0 changed userdata from values to objects, subject to garbage collec- tion. Users could use the userdata finalizer (the garbage- collection tag method) to free the corresponding C structure. The correctness of Lua’s garbage collector ensured that a userdata could not be used after being collected. However, userdata as objects created an identity problem. Given a userdata, it is trivial to get its corresponding pointer, but frequently we need to do the reverse: given a C pointer, we need to get its corresponding userdata.18 In Lua 2, two userdata with the same pointer and the same tag would be equal; equality was based on their values. So, given the pointer and the tag, we had the userdata. In Lua 3, with userdata being objects, equality was based on identity: two userdata were equal only when they were the same userdata (that is, the same object). Each userdata created was different from all others. Therefore, a pointer and a tag would not be enough to get the corresponding userdata. To solve this difficulty, and also to reduce incompatibili- ties with Lua 2, Lua 3 adopted the following semantics for the operation of pushing a userdata onto the stack: if Lua already had a userdata with the given pointer and tag, then that userdata was pushed on the stack; otherwise, a new user- data was created and pushed on the stack. So, it was easy for C code to translate a C pointer to its corresponding userdata in Lua. (Actually, the C code could be the same as it was in Lua 2.) 18 A typical scenario for this need is the handling of callbacks in a GUI toolkit. The C callback associated with a widget gets only a pointer to the widget, but to pass this callback to Lua we need the userdata that represents that widget in Lua. 2-21
- 51. However, Lua 3 behavior had a major drawback: it com- bined into a single primitive (lua_pushuserdata) two ba- sic operations: userdata searching and userdata creation. For instance, it was impossible to check whether a given C pointer had a corresponding userdata without creating that userdata. Also, it was impossible to create a new userdata re- gardless of its C pointer. If Lua already had a userdata with that value, no new userdata would be created. Lua 4 mitigated that drawback by introducing a new func- tion, lua_newuserdata. Unlike lua_pushuserdata, this function always created a new userdata. Moreover, what was more important at that time, those userdata were able to store arbitrary C data, instead of pointers only. The user would tell lua_newuserdata the amount memory to be allocated and lua_newuserdata returned a pointer to the allocated area. By having Lua allocate memory for the user, several com- mon tasks related to userdata were simplified. For instance, C code did not need to handle memory-allocation errors, be- cause they were handled by Lua. More important, C code did not need to handle memory deallocation: memory used by such userdata was released by Lua automatically, when the userdata was collected. However, Lua 4 still did not offer a nice solution to the search problem (i.e., finding a userdata given its C pointer). So, it kept the lua_pushuserdata operation with its old be- havior, resulting in a hybrid system. It was only in Lua 5 that we removed lua_pushuserdata and dissociated userdata creation and searching. Actually, Lua 5 removed the search- ing facility altogether. Lua 5 also introduced light userdata, which store plain C pointer values, exactly like regular user- data in Lua 1. A program can use a weak table to associate C pointers (represented as light userdata) to its correspond- ing “heavy” userdata in Lua. As is usual in the evolution of Lua, userdata in Lua 5 is more flexible than it was in Lua 4; it is also simpler to explain and simpler to implement. For simple uses, which only require storing a C structure, userdata in Lua 5 is trivial to use. For more complex needs, such as those that require mapping a C pointer back to a Lua userdata, Lua 5 offers the mechanisms (light userdata and weak tables) for users to implement strategies suited to their applications. 6.11 Reflectivity Since its very first version Lua has supported some reflective facilities. A major reason for this support was the proposed use of Lua as a configuration language to replace SOL. As described in §4, our idea was that the programmer could use the language itself to write type-checking routines, if needed. For instance, if a user wrote something like T = @track{ y=9, x=10, id="1992-34" } we wanted to be able to check that the track did have a y field and that this field was a number. We also wanted to be able to check that the track did not have extraneous fields (possibly to catch typing mistakes). For these two tasks, we needed access to the type of a Lua value and a mechanism to traverse a table and visit all its pairs. Lua 1.0 provided the needed functionality with only two functions, which still exist: type and next. The type func- tion returns a string describing the type of any given value ("number", "nil", "table", etc.). The next function re- ceives a table and a key and returns a “next” key in the ta- ble (in an arbitrary order). The call next(t,nil) returns a “first” key. With next we can traverse a table and process all its pairs. For instance, the following code prints all pairs in a table t:19 k = next(t,nil) while k do print(k,t[k]) k = next(t,k) end Both these functions have a simple implementation: type checks the internal tag of the given value and returns the corresponding string; next finds the given key in the table and then goes to the next key, following the internal table representation. In languages like Java and Smalltalk, reflection must reify concepts like classes, methods, and instance variables. Moreover, that reification demands new concepts like meta- classes (the class of a reified class). Lua needs nothing like that. In Lua, most facilities provided by the Java reflective package come for free: classes and modules are tables, meth- ods are functions. So, Lua does not need any special mech- anism to reify them; they are plain program values. Simi- larly, Lua does not need special mechanisms to build method calls at run time (because functions are first-class values and Lua’s parameter-passing mechanism naturally supports call- ing a function with a variable number of arguments), and it does not need special mechanisms to access a global vari- able or an instance variable given its name (because they are regular table fields).20 7. Retrospect In this section we give a brief critique of Lua’s evolutionary process, discussing what has worked well, what we regret, and what we do not really regret but could have done differ- ently. One thing that has worked really well was the early de- cision (made in Lua 1.0) to have tables as the sole data- structuring mechanism in Lua. Tables have proved to be powerful and efficient. The central role of tables in the lan- guage and in its implementation is one of the main character- 19 Although this code still works, the current idiom is ‘for k,v in pairs(t) do print(k,v) end’. 20 Before Lua 4.0, global variables were stored in a special data structure inside the core, and we provided a nextvar function to traverse it. Since Lua 4.0, global variables are stored in a regular Lua table and nextvar is no longer needed. 2-22
- 52. Other documents randomly have different content
- 53. that she brightened perceptibly at his approach, and was always very willing to undertake any message or errand with him. So she fooled him exquisitely, solacing her wounded pride thus. Whilst he, too great-hearted to pry for petty faults, dowered her lavishly from the generosity of his own noble nature, with all the classic virtues. With what reverent fingers we hang virtues upon the lay figures of our imagination! How we becrown them, and worship them and offer them the incense of our efforts! Yet, it is pleasant pastime, and sanctifying too, for incense purely offered hallows the hand which gives it, perchance more than the God to whom its smokes ascend. All this is well, and though the world gape and wonder at our adorations, what is that to the devotee? Only, to some of us comes the hour when with trembling hands we must undrape our false gods, lay bare their feet of clay to jeering eyes, fold away the rich draperies in which our love has clothed them as a mother folds and hides away the garments her dead child wore, and carry the manikins to the grave. Happy for us if we can bury our dead decently; but bury them never so deep, they rise and walk down the vistas of our happiest hours, infecting their sunshine with the pollution of dead faith. During these long walks together Vashti and Sidney talked much, and of more vital subjects than are generally discussed between young men and women. The fashionable chit-chat about theatres and plays, receptions and fashions was utterly missed from their calendar of subjects. Now and then, Sidney, being a man, could not forbear to let her know how beautiful he found her; but empty compliment, the clipped coin of conversational commerce, he did not offer her; nothing but pure gold minted by her sweet looks in his heart was worthy of her acceptance. Thus they fell back upon the old immortal themes which have been discussed since the world began. They
- 54. looked at life from widely different standpoints, but their conclusions were equally forceful. Vashti Lansing had nothing of the simpering school-girl about her, and none of the fear which makes women reticent sometimes when speech would be golden. It has been said that to know the Bible and Shakespeare is to have a good English vocabulary. Vashti did not know Shakespeare, but she knew her Bible thoroughly. Her speech, unweakened by the modern catch-words which, if expressive, are yet extraneous and dangerous growths, had all the trenchant force of the old Anglo- Saxon, with much in it too of imagery and beauty; for she did not fear to use such metaphors as nature or life suggested. Steeped in the stern Mosaic law, she knew well the stately periods of its prophets. The gentle Christ-creed of forgiveness did not find favour in her sight. “An eye for an eye, a tooth for a tooth” was a judgment which she said only timorous souls feared. She read with grim delight the tales of the kings, with their feet upon their captives’ necks; an evil sympathy with their triumph lighted her eyes with wicked light. What a spouse she would have been for one of these cruel kings! she thought sometimes. And she applied a relentless utilitarian philosophy to life. The weakest go to the wall and the strong triumph. She accepted that with the stoicism which springs from conscious strength, but in her system she rather confused strength with righteousness. She watched the movings of life about her with cold, curious eyes, and yet her philosophy of life was but an expanded egotism. She comprehended only those sets of actions which might have taken place had she given free rein to her own inclinations; she judged of all motives by the repressed impulses of her own bosom. She scrutinized others unsparingly, prying into the most sacred griefs, the most holy joy without shame or remorse, and she did not spare herself more than others. The dim, terrifying impulses and visions which girls put behind them, shudderingly and uncomprehendingly, hiding them away with the other spectres which people the realm of the unknown, until
- 55. such time as life’s meanings shall be expounded in a sacred mystery play of sense and spirit, she marshalled forth into the light of day and considered calmly and cynically. She applied the foot-rule of her own lymphatic temperament to the morals of her fellows and was never disappointed when they fell short. She was well versed in all the wisdom of the Pharisees, and at the sewing circle talked always to the older women, and was never found in the corner where the clear-eyed girls whispered together. And quickening and vitalizing all her existence there was that sense of Power. Power uncomprehended, undeveloped, yet there; and as a thunder-cloud gives premonition of its potent force even before the brand leaps from its cloudy sheath, so Vashti Lansing’s personality was instinct with potentiality. This was the woman Sidney Martin, idealist and dreamer, loved. The days sped swiftly, the present lapsing into the past, the future flying forward with the unique tirelessness of time. How wrong to typify Time with hoary head and tottering limbs. Crowned with the vigour of eternal youth, does he not leap forward triumphantly like the messenger of the gods fresh plumed with flame? Ah, he is not old, but young and swift. Strive if you will to stay his flight for but one single precious instant, stretch forth your hand while yet his wings brush your face, and ere the fingers may close upon his pinions, he is gone, leaving but the largesse of lost days. The harvest was done, the ploughshare and the harrow were tossing the earthy bed for the new grain. Day after day, through the clear air, there came from different points the blowing of the traction engine which dragged the one threshing mill in the section from farm to farm.
- 56. It was the custom of the neighbourhood that the farmers should assist each other with the threshing. Sidney was charmed when he heard this—how idyllic it was this community of helpful effort! To be strictly truthful, this custom had its genesis in less worthy reasons than he imagined, the simple fact being that in the little hide-bound community there were no odd men left unemployed, therefore as labour could not be hired the farmers perforce clubbed their efforts. “I say, girls,” said Lanty, rushing out from his uncle’s big barn to where the two girls and Sidney stood beside the engine, “I say, isn’t that engine exactly like Mrs. Ranger in church?” His face was begrimed with dust, thistle-down rested whitely upon his yellow hair, his blue eyes were alight with hope and happiness and that exaltation which a strong man feels in effort. The girls shook their heads warningly, but laughed. The traction engine, its wheels shackled, puffed and panted with a ludicrous simulation of bottled-up energy, and to the minds of the three young people it was decidedly suggestive of the irate patience expressed in Mrs. Ranger’s attitude when placed in conditions where she could not answer back. Nathan Peck, watching the engine, stored up the saying for Temperance’s delectation, and wished she had come out with the girls. Above the rattle and hum of the threshing mill sounded the hoarse voices of the men shouting jokes at each other—threshing time being always a jovial season. A good or bad harvest meant often life or death to these people; but, having done their best, they could but accept the results. It was a point of honour to accept unflinchingly the verdict of a poor yield, yet many wives could tell of despairing hours when, after their neighbours had departed, husband and wife essayed to reconcile ways and means. Clouds of golden dust, starred here and there by a silver thistle- down, shimmered out of the barn door; there was an aroma of
- 57. crushed straw, a scent of charred wood from the engine fire, a sense of eager, healthy life. The swallows flew agitatedly above the barn, yearning over their clay nests beneath its eaves. “What are you doing?” asked Vashti. “Measuring,” said Lanty. “Uncle said he’d take the bushel for a little though when he saw your petticoats out here——” “Who’s in the mow?” “Ab Ranger is cutting bands, and he’s let my bone-handled pruning knife go through the mill; Tom Shinar is feeding; there’s three on the mow and four on the stack.” “How is it turning out?” “Splendidly, no straw to speak of, but finely headed—like you, Mabella,” he whispered, blushing through the dust. “Come on here, Lanty,” roared a voice from the barn. “You can spark in the noon-spell if you want to.” A laugh followed. Mabella blushed hotly, and as a maiden is expected to do under the circumstances, looked absently into vacancy. “Well, you’ll be too busy eating in the noon-spell to notice,” Lanty called back to the unseen speaker. This, being the retort courteous, was received with applause. “Well, I must go, girls; uncle’s back will be aching by this time toteing that bushel. I hope you’ve made heaps of good things for dinner, we’re all hungry as hunters.” “Trust Temperance for that,” said Sidney. “Yes, indeed,” said Lanty. “Ta-ta, girls.” “Lanty,” said Mabella, “be careful of the belt.”
- 58. “Surely,” he said, his voice softening. The next moment his strong, lithe figure had swung jauntily through the narrow space between the broad whirling belt and the door. “Nathan,” said Mabella, “Temperance wants you to get someone to mind the engine for ten minutes before dinner, so that you can come round and carve the meat.” “I’ll be there,” said Nathan, then he added with an irrepressible and comical self-importance: “Meat ain’t worth puttin’ teeth into if it ain’t cut up proper.” “That’s very true,” said Sidney, who felt a great kindliness in his heart for this patient lover. “Well,” said Mabella briskly, “I’m going round to help set the table.” Having seen Lanty, Mabella wished to get off alone to think over his perfections, which impressed her afresh each time she saw him. “O! can’t you come for a little wander?” asked Sidney of Vashti. “There’s nothing to be done in the house; besides, that imp from the preacher’s is there, and I’m sure she is a host in herself.” “Yes,” said Vashti, her voice more than usually vibrant. “Yes, I will come.” She was very pale. She turned away as Jephthah’s daughter turned from the promise of her bridal bower. For, during these few minutes of idle speech amid the whirr of the threshing mill, Vashti Lansing had taken her final decision. She would marry Sidney Martin; but on her own terms, she added to herself. And then she went with him across the stubble, where the late rains had made a phantom spring of fresh green grass and over-eager weeds, which were putting forth their tender tops only to be a prey to the first sneering frost. Ah, how futile and inconsequent it is to trace laboriously the windings of cause and effect; a touch often sends one over the precipice, and a smile, a sigh or a silence brings us face to face with Fate. Can we by searching find these things?
- 59. And Sidney, too, felt the fateful words trembling upon his lips, a keen envy of personal happiness possessed this man, who so rarely sought his own good. A great longing to stand as Lanty had stood, with the promise of life’s fulfilment at his side. Sidney and the woman beside him walked across the stubble to where a little belt of scrubby oaks followed the course of a ditch between two fields; here and there a vivid red patch against the underwood showed a dogwood bush. Here and there an elm tree sprang up spire-like above the lower oaks. “See,” said Sidney, “that row of elm trees. Can you not fancy that upon just some such day as this the seed was sown? Does it not give a delightful sense of the continuity and endurance of nature’s miracles to think that a gentle wind, such as now stirs their topmost leaves, chased the seed vessel playfully along the ground? The wind laughed then, thinking it was making fine sport of its little playfellow, but see, at every pause a seed was dropped, and like an egotistical king who marks the stages of his journey, the fragile cluster of seed has left its memento. You have seen the seed of the elm tree?” “Yes, it resembles a hop. I suppose the seeds are between the little scales. I can fancy it fluttering along the ground like yon leaf.” “Yes,” he said, delightedly, and then, pleased with her comprehension of his thought, he looked far across the field. After all Mabella had not been in such a hurry to get to the house. She was running up and down like a child with the little brown calves in their special paddock near the house. Her sunbonnet was in her hand, her hair glittered in the sun like ripe wheat. From her Sidney’s eyes turned to Vashti, and his very heart stood still, for dimming the splendour of her eyes two great tears hung between her eyelids. There was no quiver of lip or cheek, no tremour of suppressed sobs; her bosom seemed frozen, so statuesque was her pose. “Vashti!” he said. It was the first time he had called her by name— used thus the one word was eloquent. “Don’t!” she said. “I—will—come—back to the house presently.”
- 60. Sidney, his heart wrung, took his dismissal without further speech. He went a few steps from her, then turning went swiftly back. Her tense attitude had relaxed. She was leaning against the grey bars of the fence, a crimsoned bramble twining round one of the upright supports hung above her as a vivid garland. “Vashti!” he cried, “I can’t leave you like this.” “Not if I wish it?” she asked, and gave him a fleeting smile, beautiful as the opalescent glimmer of the sun through rain. It shook the man to his soul. He stood for a moment blinded by the glamour of her beauty, then left her again. This time he did not look behind, but strode triumphantly across the fields, for he felt that smile had given him definite hope. Sidney, despite his perfections, was only man. For a moment he had forgotten her tears; then remembering, he said to himself that soon he would kiss away all tears from her eyes. The best of men are prone to consider their kisses a panacea for all woman’s ills. Perhaps, with the irrefutable logic of the homœopathists, they argue that what produces an ill will cure it! “Tears, idle tears, I know not what they mean; Tears from the depth of some divine despair Rite in the heart and gather to the eyes In looking on the happy autumn fields, And thinking of the days that are no more.” The lines sprang spontaneously to his lips. This was the secret of Vashti’s tears. How often he had felt that almost intolerable regret, begotten by the recognition of the evanescence of beauty. And Vashti with her splendid natural soul must feel with treble keenness all these things.
- 61. Doubtless to her the crimsoning of the leaves was as the hectic flush upon an ailing child’s cheek to mother eyes. “The days that are no more,” ah! could it be she thought of the days when the grain was growing high, the first days of their companionship? Deluding himself thus with futile fancies he turned slowly, slowly towards the house, arriving to find Vashti already there in the midst of the housewifely bustle. Whilst the visionary Vashti bore him company, the real Vashti had passed him unseen. So it was ever. The real Vashti eluded his vision; her place was filled by a mimic Vashti created of an ideal and his love, and tricked out in all the virtues. At the house every one was busy. The preparations for dinner were approaching a crisis. Temperance, with a look of ineffable importance such as only a managing and forehanded woman can wear upon such an occasion, was cutting pies, piling plates with biscuits, arranging pickles in glass dishes, and between whiles taking flights to the oven, where a huge roast was browning. Mabella was arranging the table with knives and forks; she reckoned up, six or eight times, the number of people to set for, substracted two for the ends, and divided to find how many for each side. Mabella had no head for figures, so she made a mistake in this process; but as the basis of her calculation was wrong the result was correct. An unexpected thing! But Mabella, cheerfully confident in her methods, had no thought of all this; she trotted about the table with the gladness of one who does not save steps. Vashti was bringing chairs out from the other rooms to complement the number in the kitchen; and Sally, the preacher’s handmaiden, was arranging the tin basins with soap and water for the men to wash in, and varying the monotony by tantalizing the chained-up mastiff till he was nearly crazed to get at her, drawing back to his kennel door and launching himself forward with magnificent disregard of the chain which at each attempt jerked him off his feet.
- 62. Sidney leaned against the door-jamb watching the homely scene with just the faintest tinge of proud proprietorship in his eyes when they rested upon Vashti. Presently she came and stood before him. Her figure was so suavely graceful that her most ordinary movements took on an artistic significance. Just now her attitude was that of a queen who fain would ape the serving maid, but who could not cast aside her sovereignty. “Will you sit down with the men?” she asked. “Your father does, doesn’t he?” “Indeed, yes.” “Then I will also.” “Then I’ll wait on you,” she said, and primmed her mouth into a quasi-humble expression. “If you do——” his grey eyes dilated. “Yes.” Just then Nathan came round from the barn. “They’ll be here in ten minutes,” said Vashti, and hurried away. Temperance, flushed with housewifely pride, had the big carving platter ready with the steel beside it. The latter was a concession to appearances, for Temperance always sharpened the knife for Nathan in a peculiar fashion of her own. When Nathan entered she was sharpening it vigorously on the back of the kitchen stove. “Well,” said Nathan, “here I be; where’s the water?” He had seen the basins upon the apple-tree blocks, where they had stood for time out of mind at the Lansing threshings, but he thought Temperance might be prompted to come and get it for him. Temperance paused in the sharpening process, but at that moment a tow-head appeared at the door.
- 63. “Here ’tis, Mr. Peck,” said Sally, “right here under the shade; fresh water, sweet water, well water. Come up, run up, tumble up, anyway t’ get up; here’s where you gits water. Step up, ladies and gents. Everything inside as represented on the banners, and all without money and without price,” concluded Sally, putting a frosting of the parsonage piety upon the vernacular of the Blueberry Alley dime shows. Mabella, Vashti and Sidney laughed. Temperance resumed her knife-sharpening with a click. “That child will come to no good end,” she said to Nathan, when he re-entered. “She won’t,” agreed Nathan with some asperity; his waistcoat and shirt were drenched. He had asked Sally rashly to pour a dipper of water on his head to “rense him off.” Sally complied with alacrity, only she emptied a pailful over his bent head instead of a dipperful. “Drat that young ’un,” said Temperance, enraged at this. “I believe, I really do, that Mrs. Didymus sent her over here to be shet of her for a day, and if this is a sample of her doin’s I don’t know as I blame Mrs. Didymus, but if there’s any more goin’s on I’ll trapse her back quicker.” By this time the roast was out of the oven, and Nathan began his work with the enthusiasm of an artist. Nathan was always greatly in demand when there was any carving to be done, and he was very proud in a candid childish way of his proficiency. Perhaps his practice with the plane and the drawknife stood him in good stead, for certainly Temperance was justified in thinking proudly that no man could carve like her Nat. “They’ve blew,” announced Sally, tumbling into the kitchen in great excitement. This was somewhat unnecessary information as the whistle was making itself perfectly audible; ere its shrill echo died away the men, begrimed and laughing, came round the corner of the barn and were soon spluttering in the basins. Lanty came into the back kitchen, but the voice of one of the men brought him out of his retreat, and in five minutes they were all at
- 64. table. Old Lansing at one end with Sidney at one side. Lanty at the other end with Nathan beside him. “Open the ball, Nat,” said Lanty, passing Nathan the platter. Nathan helped himself with the deprecating modesty of one compelled to pronounce judgment upon his own handiwork; then the platter made the round of the table in pursuit of the one which had started from Mr. Lansing’s end. “Guess you had something to do with this, Nat,” said Ab Ranger. “I know your shavings.” Nathan admitted the impeachment. “Well,” said Sidney, “we can’t beat that in Boston.” And Nathan ate vigorously to hide his embarrassment. The girls flitted about seeing everyone was supplied. Did calm-eyed Vashti know what she did, when she bent over between Sidney and her father ostensibly to remove an empty plate, and let her palm rest as if by chance for a moment on Sidney’s shoulder? Did ever electricity shoot and tingle through the veins like that touch? He watched her as she passed serenely along the other side of the table, and longed for the moment when he might have speech with her. Temperance poured the tea and coffee in the back kitchen. Sally performed prodigies in carrying it to the table, and grimacing, as she set it down, behind the unconscious backs of the recipients. Sidney won golden opinions at this dinner by his frank friendliness. “He ain’t big feelin’, that’s one thing,” the men said to one another as they swaggered out to rest the noon-spell under the trees. Lanty and Sidney with great affectation of helpfulness asked the girls to stand aside and watch them clear the table. Temperance was not to be seen, they would surprise her when she arrived. They succeeded beyond their expectations.
- 65. “It isn’t such a job to clear a table as you’d think,” said Sidney complacently to Lanty. “No, ’tain’t for a fact. I’ve seen girls take half an hour at it.” The two young men had cleared the table by removing the dishes and débris indiscriminately and depositing them upon the table in the back kitchen. When Temperance returned from a little chat with Nathan beside the smoke house, she eyed the chaos upon the table wrathfully. “Laws!” she said. “Of all the messes! Lanty Lansing, ain’t you ashamed to be so redecklus? And them girls standin’ gawkin’ and laughin’! As for you,” eyeing Sidney severely, “I should ha’ thought you’d more sense, but blessed is them that has no expectations! Lanty! Are you or are you not feedin’ that brute with good roast? Where’s the cold meat fer supper to come from, I’d like to know?” No one volunteered a response till suddenly Sally piped forth in her thin reedy voice: “Take no heed for the morrow what ye shall eat, or——” “You blasphemous brat!” said Temperance, her wrath diverted to another channel. Sally subsided into silent contemplation of the dish of pickled beets from which she was helping herself with pink-stained fingers. Temperance was not Mrs. Didymus, and Sally in many combats in Blueberry Alley had learned to gauge her antagonists. The offended Miss Tribbey left the back kitchen in indignant silence and set about arranging the table for her own and the girls’ dinner, murmuring to herself meanwhile a monologue of which such words as “messes,” “sinful,” “waste,” and “want o’ sense,” were distinctly audible. “I don’t believe that was really an unqualified success,” said Sidney to Lanty.
- 66. “No,” said Lanty, “I don’t believe it was. What did you mix everything up for?” “How did I know they were to be separated? What did you feed the dog with the roast for?” “Did you ever see such an imp as that Sally?” “Never,” said Sidney. “But Temperance squelched her!” “She did,” said Lanty. “I say, wasn’t she ripping?” Meanwhile Temperance’s short-lived wrath had died away, and she was pressing food upon Sally in quantities calculated to appal any but a Blueberry Alley child. Temperance rose in the midst of her second cup of tea, and, going up stairs, came down with a large fresh bandana handkerchief. She went out to where Lanty and Sidney stood talking. “Here’s the handkerchief you wanted to keep the dust out of your back,” she said with ill-assumed hauteur. Lanty took it with laughing penitence on his face. “I say, Aunty,” he said, “would you ask Mabella to put it on?” Miss Tribbey’s severity relaxed; a vain-glorious satisfaction stole over her face in a smirk. To have Lanty call her Aunty! Certainly Lanty Lansing “had a way” with women that was well- nigh irresistible. “Yes,” she said, then with comical apology, she addressed herself to Sidney. “Them children is a most tormented trouble, ’specially when they meddle with things they don’t know nothing about.” “That’s so,” agreed Sidney with emphasis, and Temperance, highly delighted with her Parthian shot at him, departed. And presently Mabella came to the door, a riante little figure, and demanded with mutinous affectation of indifference: “Did any one want me?”
- 67. “Yes, badly,” said Sidney, and took himself off to the garden, laughing. “That’s true,” said Lanty. “I did want you badly.” Her eyes were wavering beneath his masterful regard, but she said—“Oh, you did want me! Don’t you now?” The words were brave, but her eyes fell. “Mabella,” he said—silence. “Mabella, look at me.” Slowly she raised her eyes and crimsoned. “Do you know now?” he asked lovingly. “Ah, what a wicked teasing bird it is when its wings are free, but after all they are gone to the barn and——” he advanced a step. “Lanty!” said Mabella, and in an instant he was grave. “Dear girl,” he said, “you don’t think I would do anything to make you feel badly?” The warning shriek of the whistle came to them. “See, tie this round my neck, will you?” She folded it with an adorable air of anxiety and precision, and stood on tiptoe to lay it on his shoulders and again on tiptoe to knot it under his chin, a process Lanty rendered arduous by putting down his chin and imprisoning her hands, a performance he found most satisfying. But at length he was off, and Mabella watched him round the corner of the barn, and then went indoors to attack the chaos upon the table with a good heart. “Where’s Vashti?” she asked. “Spooning her young man in the garding,” said Sally, emerging from her shell. “Of all the impses I ever see!” ejaculated Temperance. “G’long and fetch in some wood.” Sally departed. “Vashti’s in the garden peeling apples for supper,” continued Temperance to Mabella, with an attempt at unconsciousness. Mabella gave her a hug.
- 68. “It’s a sugar plum for Mr. Martin because you were bad to him, isn’t it?” “Yes, Lanty’s had his——” Mabella blushed, and an irrepressible ripple of laughter broke from her. “Well, you needn’t laugh,” said Temperance. “Mr. Martin thinks Vashti’s just about right. Well, there’s no accountin’ for taste. ‘Everyone to their taste,’ as the old woman said when she kissed her cow.” “Temperance!” said Mabella, “you don’t mean——” Temperance nodded oracularly, “Nathan thinks so too.” “Well!” said Mabella, and relapsed into silence. Here was news for Lanty. If Nathan and Temperance thought so it must be so. A fellow feeling not only makes us kind but often very acute; and in all Dole there were no such keen eyes for any “goin’s on” (as courtship was disrespectfully designated) as those of Temperance and Nathan. “Love, it is a funny thing; It puzzles the young and the old; It’s much like a dish of boarding-house hash, And many a man gets sold.” Sally’s falsetto voiced this choice ditty with unction, as she entered with an enormous load of wood in her thin arms. She deposited the wood with a bang. “Sakes!” said Temperance. “I wonder if she sings them songs to the preacher?” Whereupon Sally, in vindication of her judgment, began a lugubrious hymn. “Stop it,” said Temperance. Sally stopped.
- 69. Beneath the trees Vashti peeled her apples busily, the narrow parings of the greenings twined about her white wrist, the thin slices fell with little splashes into the bowl of water which was to prevent them turning brown before being cooked. Miss Tribbey’s apple-sauce was always like white foam. A voyaging wasp came, and settling upon the cores was very soon drunk, so that he was an easy prey to a half dozen ants which wandered by that way. The distant buzz of the threshing mill filled the air with a drowsy murmur as if thousands of bees hummed above a myriad flowers, here and there a thistle- down floated glistening in the sun. The scent of the overblown flowers mingled with the odour of the apples. “Are we done now?” asked Sidney, as she laid down the knife. “We are,” she said with meaning emphasis. “Do you feel very tired after your exertions?” “Not so tired as you’d imagine,” said Sidney. “The truth is I couldn’t bring myself to offer my services, for if you had accepted them I would have had to look at the apples instead of at you, and I did not have strength to make the sacrifice.” “Could you make sacrifices?” she asked. “Try me,” he half whispered. There was a tense moment. Mabella’s voice came ringing from the house, the whirr of the threshing mill suddenly seemed near at hand, and through it there came Lanty’s voice shouting some directions to the men on the stack. “Perhaps I may some day,” she said. “You know,” he said, his voice enchaining her attention even as she strove with bitter thought, “you know you will have the opportunity to ask anything, everything of me.” “Ah, how should I know?” she said, as one who has not deigned to observe too much. Sally, sent out for the apples, appeared round the corner of the house.
- 70. “Promise me,” said Sidney, “that you will come for a walk after supper; promise.” For an instant the boulders of Mullein meadow and the dimness of the twilight sky blotted out the crimson of the Virginia creeper on the porch which flamed in the sun. “I will come,” she said. “Ah——” he said no more. “Sorry t’interrupt,” said Sally genially, as she stood beside them. “But painful as the duty is it must be did; but don’t mind me, I’m blind in one eye and can’t see out of the other.” “Sally,” said Sidney very gently, “you talk too much.” For the first time in her life Sally blushed, and gathering up the apples and the parings departed abashed. “You are not going in?” he said rising as Vashti stood up. She held up her hands. “I must wash my hands,” she said, “and I want to rest a little.” The slightest hint of fatigue or illness in the splendid creature before him always touched him strangely. It was like a sudden assertion of the human in something divine. “Do,” he said; “and Vashti,” using her name with happy boldness, “you won’t forget your promise.” “I never forget,” she said, simply and sweetly. He stood bareheaded watching while she entered. Then looking about, he suddenly noticed that in the garden the summer flowers were overspent, the little battalion of ants tugged viciously at their victim, whose yellow and black had shone so gallantly in the sunlight as he lighted down to sip the apple juice. The whirr of the threshing machine made melancholy cadences which sighed through the trees; and all at once the whole scene darkened.
- 71. It was only that the sun had dipped beyond the house, and the crimson Virginia creeper seemed in the shadow to be more brown than red, two or three of its leaves fell desolately to the earth, as dreams die when hope is withdrawn. And Sidney, with the fatuity of lovers, said, “She has taken the glow with her.” But the torch which lighted Vashti Lansing’s way was not filched from flowers and sunshine, but shone fed with the evil oils of anger and revenge, baulked will and disappointed love.
- 72. CHAPTER VII. The grey of twilight was paling the gold of the after-glow. A quiet hush had fallen upon the earth—rather intensified than disturbed by the lowing of far-away cattle. It was the quiet of raptured anticipation, as if great hands held the earth up to the baptismal font of the heavens to receive the chrism of night; and the earth, like a wise and reverent child, waited with hushed heart-beats for the benediction. Sidney Martin waited in the porch for Vashti to keep her tryst, and presently he heard her footsteps. The echo of each step gathered in his heart, dilating it with happiness as an already full glass is brimmed above the brink by drop after drop. From his position, where he stood spellbound, he commanded an angled vista of the stairs, and slowly she descended within his range of vision; first the beautiful foot, proportioned so perfectly to the body it bore, then the long exquisite lines from heel to hip, and the yet more exquisite curve from hip to shoulder, and the melting graduation of breast to throat, and then the perfect face of her. She paused for a moment upon the last step, as if loath to step out of her pure rarefied atmosphere of maidenhood into the air vibrant with the sobs and sighs, the hopes and despairs, the gains and losses of human life; and standing thus, for one fleeting second there rose before Vashti a vision of renunciation. She saw herself, lonely but clad in righteousness, going on her way; but the next instant the austere dream vanished, brushed aside by a hateful, sneering cynicism. With a heart full of self-mockery, more evil than her evil intent, Vashti took the step to Sidney’s side, and stood there the typification, as he thought, of gentle dignity and dignified womanhood. “How good you are,” he said gently.
- 73. They took the way almost in silence. She wondered vaguely where he would take her, to the far-away pastures, the little knolls nestling upon the hills which he loved, or to the oak trees where they had talked in the morning. When they reached the road she submitted her steps to his guidance with outward meekness and inward indifference. He turned away from Dole. It was to be the far-away pastures then—as well there as anywhere. But he had passed the gate! And then it dawned upon her. He was taking her to Mullein meadow! Her indifference fell from her like a rent garment, bitter remembrance tore at her heart. How dare he bring her here and bid her masquerade amid these grey boulders where she had known such agony! She imagined those implacable rocks rejoicing in her humiliation. Were not her own curses yet hissing across the eerie barrenness of this wide waste field? Ah, even so, Vashti—if our curses do not seek us out we ourselves return to their realm; there is great affinity between a curse and the lips which utter it. The flame of her resentment fluttered to her cheeks, giving them an unwonted touch of rose. As they reached the entrance to Mullein meadow, she half stumbled; she recovered herself quickly, Sidney’s swift touch being hardly needed to restore her poise. To Sidney, her silence, the strange, sweet colour in her cheeks, her uncertain step, pointed but to one thing—the natural agitation of a girl about to have a man’s love laid at her feet. Surely never man was so exquisitely befooled as this one? He took the path straight for the little spot where that happy betrothal had taken place. Vashti hesitated—this was too much. “I—,” she opened her lips to speak, but the words died away, unmerciful resolution freezing them at their source. “Come,” urged Sidney with tender insistence, and with an appearance of sweet submission she yielded, and at length they stood where those others had stood. The same grey sky bent above them, the same quiet hush brooded over the desolate reaches, the
- 74. same clear star hung scintillant in the sky, and Sidney, taking her hands, which trembled by reason of the terrible restraint she was putting upon her anger, began to speak—very gently, but with an intensity which made his words instinct with life and love. “You know,” he said, “why I have asked you to come out to-night, but you cannot know why I have brought you here to this spot? It is because it is a place of happy auguries. Here, not knowing whither I strayed, I came upon the betrothal of Lanty and Mabella. Here, heartsick with envy of their happiness I turned away to face the desolate greyness of the twilight. Here I saw a star, one lone star in the grey, which seemed to promise hope, and in my heart I named it Vashti. See—there it is, but more golden now, more full of beneficent promise, burdened, as it seems to me, with gracious benediction. Oh, Vashti, when I left those two in the solitude of their happiness you cannot dream how my heart cried for you. All the way home nature’s voices whispered in my ear ‘Vashti—Vashti,’ and my heart responded ‘Vashti,’ and it seemed to me that there was no other word in all the universe, for in it were bound all meanings. It seemed to me there was no other idea worth comprehending but the identity behind that word. Vashti, say that you love me—that you will marry me. Here, where my heart knew its bitterest longing, satisfy it with one syllable of your voice. Let me also build tabernacles here as the holy place where happiness descended upon me”; he let fall her hands. “Vashti, you know that I love you; give me your hands in symbol of yourself as a free gift.” He held out his hands. Slowly, gently, trustingly, as a woman who knows well what she does, and will abide by it, Vashti Lansing laid her hands in his. His vibrant, slender fingers closed upon them. There was an instant’s pause—— “You love me!” he cried, as one, after a long novitiate, might hail the goddess unveiled at last. Then drawing her to him he kissed her on the mouth, and from that moment was hers—body—and yet more terrible bondage—mind; and she, with an astute and evil
- 75. wisdom, forbore to make any conditions, any demands, till he had tasted the sweets of her acquiescence. Would any man give her up, having held her in his arms, having touched her lips? With shameless candour she told herself, No. So she rested her head upon his shoulder, whilst he whispered in her ear the divine incoherences of love, and intoxicated with the charm of the woman in his arms, touched the white throat by the ear, where a curl of dark hair coiled like a soft, sweet shadow. A long, contented, yet questioning sigh came to him— “Tell me?” he said. “You will let me live always in Dole?” she said. “Always—always, dear one! In Dole or anywhere else you like.” “Ah!” she said in a tone of dreamy happiness—“you will take old Mr. Didymus’s place; we will live in the parsonage; what a happy life we will have!” “Vashti!” said Sidney, almost reeling before the shock of her words. As a beautiful white mist rolls back to show some scene of sordid misery, so the glamour of the last few weeks lifted, and displayed vividly to Sidney all the awkwardness of the position which he had created for himself. Ever since that day, when stung by Sally’s impertinent words he had agonized alone upon the hillside, nothing whatever had transpired to awaken its memory. A deference rather more pronounced than necessary upon the part of the village- folk, a certain constraint upon the part of the young men, had been the only visible signs that Dole remembered. But upon the other hand nothing had occurred which gave him the opportunity of explaining to Vashti, nor, indeed, had he ever been able to decide how he could explain to her, even if given the opening. He had gone to church with the Lansings Sunday after Sunday. Under the circumstances any other course would have been an insult to the régime of the house in which he was staying. He had found nothing in the little church which jarred upon his tastes or revolted his principles. The simple, pious sermons of grey-haired Mr. Didymus
- 76. were entirely inoffensive to anyone not of malice prepense irritable. The sad experiences of his long life had mitigated his judgments. The man who in his fiery youth scoffed at death-bed repentances now spoke feelingly of the thief on the cross; the elect murmured among themselves that Mr. Didymus was “growing old and slack.” Certainly his sermons were not learned, but neither were they devoid of a certain eloquence, for the old man knew his Bible by heart, and above all, they were free from the anecdotal inanity; it would never have occurred to the old, plain-spoken man to stand in his pulpit telling his people tales suitable for the comprehension of three-year-old children. There was, perhaps, the merest trace of asceticism in Sidney Martin’s nature, and the simple doctrine of these people, their fatalistic creed, their bare little church, appealed to him as no gorgeous ritual or ornate sanctuary could have done. The hoarse, untuneful singing of these country folk, taking no shame of their poor performance, so that it was in praise of God, stirred his spiritual sympathies more profoundly than any cathedral organ—yet —he was a creature of reason, and he had always considered the Catholic Church more logical than any other, and above all, he had no belief whatever in the Christian doctrine. Ruled by a pure and lofty ideal of Truth, his life had been ideally good. His lofty aspirations did not lift him beyond sympathy with his fellows, only above their vileness. He adored nature with an almost heathenish idolatry, and had such reverence for her slightest manifestation, that he never willingly broke a leaf or crushed an insect. Literally, he worshipped the works, but not the Creator. And lo!—here was the woman round whom his very soul twined, taking it for granted that he believed all she did, and that his life could compass no higher happiness than to preach this belief to others; and what excellent grounds he had given her for thinking thus! All these things mirrored themselves in his mind in an instant, then he said: “But, Vashti, I have no need to do anything. There are many worthier men than I to fill Mr. Didymus’s place. I am not a preacher, you know.”
- 77. “Oh, but you will be for my sake,” she said, and laid her head down again upon his shoulder like a child who has found rest. Truly there are more tempting devils than the urbane gentleman of the cloven hoofs. “What had you meant to do?” she asked. “Indeed, I had mapped out no definite course,” he answered. “My mother’s money makes life easy for me, you know, but I had meant to do something, certainly. Only I was taking my time looking about. I didn’t want to do anything which would cut some fellow who needed it out of a living.” “Let me decide for you,” she murmured; the breath of the words was warm on his ear. “Think how happy you could make us all. They all think so much of you in Dole on account of your prayer. Mary Shinar says you are a saint.” Then, her arms stealing about his neck, she added, “Sidney, for my sake you said you would sacrifice anything. I didn’t think this would be a sacrifice. I thought it would be a delight; but if it is a sacrifice make it for my sake.” Alas, he had fallen among the toils! He took swift illogical thought with himself. He would preach to them a pure and exalted morality. He would be the apostle of nature’s pure creed. He would make Dole a proverb in all New England. He would teach, he would have a library, he would marry Vashti. Glamoured by his love and his sophistry, his judgment, his sense of right and wrong, failed him. Sidney caught his Delilah to his heart. “It shall be as you wish, my sweet,” he said; “and now tell me you love me.” “I love you,” she said, repressing the triumph in her voice. “I love you and I am proud of you,” she said again, holding her head high. If she had lost much in Mullein meadow she had also gained a triumph there. The short American twilight was darkening to night. The weird old boulders sentinelled round them might have been a druidical circle,
- 78. and she the priestess fulfilling the rites. Nor was the victim wanting; only instead of slaying the body with a golden knife she had killed the soul with silvery speech. “Ah,” said Sidney as they turned to thread their way out of Mullein meadow, “surely this place is holy.” She paused, looking at him—“Do you not think that suffering sanctifies more than joy?” she asked. “No, not such joy as ours, as Lanty’s and Mabella’s.” “I don’t know,” she said. “But I’m sure of it!” he answered; then with a lover’s fantastical fondness he went on, “I would not be surprised if when we visited this spot again we found it hedged in by lilies, tall white eucharist lilies, set to keep others from straying into consecrated ground.” “Sidney,” she said, “promise that you will never, never ask me to come here again—it is too sacred.” He was deeply touched by her delicate, sensitive thought. “Dear heart,” he said, “never; yet do not the most reverent lips approach the sacramental cup more than once?” “You will make a capital preacher,” she said, “but you must not persuade people to do things against their conscience.” “You shall do as you like always.” They were on the highway by this time; a waggon overtook them, and then went on at a foot pace just in advance. Vashti seemed to walk with intentional swiftness. “Vashti,” he whispered, “don’t walk so fast. Let those people get out of sight.” “We must go on,” she said. Sidney thought this touch of shyness adorable in her who was so self-poised, yet he protested with zeal. Do men always try to destroy
- 79. what they admire? Suddenly Vashti bethought herself that an extra rivet was never amiss when one wanted bonds to hold, so with a sigh as of timorous yielding, she gave him her lips again in the shadow of the porch, and left him with a glory of happiness bedimming his mental vision. The house was dim-lit and silent. After the labours of threshing- day everyone was worn out. Lights glimmered in the bedrooms, but the living rooms were dark. Sidney paced up and down the little garden path for long, feeling “caught up to heaven, yet strangely knit to earth.” Vashti sought her room, and pulling up the blind looked out where Mullein meadow lay. “A holy place!” she said to herself. “I wish I could pile the fire to burn all three of them. ‘A tabernacle,’ he said; I wish I might build me an altar there and slay them on it! I don’t think even an angel would stay my hand. ‘A sacrament’; I wish I had the filling of their cups, wormwood should they drink and the waters of Marah down to the very dregs—all three!” Her nostrils dilated like a brute’s upon traces of the prey. In the breast of such a woman love denied turns to gall. She paced up and down, up and down—her rage lent expression in grotesque gestures and evil words, words which with Vashti Lansing’s teaching and training she was superbly brave to use. It grew very late; her eyes were almost wild. She took the guttering candle in one hand and crept along the passage to Mabella’s room. She opened the door and went in. Mabella lay asleep, her candid face budding from the prim little frill like a flower from its calyx. Vashti bent above her a haggard and violent face distorted by passion. Her eyes blazed; her lips drawn tensely back showed the strong white teeth. She leaned over the sleeper, her strong fingers closing and unclosing; a long tress of her hair fell across her shoulder suddenly and touched the dreamer’s cheek—Mabella stirred, raised her hand half way to her cheek, murmured with a little happy smile—“Lanty—Lan—” her voice died
- 80. away; her soft regular breathing continued unbroken. At the sound of that name uttered thus a dreadful purpose lighted Vashti’s eyes. The fingers of her strong hand opened wide and advanced themselves toward the white throat which pulsed upon the pillow; at that moment the guttering candle fell over. Its burning wick and melted grease struck the hand which held it. Vashti instinctively uttered a smothered cry and jerked her hand; the light went out. Mabella stirred; Vashti sped to her room and got the door closed just as Temperance came to her door and said: “Did anyone call?” There was no response. “Are you all right, Mabella?” she said, going across the hall to Mabella’s door. “Yes,” said Mabella sleepily. “I think I knocked something over with my elbow and the noise woke me up.” “Are you all right, Vashti?” “Yes, what is it?” answered Vashti. “Nothin’—thought I heard a noise.” For hours Vashti Lansing lay and trembled with the only fear she knew; the fear of herself. How near she had been to terrible crime, only she and Omnipotence could know. She reflected upon consequences and told herself that never again would she give herself such an opportunity. At last she sank to rest, to be tormented till dawn by a strange vision. It seemed to her she stood again in Mullein meadow, within the circle of boulders, and that slowly, slowly they closed in upon her; closer and closer they came, narrowing about her with gradual but horrible certainty, and at last they touched her and held her tight, shackling her hand and foot so that she could not move a muscle, but they did not kill her; and whilst she was thus held all Dole defiled before her; the villagers pointed at her with scornful fingers and passed whispering on; her mother, who had long been dead,
- 81. passed with her father, but they did not look at her, nor seem to know she was there, nor did old Mr. and Mrs. Didymus who presently joined her father and mother. Then the scene grew brighter and she saw Temperance and Nathan together; they shook their heads, looking at her sadly but coldly; then a sweeter radiance flooded the view upon which she looked, and Mabella and Lanty with their little children about them drew nigh her, and they spoke kindly words to her, and put a shade over her head to keep off the sun’s heat, and raised a cup to her lips, and one of their children came and held up a child’s haphazard bouquet to her nostrils that she might smell the flowers. She tried to repulse these kindnesses; she tried to drive Mabella and Lanty away with evil words, but the stones pressed too tightly upon her to admit of speech, and while she writhed thus impotently, she looked far away where one wandered alone; there were butterflies and birds about him, and flowers springing about his feet, and he wore a look of calm ineffable happiness, and, yet, it was not the same happiness as shone upon the faces of Lanty and Mabella which lighted the eyes of this visioned Sidney. But in her dream Vashti did not dwell long upon this, her thoughts reverting to the paralyzing prison which encompassed her; and she fought, and struggled, and strove, yet she could not move those terrible stones, and casting her eyes down upon herself, it gradually dawned upon her that she could not even struggle. The terrible wrenches and efforts she had made were but imaginary, so tightly was she held that she could not so much as twitch a finger. Thus the hours passed with her. Mabella slept sweetly and healthfully, so rapt in love that even the baleful influence bending over her so terribly in the night had had no power to disturb her rest, although the gaze of even a friendly pair of eyes so often murders sleep. Sidney slept also, and high above the pale wastes of Mullein meadow, the star of promise still shone, unrecking of the presumptuous human heart which had dared to dream its silvery splendour a pennon of hope.
- 82. CHAPTER VIII. When Sidney opened his eyes next day it was upon a transfigured world that he looked. A world golden with imaginings of happiness across whose vistas shone a white path, like the milky way in the heavens, marking the life road to be trodden by Vashti and himself. Cradled in a happy trance his heart knew no apprehensions. At such a time retrospect shares the mind almost equally with anticipation. The glorious present is made still more glorious by comparison. As Sidney dwelt upon his past it was borne in upon him with peculiar force that it had been but a curtain raiser to the real drama of his life. He had been a popular man as a student and afterwards also, but it seemed strange even to himself how few real ties he would have to sever in adopting this new life—so radically different in vocation from any he ever dreamed of before. The fact was that in all his friendships he had given more than he had received. He had given liberally of that intangible vital capital called sympathy, and he had received but little in return. Although he had not realized it his friendships had been only so many drains upon his vitality. He had thought of, and for, his friends continually; they had accorded him the tribute of uncomprehending admiration which bears the same relation to real sympathy as bran does to the full, rich wheat. Thus it was that in separating himself from these friendships he felt no wrench. Separate from them he must. He knew that the keeping of his promise to Vashti was utterly incompatible with his old life; he must “come out and be separate” from all his old associates and associations. He felt, however, that this would be possible; possible without sacrilege. His attitude towards religion had always been defensive rather than offensive. He felt deeply the pathos of the Christ drama. The figure of the Man of Sorrows was a familiar one in the gallery of mental portraits to which this idealist had turned in time of trial for strength.
- 83. There was one man whose verdict upon his action he longed to know, yet dreaded to ask. A strong soul, untamed by sect, unshackled by formulated belief. A man whose magnificent active human organism was hallowed by the silver thread of mysticism. A man whose splendid logical mind was transcended by a subtle sense of premonition, intuition, which led him far beyond where his reason or his scanty learning could bear him company. A man whose eyes looked out wistfully yet eagerly from beneath penthouse eyebrows. A man whose toil-roughened fingers turned reverently the pages of books he could not read; French or German books beyond his ken. A man in whose proper person Sidney had always felt there was symbolled forth the half blind, half perceptive struggle of the human to comprehend the infinite. What would this man think of his new vows? This man who would have died for what the world called his disbeliefs. Well, Sidney told himself that his first devoir was to Vashti and the promise made to her. He would not delay. These thoughts bore him company till he was in the hall. He did not know the hour, but suddenly he was aware of a subtle, penetrating freshness in the air. He looked out of the hall door: the garden was dim with autumnal dew. Was it indeed so very early? He heard voices in the kitchen. He found there only Mr. Lansing and Miss Tribbey. “Is it so early?” he asked, smiling. “For the land’s sakes! Mr. Martin!” said Temperance. “Is that you?” Sidney laughed aloud; there was a ring in his voice which made Temperance regard him. “I have been awake for ages,” he said; “so here I am.” Temperance remembered certain days in the past when she had been wont to awaken ere the first bird sang in the dark. These were the days when Nathan, a hobbledehoy, too bashful to woo her in
- 84. Welcome to Our Bookstore - The Ultimate Destination for Book Lovers Are you passionate about books and eager to explore new worlds of knowledge? At our website, we offer a vast collection of books that cater to every interest and age group. From classic literature to specialized publications, self-help books, and children’s stories, we have it all! Each book is a gateway to new adventures, helping you expand your knowledge and nourish your soul Experience Convenient and Enjoyable Book Shopping Our website is more than just an online bookstore—it’s a bridge connecting readers to the timeless values of culture and wisdom. With a sleek and user-friendly interface and a smart search system, you can find your favorite books quickly and easily. Enjoy special promotions, fast home delivery, and a seamless shopping experience that saves you time and enhances your love for reading. Let us accompany you on the journey of exploring knowledge and personal growth! ebookgate.com