How sqlite works

How SQL Database Engines Work
D. Richard Hipp
OpenSQL
2008-11-15

How SQL Database Engines Work
D. Richard Hipp
OpenSQL
2008-11-15
SQLite
with occasional comments on how the
operation of SQLite compares to other
SQL database engines.

Why Do You Care?
So you can better understand what your query is
doing.
So that you can appreciate how much work the
database is saving you.
So that you can fix things when they go wrong.
So that you can write better SQL that runs faster
and uses less memory and disk space.

Key Concept
SQL is a peculiar programming language
Each SQL statement is a separate program
SQL describes what instead of how
An RDBMS consists of...
Compiler to translate SQL into procedures
Virtual Machine to evaluate the procedures

Example
SELECT * FROM table1;
Translates into:
Open database file containing table1
Rewind the file
while not at end-of-file
read all columns out of current record
return the columns to the caller
advance file to the next record
end-while
close the file

Imagine what this translates into:
SELECT eqptid, enclosureid
FROM eqpt
WHERE typeid IN (
SELECT typeid FROM typespec
WHERE attrid=(
SELECT attrid FROM attribute
WHERE name='detect_autoactuate'
)
AND value=1
INTERSECT
SELECT typeid FROM typespec
WHERE attrid=(
SELECT attrid FROM attribute
WHERE name='algorithm'
)
AND value IN ('sensor','wetbulb')
)

The Whole Point Of SQL...
A few lines of SQL generates the equivalent of
hundreds of lines of procedural code.
By adding an index, entirely new procedures are
used without recoding.
The SQL Query Optimizer is charged with picking
the algorithm
so that the application developer doesn't have to

Ins and Outs of SQL
Compile SQL
into a program
Run the
programSQL Prep'ed
Stmt
Result
Front Half Back Half

Ins and Outs of SQL
Compile SQL
into a program
Run the
programSQL Prep'ed
Stmt
Result
Byte code
Tree of structures
Native machine code
See Aho & Ullman
The “Dragon Book”
Virtual Machine

Architecture Of SQLite
Virtual Machine
B-Tree
Pager
OS Interface
Parser
Tokenizer
Code Generator

Parser
Tokenizer
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface

Virtual Machine
B-Tree
Pager
OS Interface
Compile SQL
into a program
Run the
programSQL
Prep'ed
Stmt
Result
Parser
Tokenizer
Code Generator

Parser
Tokenizer
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Byte code interpreter
Big switch statement
inside a for loop.
Other engines walk a
tree of structures
Similar to JVM or Parrot
or P-Code

Parser
Tokenizer
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Ordered key/value pairs
with unique keys
O(logN) insert, seek,
and delete
O(1) next and previous

Parser
Tokenizer
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Atomic commit and
rollback
Uniform size pages
numbered from 1
No interpretation of
page content
Cache

Parser
Tokenizer
Code Generator
Virtual Machine
B-Tree
Pager
OS Interface
Platform-specific
interface to the OS
Run-time changeable
Portability layer

CREATE TABLE tab(
Fruit TEXT,
GrownIn TEXT,
UnitPrice REAL
);

CREATE TABLE tab(
Fruit TEXT,
GrownIn TEXT,
UnitPrice REAL
);
Key Data

SELECT unitprice FROM tab WHERE fruit='Peach'

SELECT unitprice FROM tab WHERE rowid=93

CREATE INDEX idx1 ON tab(fruit)
idx1 tab

1:1

+

SELECT unitprice FROM tab
WHERE fruit='Orange'
AND grownin='CA'

CREATE INDEX idx2 ON tab(grownin)
idx2 tab

CREATE INDEX idx3 ON tab(fruit, grownin)
idx3 tab

CREATE INDEX idx2 ON tab(fruit, grownin)

AND grownin='CA'
4 ways to do this query
(so far...)

AND grownin='CA' 1

AND grownin='CA' 2

AND grownin='CA' 3

AND grownin='CA' 4

CREATE INDEX idx4 ON tab(fruit, grownin, unitprice)
idx4 tab

AND grownin='CA' 5

SELECT grownin FROM tab
AND unitprice<1.00

SELECT * FROM tab ORDER BY fruit
sorter

SELECT * FROM tab ORDER BY rowid

SELECT * FROM tab ORDER BY rowid DESC

SELECT * FROM tab ORDER BY fruit

SELECT * FROM tab ORDER BY fruit DESC

SELECT * FROM tab ORDER BY fruit, grownin

SELECT * FROM tab ORDER BY fruit, grownin, unitprice

SELECT * FROM tab ORDER BY fruit DESC, grownin DESC

SELECT * FROM tab
ORDER BY fruit DESC, grownin DESC, unitprice DESC

SELECT * FROM tab ORDER BY fruit DESC, grownin

SELECT * FROM tab ORDER BY fruit, unitprice

SELECT * FROM tab ORDER BY grownin

SELECT grownin, unitprice FROM tab
ORDER BY grownin

Index Column Order
SELECT x, y, z FROM t1 WHERE w=5 AND x=6 ORDER BY x, y;
CREATE INDEX t1i1 ON t1(w,x,y,z);

Index Column Order
WHERE clause terms first

Index Column Order
ORDER BY terms second.
No gaps!
Can overlap WHERE clause.

Index Column Order
Result set columns can appear
anywhere in the index. Gaps allowed

OR grownin='CA'
Pes ky O R

OR grownin='CA'

OR grownin='CA'
UNION

OR grownin='CA'
UNION
Sorter

AND grownin='CA'
INTERSECT
6

WHERE (fruit='Orange' AND grownin='CA')
OR grownin='NC'
INTERSECT
UNION

WHERE (fruit='Orange' AND grownin='CA')
OR grownin='NC'
UNION
INTERSECT
grownin='NC'
grownin='CA'
fruit='Orange'
AND
OR

Remember This:
SQL is a programming language
There is a compiler
There is a virtual machine
Indices are not magic – they need to make sense
Avoid too many indices
The more options the optimizer has, the more
chances it has to make a poor choice.

How sqlite works

More Related Content

What's hot (19)

Similar to How sqlite works (20)

Recently uploaded (20)

How sqlite works