My experience with embedding PostgreSQL

My
experience
with
embedding

PostgreSQL

Postgres
Conf
EU,
Dublin,
Ireland

October
2013

Jignesh
Shah,

Sr
Manager,
VMware

© 2013 VMware, Inc. All rights reserved

About
me

•  Sr
Manager,
VMware

Data

–  Leads
Postgres
team
at
VMware

–  Blog
at
hKp://jkshah.blogspot.com

•  Previously
Staﬀ
Engineer,
Sun
Microsystems

–  Team
Member
of
the
ﬁrst
published
SpecJAppServer

2004
benchmark
with
PostgreSQL

–  Performance
of
PostgreSQL
on
Solaris/Sun
systems

•  Working
with
PostgreSQL
circa
2005


Agenda

•  Business
reasons
for
embedding
database

•  Technical
architecture
journey


The
IniXal
Problem

•  >30

#
products
and
product
components

•  >20

requires

DB

•  Customers
have
to
get
their
own
DB
license

•  Increases
the
Total
Cost
of
Deployment

–  As
database
load
increases,
CPU
cores
required

increases,
so
does
the
total
license
cost
of
DB


SQL
Server
Express

•  Currently
shipped
for
some
Windows
based

products

–  However
limits
not
enough

–  Max
10GB
database
usage

•  Standard
and
Enterprise
EdiXons
get

expensive
preKy
quickly

•  Not
Available
on
Linux


Proposal:
PostgreSQL?

•  PostgreSQL
is
free
and
open
sourced

•  Long
history
of
stability

•  Friendly
BSD
License

•  Plus
I
am
at
a
Postgres
Conference
J


IniXal
External
Feedback

•  Actually
got
pushback
from
customers,
partners

–  PostgreSQL
not
part
of
their
database
standards

–  No
in-‐house
PostgreSQL
DBA
skills

–  Lack
of
integrated
tools
,
monitor,
ease-‐of-‐use

–  Large
enterprises
have
to
convince
their
managed

hosXng
partners
to
support
the
database
also

•  AKendees
here
are
going
to
be
educated
about
it

–  but
we
are
worried
about
the
rest
of
the
world


IniXal
Internal
Feedback

•  DB
Support
Matrix
complexity
increases

–  Every
new
DB
support
increases
N
plaborm
test

–  Large
scale
tests
are
serialized
due
to
resources


Plan:
Embedded
Default
DB

•  Embed
PostgreSQL
as
part
of
the
product

•  Make
it
the
default
choice
opXon
for
most
deployment

•  Do
not
expose
database
directly
to
endusers

•  ApplicaXon
will
manage
the
DB

•  Support
it
for
producXon
deployment

•  Customer
can
sXll
select
to
use
an
external
database
of

their
choice
or
save
DB
License
cost
with
the

embedded
one


Ground
Work:
MigraXon

•  Products
used
various
databases

–  SQL
Server

–  Oracle

–  MySQL

•  Few
had
about
1500
stored
procedures

•  Some
used
exoXc
features

•  For
example

MySQL
used
InMemory
Engine,
InnodB

engine,
MyISAM
engine
in
the
same
workload


MigraXon

•  Leveraged
contractors
to
migrate
stored

procedures

•  Performance
bake-‐oﬀ
tests
to
convince

product
teams

•  Integrated
into
the
build
infrastructure
to

ease
pulling
components


Example:
vCenter
Server
Appliance


Plan:
Backup
&
Recovery

•  DB
not
the
only
thing
to
be
backed
up

•  ConﬁguraXon
ﬁles

•  ApplicaXon
Status

•  All
data
to
be
backed
up
went
into
a
virtual
disk
backed

up
with
vADP

•  pg_dump
/
pg_restore
used
in
few
cases


IniXal
Embedded
DB
Deployments

•  Each
Product
team
is
diﬀerent

– 
has
diﬀerent
levels
of
DB
Development
skills

•  Each
architect
comes
up
with
their
own
design

–  Postgres
Service
designed
(Start/Stop)

–  own
DB
Layout
design

–  own
DB
Security
design

–  Some
Tuned
their
DB

–  Many
did
not


Early
Results

•  YES
!
Customer
started
seeing
benefit

•  However

–  Experience
of
PostgreSQL
different
on
each
product

–  Random
security
holes
idenXfied
due
to
lack
of

knowledge
of
PostgreSQL
nuances

–  Performance
configuraXon
was
an
issue

–  Upgrade
to
new
versions
of
PostgreSQL
was
an

akerthought
(What
no
in-‐place
upgrade?)

–  Legal
work
required
for
OSS
increased

–  Resulted
in
PostgreSQL
version
sXckiness


Some
encounters

•  Portable
binaries
–
Shrink
wrapped
sokware

(not
compiled
on
same
systems)

–  OSS
Libraries
used
for
compiling
PostgreSQL
have

to
be
shipped

–  Manage
security
patches
for
OSS
Libraries

•  Need
Standard
drivers
like
ODBC,
JDBC


MulX-‐Plaborm
Requirement

•  As
company
grew,
acquisiXons
grew

•  Some
products
only
worked
on
Windows

•  Few
used
exclusively
.NET
framework


SoluXon:
Embedded
DBaaS

•  Provide
End
to
End
Experience
with

PostgreSQL


InstallaXon

•  Extend
installaXon
to
setup
a
service

•  Conﬁgurable
to
set
up
as
internal
service
or

external
service
(mostly
embedded)

•  Security
handled
as
part
of
service
setup
to

follow
best
pracXcesfor
embedded

•  Security
based
on
access
and
minimum
roles

deﬁned


InstallaXon

•  RPM

installaXon

–  Dependent
OSS
Libraries
to
make
RPM
portable

–  Server
and
client
RPMs

–  Service
Setup
conﬁgurator
to
setup
a
default
service

• 
• 
• 
• 

User
setup

initdb
setup

pg_hba
setup

Password
handling

–  Provide
tools
to
quickly
setup
DB
Owner,
DB
for

various
users
of
the
service


Installer
on
Windows

• 
• 
• 
• 

PostgreSQL
Installer
on
Windows

Service
setup
on
Windows

ODBC
Support

Npgsql
Support
for
.NET
Plaborms


Backup
&
Recovery

•  While
other
backups
strategy
are
available

–  Nothing
beats
standard
database
backups

–  Design
cron
jobs
for
criXcal
data

–  Design
Archive
On
Demand
for
semng
up

replicaXon
quickly

–  Most
customer
do
not
want
to
hack
with
backup/
restore
scripts.

–  Should
work
out
of
the
box


Upgrades

•  Seamless
upgrade
to
new
version

–  RPM
Upgrades
needs
to
be
transparent
for
minor

versions

–  RPM
upgrades
for
major
version
needs
to
migrate

data
from
the
older
version
in
the
default
service


Upgrades

•  Problems
sXll
not
completely
resolved

–  Reduce
downXme
for
major
release
upgrades

–  Generally
migraXon
since
in-‐place
upgrade
does

not
provide
ways
to
roll
back
the
upgrade


Performance

•  Default
PostgreSQL
semng
are
conservaXve

•  Most
deployments
use
default
values

•  Started
doing
auto-‐tuning

–  Modiﬁed
PostgreSQL
core
engine
with
intelligence

–  For
checkpoint
segments,

shared_memory

•  Added
tool
to
auto
tune
postgresql.conf

–  Based
on
memory
share
for
DB
Service

–  Product
workload
dependent


Performance
related
tools

•  Provided
tools
to
understand
what’s

happening
with
Postgres

–  pg_top
with
changes

•  Changes
already
contributed
back
to
community


Example:
pg_top


Performance
related
future
work

•  ExisXng
tools
sXll
meant
for
expert
users

•  Lot
of
demand
for
simple
performance
reports

•  Looking
into
pg_statspack


Availability

•  Created
a
watchdog
for
PostgreSQL
to
restart

PostgreSQL
if
database
cannot
be
accessed
for

long

•  If
restart
fails
then
it
can
throw
alerts

•  Also
created
plugins
for
monitoring
the

service


Related
Problems:
Crashes

•  PostgreSQL
started
throwing
errors
with
“Out

of
Memory”
errors
in
stress
tests

•  MulXple
applicaXons
can
ﬁght
for
memory

•  Made
Performance
semngs
less
aggressively


Supportability

•  When
customers
hit
a
problem

–  Hard
to
get
access
to
the
system

•  Created
support
package
tool
to
collect

relevant
informaXon

–  Collect
conﬁg,
status
in
a
tar
ﬁle
for
analysis


Usability

•  Experts
love
command
line
uXliXes

•  Novices
love
GUI,
WebUI

•  It
needs
to
match
the
other
GUIs
VMware
ship


Pluggable
UI

•  Matching
VMware’s
GUI
Plaborm


Embedded
DBaaS
Architecture

VMware

Product

Service

Conﬁgurator

WebUI

Pg_top

JDBC

PostgreSQL

ODBC

Linux
Based
VM

Watchdog

Perf-‐Tuner

Monitoring

Support

package
tool

vSphere
Plaborm


Virtual
Appliances


ReplicaXon
between
vApps

•  Provide
simple
uXliXes
to
quickly

–  archive_command

–  create_replicaXon_user

–  run_as_replica

–  show_replica_status

–  promote_replica_to_primary


In
Closing

•  Free
Sokware
is
nice

•  Experience
–
What
business
users
care

–  User
Experience

–  DBA
Experience

–  Support
Experience

–  etc

•  Improve
the
Experience
with
the
technology


Thanks.

QuesXons?

Download
VMware
vFabric
Postgres

hKp://vmware.com/go/download-‐vfabric-‐postgres

Follow us on twitter: @vPostgres
Data Platform Blog: http://blogs.vmware.com/dataplatforms


My experience with embedding PostgreSQL

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to My experience with embedding PostgreSQL (20)

Recently uploaded (20)

My experience with embedding PostgreSQL