PostgreSQL as Enterprise Solution v1.1.pdf

PostgreSQL
As Enterprise Solution
By: Julyanto Sutandang
Equnix Business Solutions
https://equnix.asia
Copyright © 2019 - version: 1.1

#1 PostgreSQL Expert in ASEAN
Equnix Business Solutions (www.equnix.asia)
Table of Contents
Table of Contents 1
Copyright Notice 4
Relational Database Management System 5
RDBMS for Financial 5
Transaction 6
Atomicity 6
Consistency 6
Isolation 7
Durability 7
Why (we should use) Postgres? 8
PostgreSQL is an Open Source Project 8
Open Source Movement 9
Linux was born to complement 9
Open Source changes the World 10
A Brief Story of PostgreSQL 12
Reborn as Open Source Project 12
PostgreSQL Development 12
PostgreSQL Forks 13
Business Ecosystem for PostgreSQL 14
Overview 14
Research and Development 15
Government Involvement 15
Principal Supports 15
Selection Criteria 16
Professional Services 17
PostgreSQL for Financial Transaction 18
Installation 18
Tuning 18
Production System 21
Mitigation Method in HA Cases 23
Master Failover 23
Standby Down 23
Private Network Problem 23
Public Network Problem 24
PostgreSQL as Enterprise Solution - 2

Resynchronization 24
Deepgreen Data Warehouse Solution 27
Cluster Manager 31
Cluster Node 31
Storage Interface 31
Backup and Restore 32
Community Contribution 33
Online Media Socialization 33
Press Release Publication 33
Guest Lecture 33
Meetup 35
Client Gathering 36
International Speaker 38
International Sponsor 39
PGConf.Asia 2019 42

Copyright Notice
Copyright 2007-2019 Equnix Business Solutions. All rights reserved.
No part of this publication may be reproduced, distributed, or transmitted in any form or by
any means, including photocopying, recording, or other electronic or mechanical methods,
without the prior written permission of the publisher, except in the case of brief quotations
embodied in critical reviews and certain other noncommercial uses permitted by copyright
law.
Data and information of this proposal and quotation are for limited usage and may not be
disseminated. Information on the document can be changed anytime without prior notice.

Relational Database Management System
Information Technology has penetrated our lives very deeply; from waking up till going to bed,
we always meet IT implementations. Almost every IT implementation in business uses a
database system. In fact, every IT system, from the biggest one in a Supercomputing machine,
up to the smallest one like our phones, uses a database server to store their data. Not all
databases have the same level of importance, some of them are very important, as important
as the owner itself. Some industries rely very much on Data Integrity and its transaction
support. Their failure in data management, can affect their company up to out of service or out
of business. Financial related companies require Reliability, Availability and Scalability of the
data than the others. That term was shortened as RAS, the key feature of Enterprise level
delivery. Enterprise level key features are mostly used by Financial Industries.
RDBMS for Financial
Financial Industry is the driver of the
economy, positioned as the foundation
of economic development. Analogy with
the body system, when money is the
blood, then the Financial Company is
the heart who circulates the blood. As
an economy driver, the Financial
Industry requires a sophisticated
system which is able to help them grow
steadily and operate without hassle.
Database Server in the Financial
Industry has played a major role. Every
Financial system needs a sophisticated
data management, not only for storing
data, but also require them to be
"smart", like: Get what we’ve already put; and get it as fast as possible; I want to exchange money,
please do not drop in the middle; while there are tons of data, it should stay intact; Data will not be
corrupted in anyway, including when there are disasters.
Data management is a really important issue for Enterprises, especially Financial Institutions.
It requires a transactional supported database, there comes RDBMS (Relational Database
Management System) which has transactional feature support. RDBMS is designed specifically
for financial transaction, it has four aspects: Atomicity, Consistency, Isolation, and Durability,
an industry standard, since its inception in the 1970's till today is still considered as the core
feature of financial related data management. Some databases in this category are:
PostgreSQL, Oracle, Sybase, Microsoft SQL Server, etc.
RDBMS Concept was invented by IBM Fellow Scientist Edgar F Codd in 1969, it is written in his
whitepaper called “A Relational Model of Data for Large Shared Data Banks” to support

Financial Transaction in Banks. The Whitepaper document was the source of the initial
development of RDBMS, such as INGRES (which later developed into PostgreSQL) and Oracle
developed by Larry Ellison. PostgreSQL and Oracle are made based on those principles,
including: Informix (which has the same ancestor with PostgreSQL), Sybase, MS SQL (which is
derived from Sybase), etc.
The most important thing for them is transaction capability of RDBMS, which is able to
transact at once (Atomicity), proceed the data without alteration (Consistency), transact safely
without interference by other transaction (Isolation), and Every committed transaction with
stay intact whatever happens (Durability). All of those are main features of RDBMS, shorts as
ACID.
Transaction
The main proposition of RDBMS is structured data with relational capability for transactions.
Transaction is defined as an agreement between two or more parties that involves changing or
exchanging or replacing data. Transaction involves one or more data changes, and generally
occurs between two or more parties. and is the most important thing in RDBMS and must be
well maintained in terms of high performance and scalability.
RDBMS should be able to manage growing workloads, which is able to be very large and
critical.
Atomicity
Transaction must be carried out in an atomic way, it should be done with the concept of all or
nothing. Example, when there is an amount transfer between two parties, there are two
separate steps:
1. Subtract amount from the balance source account of the transfer, and
2. Add the amount to the balance recipient account of the transfer.
These two steps / queries must be carried out as a single atomic transaction. If there are
conditions that are not met, then all processes must be returned (roll-back) to the initial /
value. So, wrapping these queries in a single transactional process is the basic idea of

Atomicity, and also to ensure that transactions are done with the concept of all or nothing, and
to ensure data consistency and security.
The following code illustration explains how transactions are built:
BEGIN
UPDATE maccount SET balance = balance - amount WHERE accno = 1001;
UPDATE maccount SET balance = balance + amount WHERE accno = 1023;
COMMIT;
Consistency
When multiple queries are run together and with all clients accessing the same data, they will
get the same value. Each transaction has the potential to influence the value of data, and there
are also rules, triggers, and constraints, which are involved in how the data is changed. The
database guarantees consistency of the data provided for each transaction. This does not
guarantee the accuracy (correctness) of the transaction in a way that the application developer

is intended to. Application programming error will not violate Consistency of Database. The
Database consistency will not compromise even though it involves complex operation.
Isolation
When there are several clients running transactions in the database, involving the same tables
and records, each transaction is mutually isolated from other transactions and actually runs
the transaction context in separate temporary memory. If there are two or more transactions
accessing the same record / tuple, the newer transaction process will run after the completion
of the latter transaction. Transaction implements isolation by running a row level locking
mechanism. Each transaction that shares data in rows will be executed sequentially to ensure
data isolation and avoid race conditions.
When the previous transaction is finally committed, the latter transaction will receive the
latest committed value, and start its own data manipulation. This ensures that each transaction
is isolated from each other and ensures data integrity in each stage of the process.
Durability
RDBMS should be able to store data persistently in any condition. This capability ensures write
to disk (Transaction Log) for each commit execution. When the commit is not finished, the
data has not been saved on the disk, to ensure the transaction is running because there is still
a possibility of rollback. By running writes for each commit, eating will create a frequent write
operation at disk level. Highly write operation in database caused by a change and commit, is a
must which is not able to avoid it, unless there is another technology which is able to mitigate
writing amplification issues.
By understanding those concept and its application in RDBMS, there are a number of points
that we must apply to maintain database performance and stability: The write cache disk must
be avoided, because it reduces DURABILITY. Redundant write caching at the disk level leads to
several performance problems and data inconsistencies when there is a decrease in electrical
power.
1. Turn off filesystem level journaling.
2. Turn on the special disk volume that has a high IOPS to store LOG data used in Logging.

Why (we should use) Postgres?
Not all the RDBMS are the same, even though they comply with ANSI SQL, there are some
functionality and specific requirements which make RDBMS divided into some levels, based on
its capability to cater loads. In short we categorize them as: Enterprise class RDBMS and
Non-Enterprise class. PostgreSQL and Oracle fall into category one. Almost all financial related
companies in the world use Oracle which is also known as the RDBMS market leader. Some of
Oracle Database Servers converted into PostgreSQL which is more powerful than Oracle (well
proven in certain cases). And that runs better than when they are using Oracle.
PostgreSQL is considered as the best alternative to replace Oracle. PostgreSQL supports
NoSQL data format natively. Support Json, Jsonb, jsonpath, is easier for Designer or Developer
who want to mix data sets (structured or unstructured). It supports cutting-edge Enterprise
features: Synchronous and Asynchronous Replication; Binary or Logical Replication; Trigger
based Replication; Multi-master or Bidirectional; faster backup and recovery; Parallel Query;
SHA-256 for authentication, full snapshot nor incremental; Point in time recovery run just like
time machine, and many others.
PostgreSQL is an Open Source Project
PostgreSQL is licensed as Non-commercial licenses, therefore it belongs to the public domain,
meaning that PostgreSQL is owned by all of us including our children and grandchildren in the
future. PostgreSQL is categorized as Pure Open Source, not all Open Source are the same,
many of them are just marketing gimmicks, or Company endorsement. Some software in Open
Source requires us to buy the licenses in order to use it in a commercial area, that is a sign of
non-pure Open Source. Some vendors create a product derived from PostgreSQL, yet they use
the same or similar Postgres name, but that software is not part of the Open Source Projects
and therefore is not considered as Open Source.
Other non-pure Open Source like MySQL, Jboss, Odoo are actually developed and supported
by the majority of people who are employees of that company, and perhaps only a little part of
the development is driven by the community. Even so, we are able to access the source code,
sometimes the product which professionally runs on a production machine has different
binary or source code and features compared to the Open Source one. They release a source
code as Open Source just to raise their popularity and be endorsed by the community to gain
more popularity.
The most latent potential, non-pure Open Source could has hidden agenda. Anytime the
non-pure Open Source can easily fallback and become Propietary software or being
abandoned when the major developers pulled back their support on that.
This case definitely cannot happen in pure Open Source like PostgreSQL or Linux Kernel, since
there is no single entity which has a dominant effort to drive the development roadmap on
their own. Pure Open Source never falls back into closed source; since it is protected by
non-commercial licenses, and has strong community support spread all over the world.
PostgreSQL license statement is derived from BSD-like license, actually you can download
PostgreSQL and make your own closed source one.

Open Source Movement
The Open Source Movement is a branch of the Free Software movement that embraces the
commercialization of software. The impact of this movement enabled widespread adoption of
Open Source software by businesses.
At the beginning of software development since the 1950’s, sharing source code was the norm
because software users were trained programmers but software was rarely able to meet every
particular need, with a range of different hardware technology and devices. This helps modify
bugs and add functionality and adoption easier. Changes have occurred since the 1970’s when
operating systems and programming language
compilers became increasingly complicated, which
gave birth to software companies and encouraged
paid licenses.
In the midst of the rise of paid software, in 1983
computer scientist Richard Stallman initiated the
GNU Project to create a complete operating system
with free to distribute source code, and the Free
Software movement that emphasized the basic
freedom of users using, studying, modifying and
sharing softwares.
Linux was born to complement
In the 1990s Open Source software witnessed other advances.
Linus Torvarlds created the Linux operating system kernel in
1991 and complements the GNU Project, which at that time lack
a kernel. GNU/Linux and its variants are then widely adopted by
businesses. The dot.com company in the 2000s even relied on
free and Open Source software infrastructure; basic Linux
kernel, Apache web-server, PostgreSQL, MySQL, Java, Python,
Perl and PHP programming languages. All that gives free
software and Open Source a better position than paid software.
In 1998 the Free Software movement was increasingly seen as being unfriendly to the business
world, because it was too encouraging for their view that paid software was unjust and
unethical. This reduces free software adoption in business circles. The Open Source Movement
was born as a branch and re-branding of the Free Software movement to embrace business
users. The Open Source Movement -influenced by the writings of Eric Raymond in 1997
entitled "The Cathedral and The Bazaar"- began with the release of Netscape's source code,
since it lost in court with Microsoft Internet Explorer, in January 1998.
Eric Raymond captured that momentum to revitalize the Free
Software movement, and held a strategy session in Palo Alto,
California in February 1998. The strategy session approved a
new approach that favored an open software development
model that did not exclude business people. The term was also
chosen to distinguish from Free Software, and the term Open
Source was chosen as the official term for the new movement.

The Open Source Initiative (OSI) was later established in 1998 as the management of the Open
Source movement. OSI focuses on creating and maintaining Open Source Definition (OSD). The
OSI reviews software licenses to meet OSD standards for approval of Open Source software
classification, and maintains a list of approved licenses. OSI also acts as an Open Source
campaign activist.
The Open Source Movement focuses on aspects of public availability free of source code. This
is in contrast to Free Software, which over time is leaning towards the meaning of free. On the
other hand, with a focus on the availability of source code, Open Source software also indicates
being willing to coexist with paid software. Willingness to coexist with paid software allows
widespread adoption of Open Source software. Now Open Source software is implemented at
least in almost all companies with IT infrastructure. A combined Black Duck Software and
North Bridge survey found that 78% of companies run Open Source software in part or all of
their software operations.
Open Source changes the World
The biggest impact of free and Open Source software for the world is, literally, the Google
company. Now the search engine giant is expanding into various fields serving most needed by
modern human lives: Email, Mobile, GPS, Drive, Translation, Search, etc. But Google will not
grow until now, without penfree software and Open Source. They save on software
development resources by building their resources on free and Open Source software.
In addition to technology companies, financial companies such as Bloomberg and CME Group
also benefit from free and Open Source software by saving resource for building software.
Being more efficient is always better. The film
industry also benefits from free software and
Open Source. For example, the Titanic film uses
Linux to handle its CGI production storages.
Linux is a great operating system, easily able to
modify its filesystem to manage huge data loads
of CGI movie renders, compared to proprietary
which won’t be able to adjust as needed. FOSS
has proven better for business, especially when
the business has a frequently changing situation.
Business by nature should be flexible, adaptable,
and adjustable and FOSS has proved to be better
than the others.

A Brief Story of PostgreSQL
PostgreSQL was born from the concept of a
relational database that was invented by IBM
scientist Edgar F. Codd in 1969. From 1977 to
1985, Prof. Michael Stonebraker from the
University of California, Berkeley (UCB),
developed an INGRES (Interactive Graphics
Retrieval System) database system academic
project.
In 1986 Prof. Stonebraker continued with the
POSTGRES project. The name POSTGRES is a word puzzle on INGRES,
namely POST-INGRES (after INGRES). After eight years of
development, in 1995 POSTGRES was commercialized as ILLUSTRA,
then became INFORMIX, which was later acquired by IBM. Two Stonebraker students, Andrew
Yu and Jolly Chen, replaced the POSTGRES query language from PostQUEL to SQL, renamed it
Postgres95 and released it as Open Source software in 1996.
Reborn as Open Source Project
Since 1996, Postgres95 has entered the Open Source world even more, thanks to the hard work
of an independent group of developers outside of UCB that is spread globally. This group's
work produced a very stable database system, and introduced various important concepts such
as rules, stored procedures, and complete documentation for users and developers, with a new
name used today: PostgreSQL, which reflects the relationship between Postgres and SQL
capabilities.
PostgreSQL Development
The PostgreSQL version starts from 6.0 as a tribute to previous development. In versions 6.0 to
7.0 (1996-2000), many changes and improvements were made, including the addition of
important features such as Multiversion Concurrency Control (MVCC), various types of data
including geospatial data, various important SQL functions, and improved performance and
process speed.
Versions 7.0 to 7.4 (2000-2004) are filled with additional important functions, such as
Write-Ahead Log (WAL) and support for complex queries. In 2005, version 8.0 added
enterprise-class business features such as Point-in-Time Recovery, Tablespace, and ported to
Windows. PostgreSQL 9.0 to 9.5 provides important functions such as Streaming Replication
and Hot Standby, JSON and JSONB data types, and FDW. PostgreSQL 10 able to do parallel
SCAN, JOIN and AGGREGATE. Starts from version 10, PostgreSQL uses a new version number
and now it happen like: X.Y; where X major version and Y is the minor version. Postgres last
version is 11 up to this writing. The important features and functions of PostgreSQL were
developed by the contributions of many developers, both individuals and corporates, in the
community. And also there are many great companies which serves Postgres professionally
globally.

PostgreSQL Forks
PostgreSQL has many forks, both Open Source and
commercial ones. PostgreSQL redistribution was
started by Great Bridge in 2000, followed by Fujitsu,
Red Hat, Pervasive, and EnterpriseDB. Porting to
Windows also developed, for example NuSphere
UltraSQL in 2002, and PowerGres from Fujitsu, which
is still being updated today.
Many Big Data solutions are also built on PostgreSQL.
Netezza, a pioneer of Data Warehouse Appliance
which was then acquired by IBM, based its system on
PostgreSQL. Other forks such as AsterDB from
Teradata, Industry standard solutions for
telecommunications industry; Greenplum, a popular
Massively Parallel Processing solution; Deepgreen,
Data Warehouse with MPP capability similar to
greenplum but with steroids (5x faster than
greenplum); and Yahoo Everest, which is modified PostgreSQL to manage up to two petabytes
of Yahoo Search Engine data in 2008; and there are many other forks being used
independently.
There is also a Clustering solution for the database server cluster. Fork Clustering for example
PGCluster, PostgreSQL-R, Postgres-XL (right now, it is maintained by 2ndQuadrant), and
PG_SHARD from CitusData (now it is bought by Microsoft), as well as BDR (Bidirectional
Replication) maintained by 2ndQuadrant which is increasingly popular for enterprise-class
business implementation. Other PostgreSQL forks offer exotic functions, such as TelegraphCQ,
an adaptive dataflow engine for querying data; RecDB, engine for recommendations; TruCQ,
for continuous analytics; and PipelineDB, for streaming-SQL databases, TimescaleDb for
time-series data, and many more...
Many organizations, government institutions, academics and large enterprise-class companies
use PostgreSQL as their database system. PostgreSQL can be found on Cisco, NEC, NTT Data,
Fujitsu, Hitachi, NEC, Yahoo, Amazon, Skype, Instagram, Last.fm, US Department of Forestry,
even at the Caixa Economica Federal bank in Brazil. In Indonesia PostgreSQL has been used in
many places such as Mandiri Bank, BCA Bank, BJB Bank, Danamon Bank, Resona Perdania
Bank, FIF, Prudential, Telkomsel, Truemoney, Transmart Carrefour, XL Axiata, Jember
University and Adira Finance.
Now, PostgreSQL is known as the most advanced, object-oriented Relational Database
Management (ORDBMS) Open Source System in the World. Departing from an academic
project based on the concept of a relational database, PostgreSQL has gone through many
developments and transformed into the most sophisticated and open source relational
database system used by enterprise-class businesses.

Business Ecosystem for PostgreSQL
Overview
A technology requires an ecosystem. A business
product requires a business ecosystem. A car is a
product of a technology, requires Manufacturer,
Research in private as well as in academic, Market,
second hand market, Distributions, Services, after
sales services, even non-principal services
workshop for old one. All that component forms
into Business Ecosystem.
That also happens to PostgreSQL, as a product, as a
technology. It requires all brand new
component/entity or institution to support this
new technology. Without any great support from
the ecosystem, PostgreSQL will not able to sustain
in the long run.
A quite unique technology which even patented,
and supported well by a single institution will not
be able to survive well since there will always be
competitors who try to disrupt their business. It is only creates a lose-lose situation, a red
bloody ocean. So, collaboration, building up the ecosystem altogether is the best way to do,
especially in Open Source based technology.

Research and Development
First things first, before an Institution can become a PostgreSQL Professional Service Provider,
One should have research and development team. Building up the research team requires
great effort and time, therefore collaboration in the research and development is a great step.
Collaboration can be done by building up our own team, hire contributors, partnership with
academics, and so on. There is no exact step in achieving research excellence.
Ideal condition is having partnership between companies who has the same interest and
exchange the research result is such mutual manner. Another approach is having mutual
partnership with Academic who has research resources for their obligation in the study.
Government involvement in this area are greatly appreciate it, since the private sector tends
do not like to exchange information and has potentially deconstructive effort since it is run
without coordination and big great roadmap.
It is government duty to lay down Masterplan and develop a roadmap for the future according
to their power to make laws and rules. Government has a very strategic position and important
portion develop the business ecosystem.
Government Involvement
There are 3 Important Key Sector in order to have a successful business ecosystem for
PostgreSQL: Government, Academic and Business. Even Though the business ecosystem
(shortened to BE) can be developed privately without Government involvement, in the Open
Source based product like PostgreSQL, it will be greatly helpful and can boost the solutions
and benefit far better above our imagination.
Government has 3 main position, the government agent will play all three rule closely, it is:
1. Research director and Incentif, Managed Expert and creates blueprints and roadmaps.
Managed research according to the roadmap, evaluate every year. It creates a research
competition for developer/researcher and run meetups (Local, International,
Conferences, etc)
2. Regulator, create a set of rules for implementation, certifications, procedures, penalty,
compliance, awards, etc
3. Campaign, Socialization, for users, creates a seminars, trainings, competitions, etc.
Principal Supports
Like other paid software, the usage of the Open Source software also needs support up to the
principal level. General principal level supports: distribution of bug fixes, performance and
security patches, and general consultations.
In terms of principal support, Open Source software has different approach compared to
proprietary paid software. Paid software has its singular manufacturer as the principal. While
Free and Open Source software enable many parties to learn and become self-claim experts on
the software, anybody who think they have capability can claim as the Acting Principal. So,
many parties emerged as principals for various Open Source software, and provided many
choices for users. This can be good or bad.

Because of the lack of regulation, Users must wisely and independently choose principal for
their Open Source software implementation. The measurement of the competence of the
Vendor as the main support of the implementation of FOSS is really important. The criteria is
"tricky", and the high risk lurks behind.
It is recommended that the Open Source software principal be selected by filtering according
to certain criteria. Here some key points for Vendor evaluation: Portfolios, Contributions,
Expertise in Software Development, etc.
Selection Criteria
The principal's portfolio will expressly
show their history and experience in
providing principal-level support for
their clients. There are several aspects
of the portfolio that increasingly
distinguish between principals, namely;
1. Number of accounts, Simply
put, the number of users who choose
the principal shows how trustworthy
the principal is.
2. Variations in the type of
industry account, each industry
certainly has different demands for
using the Open Source software. More
kinds of industry type strengthen its
position. Every industry needs their
own tuning approach since they had their own specific business process, and their own
best practice.
3. Number of instances, Instance is a measure of how many times the software is run.
While the number of the account is small, high number of instances would be a great
achievement. More instances more risk and reward, it also sharpen our support with
more cases to solve.
4. Different kinds of Services, more services means more expertise. Type of service
usually: Installation, Configuration, Sizing, Tuning, Fine Tuning, Performance Tuning,
Scalability, Availability, Security, Consultation, Optimization, Migration, High
Availability, Disaster Recovery, Risk Mitigation, Monitoring and Control, Alert System,
etc.
5. Year of Service, of course the longer years of service provide more experience in
solving day to day problems, reveal its maturity and professionalism in managing the
support of the principal level. The longer "flight hours" of one principal, the more
experienced they are in providing support for the principal level for the Open Source
software that they support.
All aspects of the portfolio must be considered together in selecting principal for Open Source
software. The portfolio is an important initial measure for assessing which parties are able to
provide principal level support.

Professional Services
Professional services provided by service provider companies (hereinafter referred to as SP) to
companies using PostgreSQL (hereinafter referred to as clients) to guarantee continuous
service availability. The SP should have some certain level of Expertise, Government
involvement in the SP certification would help client to select which SP is appropriate for
them. In short, this services support are divided into 3 levels:
1. Principal level Support,
The principal responsibility in open source based services is to replace what is the most
and usual principal responsibility in the paid software industries. The core competence
is to be able to fix the bug, patch the glitch and errors: performance, stability and
security. To ensure every tested function runs well.
Not only being reactive support, in other way Principal support should have very
proactive: research new features, creates procedures and best practice, discovery
potential threat, avoid misconduct feature, disable potential bug, etc.
As principal level support, SP didn’t meet the client directly; But, that is not always
2. Expert Level Support,
Some of the services provided by the SP to the client are best practice, set of
standardized procedure. SP might not able to do research or having research team,
instead SP can become a partner to the one who has the ability and capability to do as
principal support. In
3. Operation Level Support.
SP becomes a first hand when handling any client’s issue. Therefore SP has to have any
knowledge regarding for troubleshooting and analyzing the cases, using best practices.
When want to be Operation Level Support, SP has to be certified by following the
certification test and proper training modules.

PostgreSQL for Financial Transaction
Installation
The installation on production server should be done by vendor using binary
executables compiled before and has gone through comprehensive Quality Control and
patched by vendors to adjust to the conditions of the production system.
In the installation, all required extensions that have been specified as extensions
that must exist on delivery such as pgstattuple, pg_tgrm, and pg_stat_statement. Not only
extension, vendor should also install monitoring tools, pg_activity. pg_activity used to
perform live monitoring database as queries are running, waiting query, and the query
blocking. Monitoring is also done on vital resources such as CPU usage, Memory Usage, and
Disk Usage. When the value of the resource touches the predetermined threshold point, the
monitoring system will send an email to the Client Technical PIC to conduct a direct check.
The requirements of related installation to Layout Storage, where each of these
mounting points uses a separate disk (not using just one disk, then separate the partitions),
as follows:
1. Data directory, according to the current target data size, can use SAS/SCSI/HDD, no
need as fast as SSD.
2. Transaction log directory, 20% of data directory, but can be adjusted also with the
growth of the existing WAL (transaction log). Must use at least an SSD (if you can use
NVMe-SSD better), because it requires the fastest IOPS.
3. Index directory, at least 50% of data directory, but can be adjusted later after further
data growth analysis, can use SAS / SCSI / HDD, no need as fast as SSD.
4. Tablespace directory, at least 50% of data directory, but can be adjusted later after
further data growth analysis, can use SAS/SCSI/HDD, no need as fast as SSD.
5. Archive directory, for at least 3 times the size Transaction log directory, to maintain
WAL for the past 3 weeks, can use SAS/SCSI/HDD, no need as fast as SSD.
6. Full backup directory, for at least 3 times the size data directory, to maintain directory
data for the past 3 weeks, can use SAS/SCSI/HDD, no need as fast as SSD.
Tuning
Tuning is an important role in installation of PostgreSQL, There is no installation
PostgreSQL without Tuning, since PostgreSQL is utilizing the CPU, Memory and Disk I/O
resources aggressively (it depends on the situation) then maintaining PostgreSQL is not the
same as maintaining any ordinary application. Most application is memory bound, they
don’t have any issue with disk writing, not even has any complex or massive calculations.
PostgreSQL highly utilize disk I/O for writes every commit, then it should be taken
care well by providing the right storage, and the right mount path to address this issue.
There are some volume should have a quite fast I/O, there should be another one with not
really fast I/O to save budget, and there are some volume with slow I/O only for storing
archive data.

PostgreSQL blocks memory into heaps, shared and kernel cache for their own
process with specific tasks; having a great amount of memory is not enough, ti should
allocate each memory to each task wisely. This is quite important since mismanage this
issue will affect to inefficiency of memory usage. So, it is not only having a great amount of
memory but also tuning the memory parameter is important.
PostgreSQL is a Relational Database System, and therefore it creates relation
between tables, objects. In CPU perspective, relation is created by comparing between 2
data, and therefore it requires mathematical operation (comparing utilize subtraction
operation) when Postgres relate a million rows with another million rows, practically it is a
computational operation and will occupy CPU usage. Another operation which consume
CPU is a spinlock. Spinlock is a lightweight locks which spins the hardware thread waiting
for another lock, higher CPU clock will faster those operation and minimize waiting time
for spinlock, avoiding lock timeout or deadlock.
Here below the list of section for Tuning:
a. Hardware
Hardware tuning includes disk usage, memory (RAM), and the CPU, but which should
be tuned in addition to using sufficient memory and CPU which has a high-speed disk
usage is also very influential. On the storage side, things should be tuned are data
sharing, transaction log, table space into several separate disks.
As previously explained, the transaction log (WAL) should use a very fast disk. Currently
this very fast disk is held by NVMe SSD (Non-Volatile Memory Express Solid State Disk),
which is an SSD disk that is installed directly through the PCI path, so there will be no
bottleneck from cable settings like SSDs in general. The disk for transaction log needs
to use a disk that is very fast because every transaction that occurs in the database will
initially be written first into the WAL file which is in the transaction log directory.
Whereas disks are used for data, tablespaces (indexes and data), and other partitions
such as application logs (options) can use SAS/SCSI/HDD disks and do not need to be as
fast as SSDs. But to improve performance and minimize spindle disks can divide each
of these functions into separate disks. So even though using a disk below the SSD, it can
improve performance because each disk is intended for each function independently.
b. Operating system
Tuning from the Operating System side includes user parameters, kernel parameters,
and operating system parameters. The user used as a super user from the database is a
user named pgsql. That means the process from PostgreSQL will run as a pgsql user.
So for some parameters in the kernel, adjustments need to be made for the user, such
as the open file limit that needs to be increased from the default value.
Other parameters that need to be adjusted and increased are semaphore parameters in
the kernel. This is to support resource usage that is more than the default value of the
Operating System, so that PostgreSQL can run more optimally by utilizing full
resources. In addition, other configurations that need to be disabled on the OS side are
Remove IPC and journaling for filesystems ext4 used for WAL.
c. PostgreSQL Parameters
Tuning from the side of the PostgreSQL parameter is very dependent on the resources
used, in terms of memory and disk usage used by PostgreSQL. There are several
memory allocation parameters that can be adjusted, for example Shared Memory is the
main parameter in PostgreSQL where the memory is the main part and the allocated

memory will be subdivided into other sub-processes.
d. Query
Other tuning is query tuning, which is done when found Slow queries appear on the
database log. There are several causes for Slow Queries to appear, including due to the
absence of INDEX in the related table that matches the query, the use of sub-excessive
query where the query is usually due to be generated by frameworks such as Hibernate
and others, the use of OR, and other causes. So that the tuning done for the Slow Query
can be done by adding INDEX or changes from the query side itself.
Some Best Practices that need to be concerned of:
1. Disable Write-Cache, for whatever reason - both using Virtual Machine or Bare Metal -
it is important to disable write cache on writing Disk. This is to ensure that the
durability of PostgreSQL is well maintained.
2. Ensure CPU Clock is high - CPU checking is done to ensure that the server uses a CPU
that has a high enough speed (> 2.5GHz for Intel Haswell and so on ).
3. An adequate number of CPU cores - this also relates to the amount of memory used, as
more and more cores that require more memory used, in order to create an optimal
parallelism.
4. Ensure Disk Read/Write Fast Performance - Checking is done by doing Write and Read
tests with each Volume installed on the server. This can determine whether the disk
capability is supported for a server with high transactions or not.
5. Software side checks - Checks include the Operating System, configuration of the
Operating System, filesystem, Kernel version, and several other parameters related to
the Software. This check is used to consider whether the Software needs to be updated
or even replace the Operating System.
6. Kernel Parameters - Not only from the Operating System used, the parameters or
kernel settings are also checked. Like the semaphore value of the kernel, the open file
limit value for the user used as a super user database, Journaling on the ext4 filesystem,
Remove IPC and several other parameters.

Production System
There are 4 PostgreSQL instances that will be configured, namely OLTP
Master, OLTP Standby, Replica, and OLAP . Each of these instances functions as:
1. OLTP Master, an instance that serves transactions from applications,
both Read and Write.
2. OLTP Standby, an instance that accepts transactional replication from
OLTP Master and can be promoted at any time to replace the Master. Spec
3. Replica, an instance that can be configured using late asynchronous
replication. Late replication is intended to give spare time (e.g. 5 minutes)
to DBA if there is a mistake done on Master Server by human error, so
DBA can retrieve the data backup from this Replica instance. This server
is optional.
4. OLAP, an instance which acts as a Data Warehouse (using Massive Parallel
Processing PostgreSQL instance, Deepgreen DB), to provide access to
applications in performing Read-only queries, such as Reporting.
For the HA configuration on, both the Master and Standby servers must have

the same resource allocation, so if at any time there is a failover, then Standby can
serve the same load as the Master. Late replication to replica is part of a Multi-tier
Backup, where DBA can retrieve lost data which is accidentally removed or
dropped. Every transaction log actually will be sent in real time to the Replica
instance, but the transaction within the WAL files replay is delayed. Replication to
Replica instance is using VIP as the primary connection string, so when Standby is
promoted if Master is down, Replica does not need to change the connection string.
The OLAP instance is configured as a Data Warehouse using Deepgreen DB.
High Availability solution can be implemented using Pacemaker and
Corosync in the constellation of Master and Standby instances. Pacemaker will
monitor each server and manage the implementation of Virtual IP on the Master
Server. There are 2 Virtual IPs, namely Public Virtual IP, which is accessed by
applications, and Private Virtual IP, which is used for communication heartbeat
(Corosync) and Replication. To reduce the factor of Single Point of Failure, the
network used for Replication and Heartbeat is recommended to use Direct Cable
Connection (direct cable connection between servers), to reduce latency on the
network, so that the delay on Replication is as small as possible.

Mitigation Method in HA Cases
a. Master Failover
When there is a problem in the Master instance, and the Master instance
can no longer serve the application or die, then Pacemaker will automatically
failover on the system, and the Standby instance will be promoted to the Master
instance in less than 10 seconds. Virtual IP (both Public and Private) will be
taken over automatically to the new Master instance (which was previously a
Standby instance). An email alert will be sent for this failover automatically so
that they can become the attention of the Technical Client PIC .
b. Standby Down
When there is a problem with the Standby instance , an email alert will be
sent to inform you that Replication to Standby has been lost. If Replication from
a Standby instance is still running from a Master instance, Virtual IP is still
present in the Master instance and failover will not be executed.
c. Private Network Problem
If the network used by the Pacemaker and replication problem, but in the
second instance is still online, the instance Standby will wait for 5 seconds to try
to demote the instance Master uses Public Network and if successful, then
failover to be executed by force to prevent the condition split-brain. The

split-brain condition is a condition where both instances assume each of them
is a Master instance, so try to take Virtual IP on both sides.
If the Standby instance cannot communicate and demote the Master
instance, the Standby instance will assume that the Master instance is dead and
immediately execute the failover process. Email alerts will also be sent to inform
this event and the same administrative command can be used to check the
status of each instance after a failover occurs .
d. Public Network Problem
If there is a problem with the Public Network connection from the active
Master instance, Pacemaker can detect and decide to failover to Standby. As
long as the Private Network is still running normally, failover will be executed
and Standby instances will take over all Virtual IP. Email alerts will be sent and
administrative commands for failover checks also apply to this case.
e. Resynchronization
After Replication is interrupted or failover is executed, to reconnect
Replication, the Resynchronization mechanism can be done manually using the
tools provided, both for instances that are Standby or Replica. After failover, the
process needs to be carried out at the instance to resynchronize old masters, so
that replication will run on the new master instance to instance of the old
masters.

Below are PostgreSQL powerful features which support Data Warehouse system:
1. Native Streaming Replication with Synchronous and Asynchronous mode without any
additional tools.
2. Automatic Parallel Query execution, which does not requires hint.
3. High interconnectivity to many data sources, using PostgreSQL Foreign Data Wrapper,
including: Oracle, MS SQL, MySQL, CSV file, XML file, etc.

4. Specially designed to boost performance on OLTP and OLAP database behaviour.
5. Full Text Search features to speed up searching on huge documents for data
warehouses.
6. Native table partitioning to spread load on selecting huge data within 1 instance or
even spread through multiple instances using FDW.

Deepgreen Data Warehouse Solution
Deepgreen DB Data Warehouse is a Clustered RDBMS specifically designed
to work with the concept of Massive Parallel Processing (MPP), where the Cluster
consists of Cluster Manager and many Cluster Nodes. Deepgreen DB is basically a
derivative of PostgreSQL with modifications, so it supports MPP and is specific to
Data Warehousing and operations related to OLAP or analytics. Deepgreen DB has
next-generation query processor which is enhanced by:
1. Better join and aggregation algorithms,
2. Better and newer subsystems to handle high loads, and
3. More advanced techniques to maximize CPU performance through
execution queries compiled with JIT (Just-in-Time), vectorized data
scanning, and data-path optimization.

Following are the features that are accommodated by Deepgreen DB:
1. The acceleration is 250% compared to Greenplum in the use of the Heap
Table.
2. 500% acceleration compared to Greenplum in performing queries for
scanning data and aggregating data from the Fact Table.
3. 300% acceleration compared to Greenplum in JOIN of transactional
tables and queries against other Dimensional Tables.

4. Has a decimal type with new arbitrary precision that is more efficient
than the normal numeric type, namely Decimal64 and Decimal 128 which
has a 16-digit and 34-digit precision level. By changing this data type,
performance in the query can be accelerated.
5. Supports JSON data types.
6. Supports sampling activities with SQL.
7. Supporting the Columnar Store called PAX, as an increase in the
Columnar Store commonly found on Greenplum. Columnar Store PAX
improves the performance that users often complain about such as:
a. "Too Many Open Files" error caused by 1 column stored as 1 file so
that the OS will assume too many open files.
b. Use too many Object IDs (OID) because 1 column uses 1 OID in the
Database.
c. Zonemap deficiency. Zonemap is an index that eliminates
redundant scans that never generate data.
8. Supports additional compression of zstd and lz4 to save data storage by
optimizing the decompression and compression process.
9. Interconnection with various data sources such as Amazon S3, HDFS,
Oracle, Elastic Search, Apache Kafka.

By utilizing the processing capabilities in parallel, the data will be
distributed through the network connected between Cluster Nodes and arranged
by the Cluster Manager. From the application side, only see 1 "main door" to access
this Cluster, namely Cluster Manager. By also utilizing the Storage Interface to
access data from sources outside the Cluster, it can speed up the extraction process
in the destination table. This parallel data processing will speed up the
performance of the Cluster as a whole, because all Cluster Nodes work
independently and the resulting data from each Node will be aggregated by the
Cluster Manager.
As a best practice, Cluster Nodes are configured as much as possible
according to the server specifications used, even though the data used is still not
that big. This is so that the data has been well distributed since the initial
implementation, so that it does not require redistribution when it is necessary to
add Cluster Nodes.

Cluster Manager
Cluster Manager is one of the main parts of the Deepgreen DB Cluster that accepts
connections and SQL queries from clients, and distributes them to each Cluster Node. If the
client connects to the database using the application, then the client will first interact with the
Cluster Manager. Cluster Manager also does not store data clients, but Cluster Manager
authenticates client access, accepts SQL commands, distributes those queries to Cluster
Nodes, and combines data obtained from Cluster Nodes so that they can be displayed back to
the client. One Cluster Manager stands as 1 PostgreSQL instance.
Cluster Node
Cluster Nodes are part of the Deepgreen DB, where each node stores a portion of the data and
the majority of query processes are carried out in the cluster node. Each data stored in each
Cluster Node varies depending on the distribution method used.
The number of cluster nodes that can be used in 1 cluster depends on the core CPU, RAM,
storage, network interface, and so on. To achieve optimal performance, it should distribute the
data evenly between the Cluster Nodes created. So that the load processed at each node is
equivalent, so that no Cluster Nodes are waiting for each other to return data to the Master.
Storage Interface
Storage Interface is an interface that mediates between text files (for example CSV) that are
uploaded with Deepgreen DB Cluster. The workings of this interface are to read the specified
directory and where in the directory contains CSV files that have been uploaded on the
database side an External Table will display the data read from the CSV file.

Backup and Restore
Basically, the Multi-Tier Backup mechanism consists of:
1. HA, or Hot-Standby, is another set of systems which standby to backup the
main server. Uses automatic or manual failover based on the requirement
2. Late Replication, which is an asynchronous replication to replica and
configured with delay for a couple of minutes.
3. Snapshot Data, which is done every Sunday at 01:00 (Configurable), and
keeps snapshots for the last 3 weeks.
4. WAL Archiving, which is done at any time and will retain the WAL Archive
for the last 3 weeks.
5. Data dump (if required), can be full or only the diff.

Community Contribution
As part of responsibility in building up the complete business ecosystem of
PostgreSQL, we are conducting some community contribution, development and
gathering:
1. Online Media Socialization
We conducted a series of publications in Social Media to educate the public and
market users of PostgreSQL about the benefit of using PostgreSQL in their
production system. Not only for business areas, we are also trying to reach out
into Academic and Government as well. Since we believe that ABG (Academic,
Business and Government) should be in synergy and the same line of vision.
2. Press Release Publication
PostgreSQL sometimes releases information to the press, Equnix provides
voluntary services for the Indonesian Community to spread the information in
the local market, as well as ASEAN markets. PostgreSQL community
organization is not mature yet, it is not yet formed and solid. There are some
communication groups, forums, but there are no ***
3. Guest Lecture
Equnix conducts guest lectures in many Universities in Indonesia, to promote
Open Source, and PostgreSQL specially. We believe that Open Source is the
great answer and option for our IT implementation in the modern world, and
we want our next generation of professional prepared and ready for the future.
We as part of the business player in the IT Sector should give feedback to the
Academic on what is happening in the Business Sectors.

Equnix Guest Lecture at Universitas Merdeka Malang
Equnix Guest Lecture at Universitas Sepuluh Nopember Surabaya

Equnix Guest Lecture at Universitas Udayana Bali
4. Meetup
Every couple of months, usually we held a local PostgreSQL Gathering as a
community event. We share technical as well as operational and PostgreSQL
implementation in mission critical production system.

5. Client Gathering
Equnix also promotes PostgreSQL not only in the community but also in
Professional manner, by conducting Client Gathering. Our Client Gathering
program is named as: EPPG - Equnix PostgreSQL Professional Gathering. This
Event is usually held for presenting hot issues, discussion, brainstorming and
networking to strengthen professional bonds in using PostgreSQL.
An Enterprise Professional PostgreSQL Gathering with the theme "Answering
the Challenges and Benefits of Migration from Oracle to PostgreSQL''
organized by PT Equnix Business Solutions on Wednesday, March 27, 2019 at the
Bunga Rampai Menteng Restaurant took place smoothly and successfully. The
Enterprise Professional PostgreSQL Gathering was attended by various national
companies in Indonesia, including from Banking, Telecommunications,
Corporate and Government industries. All of the guests were enthusiastic and
very keen on learning more about PostgreSQL. The general impression gained
from the participants is that this event is interesting, inspiring, and fun. It not
only strengthens relationships and provides networking opportunities for the
guests, but also provides views and insights regarding Oracle Migration to
PostgreSQL to them. The participants all appreciated the events, and would like
it to be held again in the following months, because many of them felt like they
benefited from this gathering.
Participants Enterprise PostgreSQL Professional Gathering from Adira Finance,
Telkom Akses, Truemoney Indonesia, Bank Jatim, Bank Central Asia (BCA), Bank
Mandiri, Bank Negara Indonesia (BNI), Bank Tabungan Negara (BTN), Bank
Bukopin, Bank Mega, PT Finansia Multi Finance, Prudential, EDI Indonesia, PT
Buditama Ciptamas Perkasa, PT Gatra Hita Wasana, PT Inova Medika Solusindo,
Telkomsel, Omni Hospital, FIF Group, Telkomsigma, Badan Kepegawaian
Negara (BKN), Badan Koordinasi Penanaman Modal (BKPM),
Pusat Sistem Informasi dan Teknologi Keuangan (PUSINTEK) Kementerian
Keuangan, Kementerian Agama, Tridaya Asira, AGCI, PT Dharma Electrindo
Manufacturing, Icon+, PT Global Infotech Solution.

6. International Speaker
As Speaker in some International PostgreSQL Conference about PostgreSQL:
PGDay, PGConf.APAC, PGConf.Asia, etc.
Mr Julyanto Sutandang was appointed by the PGConf ASIA 2017 as the Keynote
Speaker in Opening Ceremony on this International Event in Tokyo, Japan.

Mr Julyanto Sutandang was appointed as speaker at PGDay Singapore 2017
Mr Lucky Haryadi was giving talks on PGConf.Asia 2018, Tokyo, Japan
7. International Sponsor
Equnix has been supporting the PostgreSQL Community since 2014.
When PGCon in Ottawa held in 2014, Equnix has become Gold Sponsor, and any
other Conferences such as PGConf.Asia in Japan, PGDay.Asia in Singapore, etc.

Equnix become Gold Sponsor of PGConf ASIA 2017
Equnix become one of the Sponsor of PGDay Singapore 2017

Equnix become Gold Sponsor of PGConf ASIA 2018

PGConf.Asia 2019
Our next big event is the International Conference for PostgreSQL called PGConf.Asia, which
has been conducted in 3 consecutive years in Tokyo since 2016, 2017 and 2018, and this year
2019 will be held in Grand Inna Bali Beach, Indonesia.
This event is host by IDPUG (Indonesia PostgreSQL User Group) of which Equnix is the anchor
Founder of this Organization. We also invite another Organization to help us in hosting this
great event!
This International Conference invites speakers from all around the world: US, Russia, France,
Germany, Japan, Korea, Indonesia, Pakistan, India and Nepal. Total Speakers of 27 Peoples and
will be attended by 500-1000 Attendee, hopefully our government will open this Conference at
Year 2019 Month 09 Day 09 Hour 09 Minutes 09 and Second 19. (09:09:19 09-09-19)
Equnix proud to become part of it to make it happen. https://2019.pgconf.asia

PostgreSQL as Enterprise Solution v1.1.pdf

More Related Content

Similar to PostgreSQL as Enterprise Solution v1.1.pdf (20)

More from Equnix Business Solutions (20)

Recently uploaded (20)

PostgreSQL as Enterprise Solution v1.1.pdf