SlideShare a Scribd company logo
How can I be ready for the next
backup recovery of a Postgres
database?
Postgres continuous
backup and PITR with
Barman
Gabriele Bartolini
9 December 2020 - Postgres Build 2020
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.3
• PostgreSQL user since ~2000
• Community member since 2006
• 2ndQuadrant, from 2008 to 2020
• Head of Global Support
• Cloud Native Initiative Lead
• Founding member of Barman
• Now with EDB
About Gabriele Bartolini
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.4
Today’s menu
• About Barman
• Useful Patterns
• What lies ahead of us
• Final thoughts
Follow me:
@_GBartolini_
Tweet:
#pgbarman
#PostgresBuild2020
pgbarman.org
Eliel Saarinen
“Always design a thing by
considering it in its next larger
context – a chair in a room, a room in
a house, a house in an environment,
an environment in a city plan.”
About Barman
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.8
• Disaster Recovery of PostgreSQL databases
• Based on continuous physical backup
• First backup and recovery tool for PostgreSQL
• Conceived in 2011, open sourced in 2012
• Goal: foster migrations from Oracle
• Written in Python
• Open Source, distributed under GNU GPL 3
About Barman
Backup And Recovery MANager for PostgreSQL
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.9
Innovative concepts
First Introduced by Barman in the Postgres ecosystem
• Remote backup
• Remote recovery
• Multiple PostgreSQL servers
• Backup catalogue
• Independent WAL archive and hub
• WAL compression
• Multiple backup methods
• Incremental backup and recovery
• WAL streaming
• Retention policies
• Monitoring integration
• Hook scripts
Useful Patterns
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.11
• Make decisions based on:
• Recovery Point Objectives (RPO)
• Recovery Time Objectives (RTO)
• Your context is unique, and you are the expert there
• Technological aspects are important
• But they are just the means to our business goals
#0 - Focus on business goals
Let goals drive your initiatives, not the technical means
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.12
• Resilience
• Commodity storage
• Physical servers with high capacity local disks
• Integration with standby servers
• Zero data loss clusters
#1 - Separate Barman from Postgres
Let Barman and Postgres run on different servers and storage
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.13
• File lock issues
• Place “barman_lock_directory” in the local Linux disk
• Performance issues
• Latency
• Throughput (bottleneck)
• Shared storage (variability)
• Accentuated by large databases
#2 - Avoid network disks (if you can)
Some customers run Barman on NFS/CIFS/Ceph/…
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.14
• Better resilience of the local cluster
• Next tips will provide the whys and the hows
• Preferably dedicated network
• Data centre as a single point of failure?
• Yes, but ...
• … don’t let that scare you, it can be solved
#3 - Favour locality over distance
Let Barman and Postgres be next to each other, in the same data centre
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.15
• Ensure all WALs are shipped
• Rely on replication slots
• Fully automated with “create_slot” option
• Asynchronous replication by default
• Synchronous replication if …
• You add a standby server
#4 - WAL streaming
Let Postgres stream WAL records to Barman and reduce RPO
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.16
• On-demand remote fetching:
• Standby servers
• Recovery operations
• “barman-wal-restore”
• Parallel pre-fetch (performance boost)
• On the fly decompression
• Forget about configuring “wal_keep_segments” or “max_wal_size”
#5 - WAL fetching (get-wal)
Use Barman as an “infinite” hub of WAL files through “get-wal”
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.17
• Set “compression” option, globally
• Performed by the WAL archiver process
• Invoked automatically by the “cron” command
• gzip is normally fine
• Supported algorithms:
• bzip2
• pigz
• There’s more: custom compression methods
#6 - Compress your WALs
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.18
#7 - Your PostgreSQL building block
The “Flux Capacitor” Food for thought:
1. Primary instance
2. Standby instance
3. Barman instance
• High Availability
• Disaster Recovery
• Local standby servers
• You can always add more
• Simple architecture
• Yet very effective
• Symmetric architecture (next rule)
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.19
#8 - Multi-data centre architecture
Symmetric architecture is key
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.20
#9 - Encryption
Very important for GDPR compliance
Encrypt at rest Food for thought:
• PostgreSQL servers
• Backup servers
• Delegated to the operating system
• LUKS
• Volmetric
• Secure connections
• Secure access to backup servers
• Encryption of exported backup files:
• Tar files on tape
• Tar files on Cloud object stores
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.21
#10 - Constrain your RPO
Clearly define your Recovery Point Objective in PostgreSQL
• By default it is approximately a WAL file
• Recommendation: set it to a maximum of 5 minutes
• Use “archive_timeout”
• For example “archive_timeout = 5min”
• Important for geographical redundancy
• Beyond the local region
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.22
#11 - Reduce your RPO to 0
Trust PostgreSQL native synchronous streaming replication
• Requirements:
• A local standby
• A local Barman
• Two options:
• Zero data loss standby
• Zero data loss backup
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.23
#12 - Backup from a standby
Available from PostgreSQL 9.6
• Symmetric architecture
• Off-load the primary
• Redundancy
• Feature request: cluster
awareness
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.24
#13 - Let “check” be your compass
“barman check” is the most critical command
• It guides you
• Setup process
• Problem solving
• Integration with alerting tools
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.25
#14 - Monitoring
Observability is fundamental in business continuity environments
• Barman resides on a Linux system
• That system must be under monitoring
• Standard metrics
• Disk usage
• “barman check --nagios all”
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.26
#15 - Weekly backups
Define the frequency of backups through cron
• Start with weekly backups
• Evaluate daily backups if you require shorter RTO
• “barman backup all”
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.27
#16 - Retention policies
Key capability in disaster recovery solutions
• Automatically purge old backups
• Retention policies based on:
• Redundancy (quantity)
• Recovery window (time, Point of Recoverability)
• Food for thought:
• Delete hook scripts
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.28
#17 - Use rsync/SSH for backups
Barman has an optimised copy algorithm based on rsync
• Enables incremental copy
• Set “reuse_backup = link” if your file system supports hard links
• Enables parallel copy
• Set “parallel_jobs” option
• Used for incremental and parallel recovery too
• Real example from a production customer:
• Weekly backup of a 25TB database takes 13 hours to complete
• Reuses over 70% of the previous backup
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.29
#18 - Rely on object stores
By … relaying to AWS S3 compatible object stores
Food for thought:• Public/Private/Hybrid cloud
• You can use a gateway too
• Requires Boto3 library
• “barman-cloud-wal-archive” (WALs)
• “barman-cloud-backup” (Backups)
• Removing SPoF for data availability
• Data centre, Provider, Continent,
Planet, … that’s it for now
• Multi-tiered backup and recovery
• Enhanced DR capabilities
• Remember encryption!
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.30
#19 - Public cloud, via Barman
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.31
#20 - Public cloud, direct
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.32
#21 - Local object store with gateway
To be implemented
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.33
#22 - Geo-redundancy
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.34
#23 - Aggressive start
Enable “immediate_checkpoint”
• Speed up the start of the backup
• Request a checkpoint without waiting for the scheduled one
• Not available when backing up from a standby server
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.35
#24 - Precautions
Best practices that increase the resilience through “barman check”
• “minimum_redundancy”
• Safety measure: set it to 1
• “last_backup_maximum_age”
• Based on the backup frequency
• E.g. “1 week” in case of weekly backups
• “max_incoming_wals_queue”
• Let “barman check” fail if your incoming queue of WALs gets too high
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.36
#25 - Server configuration files
Ideal for configuration management tools and automation
• Use a separate configuration file per server
• .conf suffix is required
• Place the file in the /etc/barman.conf.d folder
• Hint: add the major PostgreSQL version as a suffix to the file (and server ID)
• SERVER_ID-PGVERSION.conf
• Example: chat-13.conf
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.37
#26 - Last version always wins
Always install the latest version of Barman
• Barman is backwards compatible
• Trunk based development
• Comprehensive set of automated tests
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.38
#27 - Enjoy convention over configuration
• Most options in Barman:
• Can be set globally in the configuration file
• Can be overridden at server level
• Have default values
• If you use our packages, system configuration is already taken care of
• User
• Cron
• Log rotation
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.39
#28 - Get hooked
Integrate Barman with external systems through hook scripts
• Available Before/After certain events
• Two types:
• Standard: in case of failure, no retry
• Retry: in case of failure, retry
• Typical: before WAL archive to relay WAL files in the Cloud
• See “barman-cloud-wal-archive” man page
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.40
#29 - Work in small batches
Configure the WAL archiver batch size
• WAL archiver is automatically run by the “cron” command
• Every minute
• Processes incoming WAL files and archives them compressed (if required)
• By default, the batch is unlimited
• Batch size can be set for both standard archiving and streaming channels
• Tune it based on the number of expected WALs between two cron runs
• Good value to start with is between 10 and 100
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.41
#30 - JSON output
• Every command supports “-f json”
• Integration with other applications
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.42
#31 - Principle Of Least Authority (POLA)
• Avoid using a superuser for Barman
• The “barman” user can be a standard user
• With specific grants for backup and read operations
• barman_streaming can be used for replication connections
• Requirements:
• PostgreSQL 10+
• Barman 2.11+
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.43
#32 - Periodically recover your backups
• Use post backup hook scripts to issue a remote recovery
• Use pre recovery scripts to control and prepare remote environments
• Use post recovery scripts to perform data reconciliation and transformation
• You can regenerate a standby server, a reporting/BI database or a QA environment
• The known unknowns:
• Will my backup work?
• How long will it take to recover from a backup (worst case scenario RTO)?
Use hook scripts to systematically recover and test your backups
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.44
#33 - Enhance gradually
• Can I achieve Disaster Recovery?
• Can I achieve High Availability?
• Can I achieve Business Continuity?
• What are my RPO and RTO?
• Start with your goals, add components gradually to improve them
• KISS
• Remove your next Single Point of Failure
Consider a cluster with just a PostgreSQL instance and a Barman instance
What lies ahead
of us
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.46
EDB powers Barman
EDB acquired 2ndQuadrant on 29 Sept 2020
• EDB is committed to Barman
• New leadership, management and dev team
• Original team stepping down after ~ 10 years
• We will serve as advisors
• Focus on Kubernetes integration
• Want a scoop?
• Abhijit Menon-Sen will lead Barman Development
• Sebastiaan Mannem will be the Product Manager
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.47
Some ideas
An anticipation of possible future developments
• Barman Operator for Kubernetes
• Stay tuned, lots of cool stuff around the corner!
• Tier-2: compressed/encrypted backups
• tar_retention_policy
• Tier-3: object store backups
• S3_retention_policy
• Seamless integration with pg_verifybackup
• Snapshot backups
Final thoughts
© Copyright EnterpriseDB Corporation, 2020. All rights reserved.49
Conclusion
• Use these patterns as enablers for dialog in your team.
• Enhance gradually! Focus on the goals, not the means!
• Constrain your RPO and don’t underestimate RPO=0 scenarios
• Test your backups. Always. Measure your RTO. Be ready for the next incident.
• Remember the “Flux capacitor”
• Barman is stable, robust & backwards compatible
• Barman is Open Source
Follow me:
@_GBartolini_
Tweet:
#pgbarman
#PostgresBuild2020
pgbarman.org

More Related Content

PDF
PostgreSQL WAL for DBAs
PDF
Blazing Performance with Flame Graphs
PDF
PostgreSQL Replication Tutorial
PDF
Replicação Lógica no PostgreSQL 10
PDF
Linux tuning to improve PostgreSQL performance
PPTX
Linux Network Stack
PDF
YOW2021 Computing Performance
PDF
Mastering PostgreSQL Administration
PostgreSQL WAL for DBAs
Blazing Performance with Flame Graphs
PostgreSQL Replication Tutorial
Replicação Lógica no PostgreSQL 10
Linux tuning to improve PostgreSQL performance
Linux Network Stack
YOW2021 Computing Performance
Mastering PostgreSQL Administration

What's hot (20)

PDF
Flink Forward Berlin 2018: Stefan Richter - "Tuning Flink for Robustness and ...
PDF
PostgreSQL replication
PDF
Understanding PostgreSQL LW Locks
PDF
Troubleshooting PostgreSQL Streaming Replication
PDF
PostgreSQL Deep Internal
PDF
Spark shuffle introduction
PDF
Postgresql database administration volume 1
PDF
Linux Profiling at Netflix
PDF
EnterpriseDB's Best Practices for Postgres DBAs
 
PDF
YOW2020 Linux Systems Performance
PDF
Apache Bigtop3.2 (仮)(Open Source Conference 2022 Online/Hiroshima 発表資料)
PPTX
Getting started with postgresql
PDF
Container Performance Analysis
PDF
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
PDF
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
PDF
PostgreSQL High Availability in a Containerized World
PPTX
PostGreSQL Performance Tuning
PDF
MyRocks Deep Dive
PDF
PostgreSQL and RAM usage
PDF
PostgreSQL Performance Tuning
Flink Forward Berlin 2018: Stefan Richter - "Tuning Flink for Robustness and ...
PostgreSQL replication
Understanding PostgreSQL LW Locks
Troubleshooting PostgreSQL Streaming Replication
PostgreSQL Deep Internal
Spark shuffle introduction
Postgresql database administration volume 1
Linux Profiling at Netflix
EnterpriseDB's Best Practices for Postgres DBAs
 
YOW2020 Linux Systems Performance
Apache Bigtop3.2 (仮)(Open Source Conference 2022 Online/Hiroshima 発表資料)
Getting started with postgresql
Container Performance Analysis
Virtual Flink Forward 2020: Autoscaling Flink at Netflix - Timothy Farkas
Tuning Apache Spark for Large-Scale Workloads Gaoxiang Liu and Sital Kedia
PostgreSQL High Availability in a Containerized World
PostGreSQL Performance Tuning
MyRocks Deep Dive
PostgreSQL and RAM usage
PostgreSQL Performance Tuning
Ad

Similar to PostgreSQL continuous backup and PITR with Barman (20)

PPTX
An overview of reference architectures for Postgres
 
PPTX
An overview of reference architectures for Postgres
 
PPTX
Database Dumps and Backups
 
PPTX
Automating a PostgreSQL High Availability Architecture with Ansible
 
PPTX
Beginners Guide to High Availability for Postgres
 
PDF
Managing Postgres at Scale With Postgres Enterprise Manager
 
PPTX
Enhancing Data Protection Workflows with Kanister And Argo Workflows
PDF
How to Integrate Hyperconverged Systems with Existing SANs
PPTX
Webinar: Enterprise Trends for Database-as-a-Service
PDF
Postgres & Red Hat Cluster Suite
 
PPTX
Beyond the Basics 1: Storage Engines
PDF
Updates to Apache CloudStack and LINBIT SDS
PDF
Webinar NETGEAR - Storagecraft e Netgear: soluzioni per il backup e il disast...
PDF
Cloud Migration Paths: Kubernetes, IaaS, or DBaaS
 
PDF
SNW Pres City Of Safford 101209 Lr
PPTX
Real-time analysis using an in-memory data grid - Cloud Expo 2013
PDF
Cloud Native PostgreSQL - APJ
 
PDF
PostgreSQL Disaster Recovery with Barman (PGConf.EU 2013)
PDF
Preparing your dockerised application for production deployment
PDF
Oracle Storage a ochrana dat
An overview of reference architectures for Postgres
 
An overview of reference architectures for Postgres
 
Database Dumps and Backups
 
Automating a PostgreSQL High Availability Architecture with Ansible
 
Beginners Guide to High Availability for Postgres
 
Managing Postgres at Scale With Postgres Enterprise Manager
 
Enhancing Data Protection Workflows with Kanister And Argo Workflows
How to Integrate Hyperconverged Systems with Existing SANs
Webinar: Enterprise Trends for Database-as-a-Service
Postgres & Red Hat Cluster Suite
 
Beyond the Basics 1: Storage Engines
Updates to Apache CloudStack and LINBIT SDS
Webinar NETGEAR - Storagecraft e Netgear: soluzioni per il backup e il disast...
Cloud Migration Paths: Kubernetes, IaaS, or DBaaS
 
SNW Pres City Of Safford 101209 Lr
Real-time analysis using an in-memory data grid - Cloud Expo 2013
Cloud Native PostgreSQL - APJ
 
PostgreSQL Disaster Recovery with Barman (PGConf.EU 2013)
Preparing your dockerised application for production deployment
Oracle Storage a ochrana dat
Ad

More from EDB (20)

PDF
Die 10 besten PostgreSQL-Replikationsstrategien für Ihr Unternehmen
 
PDF
Migre sus bases de datos Oracle a la nube
 
PDF
EFM Office Hours - APJ - July 29, 2021
 
PDF
Benchmarking Cloud Native PostgreSQL
 
PDF
Las Variaciones de la Replicación de PostgreSQL
 
PDF
NoSQL and Spatial Database Capabilities using PostgreSQL
 
PDF
Is There Anything PgBouncer Can’t Do?
 
PDF
Data Analysis with TensorFlow in PostgreSQL
 
PDF
Practical Partitioning in Production with Postgres
 
PDF
A Deeper Dive into EXPLAIN
 
PDF
IOT with PostgreSQL
 
PDF
A Journey from Oracle to PostgreSQL
 
PDF
Psql is awesome!
 
PDF
EDB 13 - New Enhancements for Security and Usability - APJ
 
PPTX
Comment sauvegarder correctement vos données
 
PDF
Cloud Native PostgreSQL - Italiano
 
PDF
New enhancements for security and usability in EDB 13
 
PPTX
Best Practices in Security with PostgreSQL
 
PDF
Best Practices in Security with PostgreSQL
 
PDF
EDB Postgres & Tools in a Smart City Project
 
Die 10 besten PostgreSQL-Replikationsstrategien für Ihr Unternehmen
 
Migre sus bases de datos Oracle a la nube
 
EFM Office Hours - APJ - July 29, 2021
 
Benchmarking Cloud Native PostgreSQL
 
Las Variaciones de la Replicación de PostgreSQL
 
NoSQL and Spatial Database Capabilities using PostgreSQL
 
Is There Anything PgBouncer Can’t Do?
 
Data Analysis with TensorFlow in PostgreSQL
 
Practical Partitioning in Production with Postgres
 
A Deeper Dive into EXPLAIN
 
IOT with PostgreSQL
 
A Journey from Oracle to PostgreSQL
 
Psql is awesome!
 
EDB 13 - New Enhancements for Security and Usability - APJ
 
Comment sauvegarder correctement vos données
 
Cloud Native PostgreSQL - Italiano
 
New enhancements for security and usability in EDB 13
 
Best Practices in Security with PostgreSQL
 
Best Practices in Security with PostgreSQL
 
EDB Postgres & Tools in a Smart City Project
 

Recently uploaded (20)

PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
PDF
KodekX | Application Modernization Development
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PPTX
Big Data Technologies - Introduction.pptx
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PDF
Unlocking AI with Model Context Protocol (MCP)
PDF
Encapsulation theory and applications.pdf
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Empathic Computing: Creating Shared Understanding
PDF
Modernizing your data center with Dell and AMD
PPTX
MYSQL Presentation for SQL database connectivity
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
Encapsulation_ Review paper, used for researhc scholars
PPT
Teaching material agriculture food technology
PDF
NewMind AI Weekly Chronicles - August'25 Week I
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
KodekX | Application Modernization Development
Digital-Transformation-Roadmap-for-Companies.pptx
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Big Data Technologies - Introduction.pptx
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Per capita expenditure prediction using model stacking based on satellite ima...
20250228 LYD VKU AI Blended-Learning.pptx
Unlocking AI with Model Context Protocol (MCP)
Encapsulation theory and applications.pdf
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Advanced methodologies resolving dimensionality complications for autism neur...
Empathic Computing: Creating Shared Understanding
Modernizing your data center with Dell and AMD
MYSQL Presentation for SQL database connectivity
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Encapsulation_ Review paper, used for researhc scholars
Teaching material agriculture food technology
NewMind AI Weekly Chronicles - August'25 Week I

PostgreSQL continuous backup and PITR with Barman

  • 1. How can I be ready for the next backup recovery of a Postgres database?
  • 2. Postgres continuous backup and PITR with Barman Gabriele Bartolini 9 December 2020 - Postgres Build 2020
  • 3. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.3 • PostgreSQL user since ~2000 • Community member since 2006 • 2ndQuadrant, from 2008 to 2020 • Head of Global Support • Cloud Native Initiative Lead • Founding member of Barman • Now with EDB About Gabriele Bartolini
  • 4. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.4 Today’s menu • About Barman • Useful Patterns • What lies ahead of us • Final thoughts
  • 6. Eliel Saarinen “Always design a thing by considering it in its next larger context – a chair in a room, a room in a house, a house in an environment, an environment in a city plan.”
  • 8. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.8 • Disaster Recovery of PostgreSQL databases • Based on continuous physical backup • First backup and recovery tool for PostgreSQL • Conceived in 2011, open sourced in 2012 • Goal: foster migrations from Oracle • Written in Python • Open Source, distributed under GNU GPL 3 About Barman Backup And Recovery MANager for PostgreSQL
  • 9. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.9 Innovative concepts First Introduced by Barman in the Postgres ecosystem • Remote backup • Remote recovery • Multiple PostgreSQL servers • Backup catalogue • Independent WAL archive and hub • WAL compression • Multiple backup methods • Incremental backup and recovery • WAL streaming • Retention policies • Monitoring integration • Hook scripts
  • 11. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.11 • Make decisions based on: • Recovery Point Objectives (RPO) • Recovery Time Objectives (RTO) • Your context is unique, and you are the expert there • Technological aspects are important • But they are just the means to our business goals #0 - Focus on business goals Let goals drive your initiatives, not the technical means
  • 12. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.12 • Resilience • Commodity storage • Physical servers with high capacity local disks • Integration with standby servers • Zero data loss clusters #1 - Separate Barman from Postgres Let Barman and Postgres run on different servers and storage
  • 13. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.13 • File lock issues • Place “barman_lock_directory” in the local Linux disk • Performance issues • Latency • Throughput (bottleneck) • Shared storage (variability) • Accentuated by large databases #2 - Avoid network disks (if you can) Some customers run Barman on NFS/CIFS/Ceph/…
  • 14. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.14 • Better resilience of the local cluster • Next tips will provide the whys and the hows • Preferably dedicated network • Data centre as a single point of failure? • Yes, but ... • … don’t let that scare you, it can be solved #3 - Favour locality over distance Let Barman and Postgres be next to each other, in the same data centre
  • 15. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.15 • Ensure all WALs are shipped • Rely on replication slots • Fully automated with “create_slot” option • Asynchronous replication by default • Synchronous replication if … • You add a standby server #4 - WAL streaming Let Postgres stream WAL records to Barman and reduce RPO
  • 16. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.16 • On-demand remote fetching: • Standby servers • Recovery operations • “barman-wal-restore” • Parallel pre-fetch (performance boost) • On the fly decompression • Forget about configuring “wal_keep_segments” or “max_wal_size” #5 - WAL fetching (get-wal) Use Barman as an “infinite” hub of WAL files through “get-wal”
  • 17. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.17 • Set “compression” option, globally • Performed by the WAL archiver process • Invoked automatically by the “cron” command • gzip is normally fine • Supported algorithms: • bzip2 • pigz • There’s more: custom compression methods #6 - Compress your WALs
  • 18. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.18 #7 - Your PostgreSQL building block The “Flux Capacitor” Food for thought: 1. Primary instance 2. Standby instance 3. Barman instance • High Availability • Disaster Recovery • Local standby servers • You can always add more • Simple architecture • Yet very effective • Symmetric architecture (next rule)
  • 19. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.19 #8 - Multi-data centre architecture Symmetric architecture is key
  • 20. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.20 #9 - Encryption Very important for GDPR compliance Encrypt at rest Food for thought: • PostgreSQL servers • Backup servers • Delegated to the operating system • LUKS • Volmetric • Secure connections • Secure access to backup servers • Encryption of exported backup files: • Tar files on tape • Tar files on Cloud object stores
  • 21. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.21 #10 - Constrain your RPO Clearly define your Recovery Point Objective in PostgreSQL • By default it is approximately a WAL file • Recommendation: set it to a maximum of 5 minutes • Use “archive_timeout” • For example “archive_timeout = 5min” • Important for geographical redundancy • Beyond the local region
  • 22. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.22 #11 - Reduce your RPO to 0 Trust PostgreSQL native synchronous streaming replication • Requirements: • A local standby • A local Barman • Two options: • Zero data loss standby • Zero data loss backup
  • 23. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.23 #12 - Backup from a standby Available from PostgreSQL 9.6 • Symmetric architecture • Off-load the primary • Redundancy • Feature request: cluster awareness
  • 24. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.24 #13 - Let “check” be your compass “barman check” is the most critical command • It guides you • Setup process • Problem solving • Integration with alerting tools
  • 25. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.25 #14 - Monitoring Observability is fundamental in business continuity environments • Barman resides on a Linux system • That system must be under monitoring • Standard metrics • Disk usage • “barman check --nagios all”
  • 26. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.26 #15 - Weekly backups Define the frequency of backups through cron • Start with weekly backups • Evaluate daily backups if you require shorter RTO • “barman backup all”
  • 27. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.27 #16 - Retention policies Key capability in disaster recovery solutions • Automatically purge old backups • Retention policies based on: • Redundancy (quantity) • Recovery window (time, Point of Recoverability) • Food for thought: • Delete hook scripts
  • 28. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.28 #17 - Use rsync/SSH for backups Barman has an optimised copy algorithm based on rsync • Enables incremental copy • Set “reuse_backup = link” if your file system supports hard links • Enables parallel copy • Set “parallel_jobs” option • Used for incremental and parallel recovery too • Real example from a production customer: • Weekly backup of a 25TB database takes 13 hours to complete • Reuses over 70% of the previous backup
  • 29. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.29 #18 - Rely on object stores By … relaying to AWS S3 compatible object stores Food for thought:• Public/Private/Hybrid cloud • You can use a gateway too • Requires Boto3 library • “barman-cloud-wal-archive” (WALs) • “barman-cloud-backup” (Backups) • Removing SPoF for data availability • Data centre, Provider, Continent, Planet, … that’s it for now • Multi-tiered backup and recovery • Enhanced DR capabilities • Remember encryption!
  • 30. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.30 #19 - Public cloud, via Barman
  • 31. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.31 #20 - Public cloud, direct
  • 32. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.32 #21 - Local object store with gateway To be implemented
  • 33. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.33 #22 - Geo-redundancy
  • 34. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.34 #23 - Aggressive start Enable “immediate_checkpoint” • Speed up the start of the backup • Request a checkpoint without waiting for the scheduled one • Not available when backing up from a standby server
  • 35. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.35 #24 - Precautions Best practices that increase the resilience through “barman check” • “minimum_redundancy” • Safety measure: set it to 1 • “last_backup_maximum_age” • Based on the backup frequency • E.g. “1 week” in case of weekly backups • “max_incoming_wals_queue” • Let “barman check” fail if your incoming queue of WALs gets too high
  • 36. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.36 #25 - Server configuration files Ideal for configuration management tools and automation • Use a separate configuration file per server • .conf suffix is required • Place the file in the /etc/barman.conf.d folder • Hint: add the major PostgreSQL version as a suffix to the file (and server ID) • SERVER_ID-PGVERSION.conf • Example: chat-13.conf
  • 37. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.37 #26 - Last version always wins Always install the latest version of Barman • Barman is backwards compatible • Trunk based development • Comprehensive set of automated tests
  • 38. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.38 #27 - Enjoy convention over configuration • Most options in Barman: • Can be set globally in the configuration file • Can be overridden at server level • Have default values • If you use our packages, system configuration is already taken care of • User • Cron • Log rotation
  • 39. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.39 #28 - Get hooked Integrate Barman with external systems through hook scripts • Available Before/After certain events • Two types: • Standard: in case of failure, no retry • Retry: in case of failure, retry • Typical: before WAL archive to relay WAL files in the Cloud • See “barman-cloud-wal-archive” man page
  • 40. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.40 #29 - Work in small batches Configure the WAL archiver batch size • WAL archiver is automatically run by the “cron” command • Every minute • Processes incoming WAL files and archives them compressed (if required) • By default, the batch is unlimited • Batch size can be set for both standard archiving and streaming channels • Tune it based on the number of expected WALs between two cron runs • Good value to start with is between 10 and 100
  • 41. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.41 #30 - JSON output • Every command supports “-f json” • Integration with other applications
  • 42. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.42 #31 - Principle Of Least Authority (POLA) • Avoid using a superuser for Barman • The “barman” user can be a standard user • With specific grants for backup and read operations • barman_streaming can be used for replication connections • Requirements: • PostgreSQL 10+ • Barman 2.11+
  • 43. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.43 #32 - Periodically recover your backups • Use post backup hook scripts to issue a remote recovery • Use pre recovery scripts to control and prepare remote environments • Use post recovery scripts to perform data reconciliation and transformation • You can regenerate a standby server, a reporting/BI database or a QA environment • The known unknowns: • Will my backup work? • How long will it take to recover from a backup (worst case scenario RTO)? Use hook scripts to systematically recover and test your backups
  • 44. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.44 #33 - Enhance gradually • Can I achieve Disaster Recovery? • Can I achieve High Availability? • Can I achieve Business Continuity? • What are my RPO and RTO? • Start with your goals, add components gradually to improve them • KISS • Remove your next Single Point of Failure Consider a cluster with just a PostgreSQL instance and a Barman instance
  • 46. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.46 EDB powers Barman EDB acquired 2ndQuadrant on 29 Sept 2020 • EDB is committed to Barman • New leadership, management and dev team • Original team stepping down after ~ 10 years • We will serve as advisors • Focus on Kubernetes integration • Want a scoop? • Abhijit Menon-Sen will lead Barman Development • Sebastiaan Mannem will be the Product Manager
  • 47. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.47 Some ideas An anticipation of possible future developments • Barman Operator for Kubernetes • Stay tuned, lots of cool stuff around the corner! • Tier-2: compressed/encrypted backups • tar_retention_policy • Tier-3: object store backups • S3_retention_policy • Seamless integration with pg_verifybackup • Snapshot backups
  • 49. © Copyright EnterpriseDB Corporation, 2020. All rights reserved.49 Conclusion • Use these patterns as enablers for dialog in your team. • Enhance gradually! Focus on the goals, not the means! • Constrain your RPO and don’t underestimate RPO=0 scenarios • Test your backups. Always. Measure your RTO. Be ready for the next incident. • Remember the “Flux capacitor” • Barman is stable, robust & backwards compatible • Barman is Open Source