Trivadis TechEvent 2017 ACFS Replication as of 12 2 by Mathias Zarick

BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
ACFS Replication as of 12.2
Mathias Zarick, Regensdorf, Techevent September 2017

Agenda – ACFS Replication as of 12.2
ACFS Replication as of 12.22 9/15/2017
1. Retrospect 11.2 to 12.1
2. Grid Infrastructure 12.2
3. ACFS Replication 12.2

Retrospect 11.2 to 12.1

ACFS – part of Grid Infrastructure
- Available as of 11.2.0.1 as a general purpose cluster filesystem, based on ASM
- In ASM disk groups automatic dynamic volumes (ADVM) are created
- On those block devices an ACFS filesystem is created (special kernel modules)
ASM Disk Group
grid@red:~/ [+ASM1] ls -l /dev/asm/*
brwxrwx--- 1 root asmadmin 252, 67073 Mar 21 12:37 /dev/asm/clusterfs-131
grid@red:~/ [+ASM1] mount | grep clusterfs
/dev/asm/clusterfs-131 on /u00/app/oracle/clusterfs type acfs (rw)
ADVM - CLUSTERFS

ACFS History till 12.1
Sep2009: Introduction with 11.2.0.1 – read only snapshots
At that time: “Oracle Cloud File System” license if used for application data
Sep2010: 11.2.0.2 brought ACFS replication, tagging, security, encryption
Changed conditions: License only needed if using such “higher” features
Sep2011: 11.2.0.3 – read write snapshots, backup pieces and data pump dumps
supported
Jun2013: 12.1.0.1 all kinds of db files supported to be stored on ACFS (default on
ODA)
Aug2013: 11.2.0.4 no ACFS enhancements
Sep2014: License not needed any more / product does not exist any more
Jul2014: 12.1.0.2 ACFS support on Exadata

Licenses as of 2017 – 12.2 License Guide
The product “Oracle Cloud File System” does not exist anymore
If used for non db files no license is needed
If used for db files an Oracle database EE license is needed only in case snapshots
are used
ACFS auditing, encryption, replication, security not available for the use with db files
Support for ACFS is only provided if server operates with any Oracle product, which
is under contract, e.g. Oracle database, Oracle Linux, Solaris

ACFS Replication till 12.1
Primary and standby file systems (only 1 standby is supported)
File system changes and captured and recorded in files called replication logs
(visible on .ACFS/repl subdirectory)
Logs are transported via Oracle Net (dbms_backup_restore.networkFileTransfer)
CRS background processes take care of transport, apply and monitor actions
Only between clusters – Oracle Restart not supported (Workaround: use 1-node
clusters)
No support in conjunction with encryption or security features

ACFS Replication Setup 12.1.0.2 (1)
ASM disk group with ‘compatible.asm’ and ‘compatible.advm’ set to 11.2.0.3 or higher
Setup is typically performed by grid user, certain steps are performed by root
Optional: create replication user in ASM instance with sysdba and sysasm privileges
tnsnames.ora entries:
# ACFS replication, italy is SCAN name
# service name ACFS_IMAGES_FS is not yet registered
ACFS_IMAGES_FS_ITALY.trivadistraining.com =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = italy)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = ACFS_IMAGES_FS)
)
)

ACFS Replication with Wallets ?
By default: password of replication or sys user is stored in OCR and used during
Oracle Net transportation (by dbms_backup_restore) for authentication
As of docs it should be possible to better hide the password in a wallet, secure
external password store
It never worked (at least for me), worked on several SRs without success

Grid Infrastructure 12.2

Some General Words about the New Version
For RAC-RDBMS homes you should have a new group (e.g. racdba) for new
OSRAC system privileges group.
Members – typically user oracle – get the new SYSRAC administrative privilege,
which is used by oraagent (not SYSDBA anymore, least privileges principle)
Flex ASM is mandatory, even on NFS !!!
No cluster without ASM anymore – at least for standalone clusters.
Image based installation: unpack, then patch, then configure
You can separate disk groups for OCR and Voting (+GRID) and for GIMR (+MGMT)
At least 36 GB usable space for +MGMT – better more

Grid Infrastructure Installation 12.2
Unpack, patch and start configuration
# as grid
unzip /u02/stage/V840012-01.zip -d /u00/app/grid/product/12.2.0.1
cd /u00/app/grid/product/12.2.0.1
rm -rf OPatch
unzip /u02/stage/p6880880_121010_Linux-x86-64.zip # OPatch update
./gridSetup.sh -applyPSU /u02/stage/26133434/ # Revision up. Jul 2017
./gridSetup.sh -applyOneOffs ... # if needed
./gridSetup.sh

Annoying Bug in GI 12.2
When running root.sh it configures the cluster. Sure, but why it takes ages …?
First run took 46 min, would have taken even longer if I did not interfere
Second run on same hardware with special approach (see below) took 16 min.
Problem is bug 25717212 in CHM component, gdb processes eat up all CPU
Workaround:
“Unexpected clusterware crashes ... 12.2 Grid Infrastructure” (Doc ID 2251437.1)
“12.2 Grid Infrastructure .. Upgrade steps … Exadata …” (Doc ID 2111010.1)
# during step "Executing installation step 18 of 19: 'ConfigNode'"
<GRID_HOME>/bin/oclumon manage -disable diagsnap
killall gdb

ACFS Replication 12.2

Fundamental Architecture Change
Replication is still asynchronous and is now based on
– Snapshots
– Ability to capture and apply differences between snapshots
– SSH (also on Windows – another “how to install Cygwin guide” in Oracle’s docs )
so the traffic it is definitely encrypted
Interesting part from docs:
A primary site running Linux, Solaris or AIX can replicate to a standby site running
any of those operating systems. A primary site running Windows can replicate only to
a standby site running Windows.
Still only one standby file system supported
Tags for subset replication are still supported

Snapshot Based Replication (1)
Primary file system
Standby file system
Last snapshots on both file systems represent same contents of data (with exception
of not replicated data of course, see tagging)
t1
t1

Primary file system
Standby file system
Create a new snapshot t2 on primary site
t1
t1
t2

Primary file system
Standby file system
Extract the data changes between t2 and t1, a so called snapshot duplication stream
t1
t1
t2
𝑑𝑎𝑡𝑎 𝑡2 − 𝑡1 = 𝑓(𝑑𝑎𝑡𝑎 𝑡1 , 𝑑𝑎𝑡𝑎 𝑡2 )

Primary file system
Standby file system
Snapshot duplication stream is applied to standby file systems,
t1 snapshot is referenced, t2 snapshot is the result
t1
t1
t2
t2

ACFS Replication Setup 12.2 (1)
ASMCA supports creation of disk groups and ACFS, but for the actual replication
setup command line is used
Create a disk group with ‘compatible.asm’ and ‘compatible.advm’ set to at least 12.2
Setup is basically the same, few differences regarding used parameters
– Create an replication apply user on OS (repluser)
– Exchange ssh keys between root on primary and repluser on standby (also in
other direction in case role changes are planned)
– Create volume and file system on both clusters
– Initialize replication on standby then on primary

Repluser needs to have OSASM right by OS group (typically asmadmin)
Generate ssh keys for root user on all primary cluster nodes and insert all public
keys of those in ~repluser/.ssh/authorized_keys on all standby nodes (docs claim we
should use authorized_keys2, but this is not working in recent version of SSH
anymore)
Root on primary needs to be able to logon as repluser on standby without password
Test all connections to produce .ssh/know_hosts entries, also use (SCAN) VIPs
# useradd -g oinstall -G oinstall,asmadmin,asmdba,asmoper -u 700 repluser
[root@zam101 ~]# ssh repluser@zam102

Validation as root, I use SCAN here, known_hosts file is extended by this call if needed
Or simply
It is comfortable (I would say even recommended) if all nodes in a cluster share the
same SSH host keys, else this will happen after a VIP moves to other cluster node
To set this up copy over /etc/ssh/ssh_host* files and restart sshd
acfsutil repl info -c -v -u repluser cluster102 /u00/app/grid/acfsmounts/images_fs
A valid 'ssh' connection was detected for standby node cluster102 as user repluser.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
ssh -o StrictHostKeyChecking=no repluser@cluster102

Docs: consider to lower sshd log level not to flood your syslog
Oracle recommend to set it to ERROR, default is INFO
– On a RHEL7 / OL7 this does not change anything, as systemd logs user sessions
– There is a workaround: create a discard filter with rsyslog (see notes)
– See https://access.redhat.com/solutions/1564823

File system setup as usual – grid user
+APX instances with 1G memory target are started on all nodes in cluster, which
mount the ACFS
asmcmd volcreate -G ACFS -s 100G images_fs
sudo /sbin/mkfs -t acfs /dev/asm/images_fs-*
sudo srvctl add filesystem -device /dev/asm/images_fs-*
-path /u00/app/grid/acfsmounts/images_fs -user grid,oracle
srvctl start filesystem -device /dev/asm/images_fs-*

Unmount file systems on both sides on all but one node
Again first the standby is initialized
Then primary, here you can configure with “acfsutil repl init primary”
– A refresh interval or constant mode (-i or -C)
– Compression vs. no compression (-z on or -z off)
– Tags
– Debug level (default is 3)
– Some ssh options
sudo acfsutil repl init standby -u repluser
/u00/app/grid/acfsmounts/images_fs

Example with tag “image_data”, compression (-z) and constant mode (-C)
Example without tag and 2 hours refresh interval (-i)
File system mount point is referenced twice (source, target), because it is allowed to
be different on both clusters
Now mount the file systems on other nodes
sudo acfsutil repl init primary -C -s repluser@cluster102
-m /u00/app/grid/acfsmounts/images_fs -z on image_data
-o sshStrictKey=n /u00/app/grid/acfsmounts/images_fs
sudo acfsutil repl init primary -i 2h -s repluser@cluster102
-o sshStrictKey=n -m /u00/app/grid/acfsmounts/images_fs

ACFS Replication Resource
In CRS there is 1 new resource (former architecture: 3 on primary, 4 on standby)
Log on primary (quite silent in debug level 0,1, more details starting with 2)
Logs on standby (one per sync, has its own debug level defined)
acfsutil repl bg info /u00/app/grid/acfsmounts/images_fs
Resource: ora.repl.dupd.acfs.images_fs.acfs
Target State: ONLINE
Current State: ONLINE on zam101
<ORACLE_BASE>/diag/crs/<hostname>/crs/trace/sdapply_<pid>.trc
<ORACLE_BASE>/diag/crs/<hostname>/crs/trace/acfsdupd_images_fs-117.trc

This is What Happens During Sync
1. create snapshot, 2. extract duplication stream and apply it, 3.delete snapshot
acfsutil snap create -r -R -S REPL_BASEFS_1501851997
acfsutil snap dup create -R -i REPL_BASEFS_1501851476
REPL_BASEFS_1501851997 /u00/app/grid/acfsmounts/images_fs |
ssh -o BatchMode=true -o StrictHostKeyChecking=no
-o Ciphers=aes128-ctr -x repluser@cluster102
acfsutil snap dup apply -R -b '/u00/app/grid/acfsmounts/images_fs'
acfsutil snap delete -R -S REPL_BASEFS_1501851476
UNIX
times
STDOUT is piped
as STDIN to
remote command

Some Explanations and Considerations
-R and -S for snapshot creation and deletion are not documented, my assumption
– -R means “special snapshot for replication purposes”
– -S silent, no output during snapshot creation
“snap dup create|apply” are documented and can be used for similar purposes also
on our own (on same server/cluster but different file system(s), from/to several local
or remote file systems, …),
create writes to STDOUT, apply reads from STDIN
-b option of apply creates a backup snapshot on standby, e.g.
SDBACKUP_1503568991_REPL_BASEFS_1503568962
which represents same contents as the corresponding snapshot on primary, e.g.
REPL_BASEFS_1503568962
If you enable compression this leads to SSH compression (-C option)

Manual and Automatic Sync
Manual sync
acfsutil repl sync /u00/app/grid/acfsmounts/images_fs
primary ACFS changed
since last refresh
noop or
sleep a
second
no
current time – last refresh
> configured interval
yes
no
sync
yes
Constant mode
– means that
interval is 0
– file system
with
permanent
write IO 
endless sync
loop
sleep
until X
Interval mode

More Facts about Snapshot Duplication Streams
An empty ACFS file system (freshly formatted) can be target of a complete sync of
any source ACFS file system
Afterwards only incremental applies (from snapshot to snapshot) are allowed
The target file system must not be manipulated, no IO writes at all! If you want to
allow subsequent incremental syncs
– Manual approach: no safeguard against this
– Regular replication: IO writes are not allowed by ACFS kernel modules on standby
– As of documentation a remaster to former snapshot should allow continuation of
syncs, but in my tests … it did not work
-b option should be used to know the currentness of target file system

More Facts about ACFS and ACFS Replication
Standby file system is read only
Regular replication uses a backup snapshot at standby file system
You can create additional custom snapshots (read only or read write) – also on the
standby file system for testing workloads (like a snapshot standby)
Information about ACFS file systems can be queried (via proxy instance +APX..)
– v$asm_acfsvolumes
– v$asm_filesystem
– v$asm_acfssnapshots
– v$asm_acfsrepl (only useful for former architecture)
– v$asm_acfstag

ACFS Replication – Operation (1)
Checks
Comparison on primary – standby must be mounted locally, e.g. nfs
Resize – volume and file system is increased (5 times if compatible.advm < 11.2.0.4)
acfsutil repl info -c /u00/app/grid/acfsmounts/images_fs
acfsutil repl bg info /u00/app/grid/acfsmounts/images_fs
acfsutil repl compare /u00/app/grid/acfsmounts/images_fs /nfs/images_fs
acfsutil size +10G /u00/app/grid/acfsmounts/images_fs

ACFS Replication – Operation (2)
Snapshots status
Replication status
IO changes on file system per time? Which bandwidth is needed?
Upgrade from old replication?
– After cluster is upgraded, replication is still old style
– Either terminate and rebuild or use “acfsutil repl upgrade”
acfsutil snap info /u00/app/grid/acfsmounts/images_fs
acfsutil repl info -c /u00/app/grid/acfsmounts/images_fs
acfsutil info fs -s /u00/app/grid/acfsmounts/images_fs

ACFS Replication – Role Changes?
No fully integrated operations for switchover or failover
Failover is like this
Switchover like this
– Stop IO on primary
– Last sync
– Terminate replication on both sides
– Setup replication in the other direction (complete sync)
sudo acfsutil repl terminate standby /u00/app/grid/acfsmounts/images_fs

Use Cases?
High availability for database with external files, e.g. BFILES
– Data Guard (database) + ACFS replication (ext. files) (better performance) or
– Data Guard + DBFS for files (better consistency)
HA for application file systems (cross platform)
HA for GoldenGate trail files
Can be useful for deployment/cloning methods for application file systems (cross
platform)

ACFS Replication – Conclusion
Fundamental change in architecture
Platforms can be mixed (but
Windows stays an island)
Not only automatic replication, sync
mechanisms can be used for
custom deployment tasks
Uncertain about synchronicity on file
systems with high IO utilization
DBFS is an alternative solution in case
you need more consistency and
synchronicity between database and file
system
No support for database files  !

Further information …
Oracle © Automatic Storage Management Administrator's Guide 12c Release 2 (12.2)
http://docs.oracle.com/database/122/OSTMG/understand-acfs-admin.htm
http://docs.oracle.com/database/122/OSTMG/configure-acfs-replication.htm

Questions and answers …
Mathias Zarick
Mathias.Zarick@trivadis.com
9/15/2017 ACFS Replication as of 12.243

Session Feedback – now
Please use the Trivadis Events mobile app to give feedback on each session
Use "My schedule" if you have registered for a session
Otherwise use "Agenda" and the search function
If the mobile app does not work (or if you have a Windows smartphone), use your
smartphone browser
– URL: http://trivadis.quickmobileplatform.eu/
– User name: <your_loginname> (such as "svv")
– Password: sent by e-mail...
9/15/2017 ACFS Replication as of 12.244

Trivadis TechEvent 2017 ACFS Replication as of 12 2 by Mathias Zarick

More Related Content

What's hot (20)

Similar to Trivadis TechEvent 2017 ACFS Replication as of 12 2 by Mathias Zarick (20)

More from Trivadis (20)

Recently uploaded (20)

Trivadis TechEvent 2017 ACFS Replication as of 12 2 by Mathias Zarick