Shareplex Presentation

Configuring Replication with high-
availability RAC clusters.
David Breinholt
Kerry Howell

Break the Ice
'I'm the key DBA in my team. I'm the one that locks up the office
at night'

Outline
• Our internal requirements
• The challenges of 99.9% uptime
• Our architecture
• How we setup SharePlex
• How we configured SharePlex
• CRS Integration
• Useful commands
• Final Thoughts

Service Level Agreement
• Must maintain 99.9% availability
• Maximum allowed outage of 43.8 minutes/month
• Not allowed to combine monthly allocations
• Financial penalties will be paid for overages
• Zero scheduled downtime allowed
This product is Highly visible to Executive team, so it is
in our best interest to proactively meet the
requirements of the Service Level Agreement.

• Upgrade Oracle – upgrade inactive silo, failover
• Patch Oracle – patch inactive silo, failover
• Quarterly software releases – inactive silo, failover
• Hardware failure/updates - failover to inactive silo
• Software failure – failover to inactive silo
• User error – use flashback table/database
• Disaster recovery – storage replication to NJ site
• OS Patches/upgrades
… all with near zero downtime!
The Challenge

Architecture
inst_a1 inst_a2
UT - Silo A - RAC Cluster
inst_b1 inst_b2
UT - Silo B - RAC Cluster
Logical
Replication
inst_a1 inst_a2
NJ - Silo A - RAC Cluster
inst_b1 inst_b2
NJ - Silo B - RAC Cluster
Block Level
Storage Replication
Block Level
Storage Replication
Logical
Replication
Bi-directional

Research
• RAC
– Upgrading with no downtime is questionable
– Does not facilitate our quarterly releases methodology
• Data Guard
– Upgrading with no downtime is questionable
– Target site is down (or read-only) when propagating changes
– Active data guard is additional cost
– Switchover takes minutes+ to complete
• Golden Gate
– Ability to compare/repair data is an add-on (expensive)
– Could not justify the cost of the product missing functionality
• SharePlex
– Seemed to satisfy all our requirements

Decision
During our discovery effort it was determined that
SharePlex was the best solution for the following
reasons:
• Intuitive setup and configuration
• Cost effective compared to other solutions
• Detects when table get out of sync
• Includes the ability to compare and repair tables
• Allows bi-directional replication

Funny
And afterward… Like most men… He performed a rollback and
never committed…
And…

Setup
In the source environment:
Create the required tablespaces
CREATE TABLESPACE SPLEXDAT DATAFILE SIZE 50M AUTOEXTEND ON NEXT 100M MAXSIZE
5G LOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE BLOCKSIZE 8K SEGMENT SPACE
MANAGEMENT AUTO FLASHBACK ON;
CREATE TABLESPACE SPLEXIDX DATAFILE SIZE 50M AUTOEXTEND ON NEXT 100M MAXSIZE
5G LOGGING EXTENT MANAGEMENT LOCAL AUTOALLOCATE BLOCKSIZE 8K SEGMENT SPACE
MANAGEMENT AUTO FLASHBACK ON;

Setup - Continued
Obtain new IP addresses in the same network as
the SCAN interface IP addresses (for each cluster):
Utah:
Silo A: 192.168.1.10
Silo B: 192.168.1.20
New Jersey:
Silo A: 192.168.2.10
Silo B: 192.168.2.20

Setup - Continued
Obtain DNS entries for the new IP addresses:
Utah:
192.168.1.10 - inst_a-splex-vip.xactware.com
192.168.1.20 - inst_b-splex-vip.xactware.com
New Jersey:
192.168.2.10 - njinst_a-splex-vip.xactware.com
192.168.2.20 - njinst_b-splex-vip.xactware.com

Setup - Continued
Create the CRS VIP to allow SharePlex to failover between RAC nodes:
Utah Silo A (only on one of the nodes)
as root:
${CRS_HOME}/bin/appvipcfg create -network=1 -ip=192.168.1.10
-vipname=shareplex.vip -user=oracle -group=dba
as oracle:
crsctl start resource shareplex.vip
Utah Silo B (only on one of the nodes):
as root:
${CRS_HOME}/bin/appvipcfg create -network=1 -ip=192.168.1.20
-vipname=shareplex.vip -user=oracle -group=dba
as oracle:
crsctl start resource shareplex.vip

Setup - Continued
Add SharePlex environment variables to .bashrc.*
Utah Silo A:
SP_SYS_HOST_NAME="inst_a-splex-vip.xactware.com" ; export SP_SYS_HOST_NAME
SP_INSTALL_DIR="/u02/app/quest"; export SP_INSTALL_DIR
SP_SYS_VARDIR="${SP_INSTALL_DIR}/vardir" ; export SP_SYS_VARDIR
SP_SYS_PRODDIR="${SP_INSTALL_DIR}/sp" ; export SP_SYS_PRODDIR
SP_OCT_ASM_SID="+ASM1" ; export SP_OCT_ASM_SID - (different for each host in the cluster)
Utah Silo B:
SP_SYS_HOST_NAME="inst_b-splex-vip.xactware.com" ; export SP_SYS_HOST_NAME
SP_INSTALL_DIR="/u02/app/quest"; export SP_INSTALL_DIR
SP_SYS_VARDIR="${SP_INSTALL_DIR}/vardir" ; export SP_SYS_VARDIR
SP_SYS_PRODDIR="${SP_INSTALL_DIR}/sp" ; export SP_SYS_PRODDIR
SP_OCT_ASM_SID="+ASM1" ; export SP_OCT_ASM_SID - (different for each host in the cluster)

Setup - Continued
Create Silo A TNS Entries:
Utah Silo A – Node 1:
INST_A_SP =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = utdb01a)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = inst_a)
(INSTANCE_NAME = inst_a1)
)
)
Utah Silo A – Node 2:
INST_A_SP =
(DESCRIPTION =
(CONNECT_DATA =
(SERVICE_NAME = inst_a)
(INSTANCE_NAME = inst_a2)
)
)

Setup - Continued
Create Silo B TNS Entries:
Utah Silo B – Node 1:
INST_B_SP =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = utdb01b)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = inst_b)
(INSTANCE_NAME = inst_b1)
)
)
Utah Silo B – Node 2:
INST_B_SP =
(DESCRIPTION =
(CONNECT_DATA =
(SERVICE_NAME = inst_b)
(INSTANCE_NAME = inst_b2)
)
)

Setup - Continued
Create the required oratab entries:
For Silo A (all nodes):
inst_a:/u01/app/oracle/product/11.2.0/dbhome_1:N
inst_a_sp:/u01/app/oracle/product/11.2.0/dbhome_1:N
For Silo B (all nodes):
inst_b:/u01/app/oracle/product/11.2.0/dbhome_1:N
inst_b_sp:/u01/app/oracle/product/11.2.0/dbhome_1:N
Make sure the entries look like this:
inst_b_sp:/u01/app/oracle/product/11.2.0/dbhome_1:N
Not this:
inst_b_sp:/u01/app/oracle/product/11.2.0/dbhome_1:N # Line added by agent

Setup - Continued
Create and mount an NFS share for each RAC cluster.
This mount point will contain the SharePlex installation
and replication configuration files.
You must use the following mount options:
rw,hard,proto=tcp,rsize=32768,wsize=32768,intr,timeo=600,retrans=2,vers=3,llock,suid
See:
https://support.software.dell.com/shareplex-for-oracle/kb/30450

HA HA!!!
There are 2 types of DBA’s
1. Those who do backups
2. Those who will do backups…
Q: What did the DBA say to the developer?
A: It doesn’t matter, he wasn’t listening anyway.

Setup - Continued
Install SharePlex (on one node of each cluster):
cd /dba/software/quest/splex
tar -xf ./SharePlex-8.0.1-b40-oracle110-sun-11-sparc-m64.tar
./SharePlex-8.0.1-b40-oracle110-sun-11-sparc-m64.tpm
Please enter the product directory location? /u02/app/quest/sp (/u02 is the NFS mount created earlier)
Please enter the variable data directory location? /u02/app/quest/vardir
Please specify the SharePlex Admin group (select a number):
1. [oinstall]
2. dba
? 1
Please enter the ORACLE_SID that corresponds to this installation? [inst_a1] inst_a (Needs to be the RAC TNS entry)
Please enter the ORACLE_HOME directory that corresponds to this ORACLE_SID? /u01/app/oracle/product/11.2.0/dbhome_1
Please enter the TCP/IP port number for SharePlex communications? [2100] <enter>
Proceed with installation? [yes] <enter>
Do you have a valid SharePlex for Oracle v. 8.0.1 license? [yes] <enter>
Please enter the License key? SORRYCANTSHARETHISWITHYOUPEOPLE
Please enter the customer name associated with this license key? XACTWARE INC

Setup - Continued
Configure SharePlex on Silo A ONLY (one node):
cd /u02/app/quest/sp/bin
./ora_setup
Enter the Oracle SID for which SharePlex should be installed [inst_a1] : inst_a_sp (this is the new sp TNS alias)
Enter a DBA user name : <sysdba user>
Enter password for the DBA account, which will not echo : <password>@inst_a_sp
Would you like to create a new SharePlex user ? [y] : y
Enter username for new user [SPLEX/SPLEX] : splex
Enter password for new user : splexpwd@inst_a_sp
Re-enter password for new user : splexpwd@inst_a_sp
Do you want to enable replication of tables with TDE? [n] : <enter>
Enter the default tablespace for use by SharePlex [USERS] : SPLEXDAT (The data tablespace created earlier)
Enter the temporary tablespace for use by SharePlex [TEMP] : <enter>
Enter the index tablespace for use by SharePlex [ ] : SPLEXIDX (The index tablespace created earlier)
Oracle ASM detected. Enable SharePlex ASM support? [y] : <enter>

Setup - Continued
For RAC configurations add License for other nodes
Run the following on the Silo A host to be added:
/u02/app/quest/sp/util/splex_uname|grep “Host ID”
Host ID = 2230858275
On the host where the SharePlex VIP is running:
/u02/app/quest/sp/install/splex_add_key
3) Add license key for alternate host
Enter option: 3
Adding license for machine ID : 2230858275 (Host ID from splex_uname above)
2) Enter license key manually
Enter option: 2
Enter Key: SORRYCANTSHARETHISWITHYOUPEOPLE
Enter Customer Name: XACTWARE INC
The SharePlex license has been successfully added for host 2230858275.
q) Quit License Utility
Enter option: q

Setup – Continued
Verify the paramdb file on Silo A:
cd /u02/app/quest/vardir
view paramdb
SP_COP_TPORT 2100
SP_COP_UPORT 2100
SP_OCT_TDE_SHARED_SECRET "" # SPO TDE Shared Secret
SP_OCT_ASM_SUPPORT "1" # ASM support on or off
SP_ORD_LOGIN_O.inst_a_SP "77bff9a39a0c2bb430c6c013e35ba8617cc4521279ff"
SP_ORD_OWNER_O.inst_a_SP "SPLEX"
SP_SYS_LIC_2230925271 "SORRYCANTSHARETHISWITHYOUPEOPLE:XACTWARE INC"
SP_SHS_SHMSIZE "33554432" # Statistics shared-memory Size
SP_OCT_USE_DST "0" # If 1, factor in DST when computing REDOLOG_ENSURE
SP_OCT_REPLICATE_SEQUENCES 0
SP_OCT_REPLICATE_DDL 0
SP_OCT_AUTOADD_ENABLE 0
SP_DEQ_THREADS "1" # Number of compare threads to run
Make sure the following entry does not exist:
SP_OCT_ASM_SID="+ASM1"

Setup – Continued
Verify the paramdb file on Silo B:
view paramdb
SP_COP_TPORT 2100
SP_COP_UPORT 2100
SP_ORD_LOGIN_O.inst_b_SP "77bff9a39a0c2bb430c6e89c013ba8617cc4511279ff"
SP_ORD_OWNER_O.inst_b_SP "SPLEX"

Setup – Continued
SharePlex Configuration Settings:
SP_OCT_USE_DST
Note: If 1, factor in DST when computing REDOLOG_ENSURE (default 1)
SP_OCT_REPLICATE_SEQUENCES
Note: If 1, replicate sequences (default 1)
SP_OCT_REPLICATE_DDL
Note: If 1, replicate alter/truncate statements (default 1)
SP_OCT_AUTOADD_ENABLE
Note: If 1, auto add new tables that match wildcard configuration (default 1)
SP_DEQ_THREADS
Note: This defines the number of compare threads to run (dynamic)
See:
http://documents.software.dell.com/DOC136923

Loosen Up
A young executive was leaving the office late one evening when he
found the CEO standing in front of a shredder with a piece of paper in
his hand.
"Listen," said the CEO, "this is a very sensitive and important
document here, and my secretary has gone for the night. Can you
make this thing work?"
"Certainly," said the young executive. He turned the machine on,
inserted the paper, and pressed the start button.
"Excellent, excellent!" said the CEO as his paper disappeared inside the
machine.
"I just need one copy."

Configuration
Create the Silo A SharePlex configuration file:
cd /u02/app/quest/vardir/config
vi schema_name_all
datasource:o.inst_a_SP
#source tables target tables routing map
expand schema_name.% schema_name.% inst_b-splex-vip@o.inst_b_SP
Note: This will replicate the schema_name schema from silo a to silo b.

Configuration - Continued
Create the Silo B SharePlex configuration file:
cd /u02/app/quest/vardir/config
vi schema_name_all
datasource:o.inst_a_SP
#source tables target tables routing map
expand schema_name.% schema_name.% inst_b-splex-vip@o.inst_b_SP
Note: This will replicate the schema_name schema from silo b to silo a.

Start SharePlex:
On silo A and silo B (on the host with SP VIP):
sp_cop &
Stop Target Post:
On silo B ONLY (on the host with SP VIP):
sp_ctrl
stop post
Activate Config:
On the Silo A (on the host with the SP VIP):
sp_ctrl
activate config schema_name_all
Note: the “activate config” begins tracking changes to the schema schema_name.

Verify Process State:
On both silo A and silo B
sp_ctrl
status
Brief Status for inst_a-splex-vip
Process State PID Running Since
--------------- ------------------------------ -------- --------------------
Cop Running 4207 01-Nov-13 17:10:24
Cmd & Ctrl Running 4272 01-Nov-13 17:12:55
Capture Running 5236 01-Nov-13 17:15:41
Read Running 5310 01-Nov-13 17:15:46
Post Stopped by user 5336 01-Nov-13 17:15:51
Note: On silo B the status of post should be “Stopped by user”

The next step is to duplicate the Silo A database to the Silo B
database. It does not matter which method you use to get the
source copied to the target (impdp, rman, etc.) as long as you
have a consistent SCN to pass to SharePlex.
When using rman make sure to record the following from the
duplication log file:
scn 322033901803

He He…
Q: What do you call something who turns into Father
Christmas whenever there is a full moon?
A: A where-clause
Two MySQL DBAs walk to a NoSQL bar, but they had to
leave because they couldn't find any tables!

Setup
Configure SharePlex on Silo B ONLY (Do NOT create a new SP user):
./ora_setup
Enter the Oracle SID for which SharePlex should be installed [inst_a1] : inst_a_sp (this is the new sp TNS alias)
Enter a DBA user name : <sysdba user>
Enter password for the DBA account, which will not echo : <password>@inst_a_sp
Would you like to create a new SharePlex user ? [y] : n
Enter username of an existing user : splex
Enter password for splex : splexpwd@inst_a_sp
Do you want to enable replication of tables with TDE? [n] : <enter>
Enter the default tablespace for use by SharePlex [USERS] : SPLEXDAT (The data tablespace created earlier)
Enter the temporary tablespace for use by SharePlex [TEMP] : <enter>
Enter the index tablespace for use by SharePlex [ ] : SPLEXIDX (The index tablespace created earlier)
Oracle ASM detected. Enable SharePlex ASM support? [y] : <enter>

Setup - Continued
For RAC configurations add License for other nodes
Run the following on the Silo B host to be added:
/u02/app/quest/sp/util/splex_uname|grep “Host ID”
Host ID = 2231027673
On the host where the SharePlex VIP is running:
/u02/app/quest/sp/install/splex_add_key
3) Add license key for alternate host
Enter option: 3
Adding license for machine ID : 2231027673 (Host ID from splex_uname above)
2) Enter license key manually
Enter option: 2
Enter Key: SORRYCANTSHARETHISWITHYOUPEOPLE
Enter Customer Name: XACTWARE INC
The SharePlex license has been successfully added for host 2231027673.
q) Quit License Utility
Enter option: q

Setup – Continued
Verify the paramdb file on Silo B:
view paramdb
SP_COP_TPORT 2100
SP_COP_UPORT 2100
SP_ORD_LOGIN_O.inst_b_SP "77bff9a39a0c2bbeab430c6e89c013e35ba8617cc4511279ff"
SP_ORD_OWNER_O.inst_b_SP "SPLEX"

Setup - Continued
Lookup Datasource Name (Silo B):
Queues Statistics for inst_b-splex-vip
Name: inst_a-splex-vip (o.INST_A_SP-o.INST_B_SP) (Post queue)
Number of messages: 1029659 (Age 24 min; Size 485 mb)
Backlog (messages): 0 (Age 0 min))
Reconcile Queues:
On Silo B - issue the reconcile command.
sp_ctrl
reconcile queue queuename for datasource-datadest scn scn_number
Example:
reconcile queue inst_a-splex-vip for o.INST_A_SP-o.INST_B_SP scn 322033901804
If this Process hangs it is likely related to no activity on the source. Log into the source database and create at least one
transaction, then force a log switch.
NOTE: This will not apply transactions yet...

Setup - Continued
Cleanup Target Tables:
Run the cleanup.sql script to truncate all of the SharePlex internal tables.
Only run this on Silo B:
sqlplus splex/<pass>@inst_b_sp
@cleanup.sql

Setup - Continued
Modify Triggers:
On Silo B (or both silos if you are bi-directional), disable triggers on the replicated tables, or run
the sp_add_trigger.sql utility script so that the triggers ignore the SharePlex user. This will
prevent circular replication.
cd /u02/app/quest/sp/util
sqlplus “/as sysdba”
@sp_add_trigger.sql
Enter SharePlex username --> splex
Verify all the triggers in the replicated schema have the following:
WHEN (
USER != 'SPLEX'
)

Setup - Continued
Before moving on with the next steps, make sure that there are
no replicated tables with constraints that are “on delete
cascade”. This will cause issues with replication and must be
disabled.
ALTER TABLE table_name ADD
CONSTRAINT constraint_name
FOREIGN KEY (column)
REFERENCES table_name (column_name)
ON DELETE CASCADE
ENABLE
VALIDATE
NOTE: If you are planning on bi-directional replication, neither silo can have this set.

Setup - Continued
Sequences:
If you are planning on active-active in your configuration then you have 2 options
for sequences.
(1) - Alter one cluster to odd and the other to even and increment by 2.
Example: select sequence.nextval from dual; (until odd or even), then
alter sequence sequence_name increment by 2;
(2) - Use sequence ranges for each silo in the cluster.
The recommended method is to use odd/even to avoid conflicts.
Determine the best method for your setup and make the sequence changes
before moving on with the next step.

Setup - Continued
Start SharePlex Post Process:
On silo B, start the Post process.
sp_ctrl
start post
NOTE: Silo B will now post the changes from the beginning of the duplicate
database until current. May take some time to catch up.
Monitor Queue Status:
On Silo A and Silo B:
sp_ctrl
qstatus

Final Laugh
NULL is the Chuck Norris of the database,
Nothing can be compared to it…

CRS Configuration
Create Start / Stop Script:
logfile="/u02/app/quest/scripts/log/spctl.log" ; export logfile
sp_env="/export/home/oracle/.bashrc.inst_b" ; export sp_env
sp_start_cmd='/u02/app/quest/sp/bin/sp_cop -uinst_b &'
spctrl='/u02/app/quest/sp/bin/sp_ctrl'
case "$1" in
'start')
echo "Starting Shareplex" | tee -a ${logfile}
su - oracle <<EOF
source ${sp_env}
set|grep "SP_" >> ${logfile}
${sp_start_cmd} >> ${logfile}
sleep 30
exit
EOF
echo | tee -a ${logfile}
;;
'stop')
echo "Stopping Shareplex" | tee -a ${logfile}
su - oracle <<EOF
source ${sp_env}
set|grep "SP_" >> ${logfile}

CRS Configuration
echo "shutdown" | ${spctrl} | tee -a ${logfile}
exit
EOF
;;
‘check')
echo "Checking Shareplex" | tee -a ${logfile}
sp_check=$(ps -ef|grep "sp_cop"|grep -v grep|wc -l)
if [ ${sp_check} -gt 0 ] ; then
echo "Shareplex is running" | tee -a ${logfile}
exit 0
else
echo "Shareplex is not running " | tee -a ${logfile}
exit 1
fi
;;
'clean')
echo "Cleaning Shareplex" | tee -a ${logfile}
;;
esac

CRS Configuration
Create the CRS Resource:
On Silo A and Silo B (as root):
${CRS_HOME}/bin/crsctl add resource shareplex -type cluster_resource -attr
"AGENT_FILENAME="%"CRS_HOME"%"/bin/scriptagent,
ACTION_SCRIPT=/u02/app/quest/scripts/spctl.sh, DESCRIPTION=, DEGREE=1,
ENABLED=1, AUTO_START=always, START_TIMEOUT=0, UPTIME_THRESHOLD=1h,
CHECK_INTERVAL=10, STOP_TIMEOUT=0, SCRIPT_TIMEOUT=60,
RESTART_ATTEMPTS=2, OFFLINE_CHECK_INTERVAL=0,
START_DEPENDENCIES=hard(shareplex.vip) pullup(shareplex.vip),
STOP_DEPENDENCIES=hard(intermediate: shareplex.vip),CARDINALITY=1,
FAILURE_INTERVAL=0, FAILURE_THRESHOLD=0, SERVER_POOLS=*,
PLACEMENT=balanced, LOAD=1, ACTIVE_PLACEMENT=1”
Set CRS Resource Permissions:
On Silo A and Silo B (as root):
${CRS_HOME}/bin/crs_setperm shareplex -u user:oracle:r-x

Administration
Useful CRS Commands:
Start:
${CRS_HOME}/bin/crsctl start resource shareplex
VIP will start automatically
# ${CRS_HOME}/bin/crsctl start resource shareplex.vip
Stop:
${CRS_HOME}/bin/crsctl stop resource shareplex
${CRS_HOME}/bin/crsctl stop resource shareplex.vip
Relocate:
${CRS_HOME}/bin/crsctl relocate resource shareplex -s <existing node> -n <new node> -f

Administration - Continued
sp_ctrl commands:
status – checks the status of shareplex processes
qstatus – shows the current status of the shareplex queues
shutdown – shutdown the shareplex sp_cop process
show log reverse – shows the event_log in reverse order
show post detail – show a detail view of the post process
show capture detail – show a detail view of the capture process
show sync – shows if there are known out of sync tables
compare – compares a table or wildcard list of tables
repair – repairs a table or wildcard list of tables
compare status – shows the current status of a compare process
repair status – shows the current status of a repair process

Compare/Repair Table:
sp_ctrl > compare sp_demo.test
ID Tablename Status Time Total Rows %Comp Total Time
------ ------------------------------------ ---------- ---------- ---------- ----- ----------
1 ”SP_DEMO"."TEST" Out Sync N/A 2 100 0:07
sp_ctrl > repair sp_demo.test
ID Tablename Status Time Total Rows %Comp Total Time
------ ------------------------------------ ---------- ---------- ---------- ----- ----------
1 ”SP_DEMO"."TEST" Repaired N/A 0 100 0:07
NOTE: After running compare and finding “out of sync” it will not show in “show sync”

Adding an object to existing configuration:
(1) Create the object on the source (and target if not using DDL
replication).
(2) sp_ctrl > activate config schema_name_all
NOTE: Do not deactivate configuration as you will lose
transactions and be forced to re-synchronize the source and
target environments.

Final Thoughts
We are pleased with the overall value of the
SharePlex product. The cost is reasonable for
the functionality it provides, and we are satisfied
with the stability and performance as well. If
you combine that with the ease of setup and
administration it becomes clear that SharePlex is
a good fit for Xactware.

Shareplex Presentation

More Related Content

What's hot (20)

Similar to Shareplex Presentation (20)

Shareplex Presentation