SlideShare a Scribd company logo
High Availability Cloud
Control Recipes - EM12C
    Session ID# 941

         Mike Gangler

     Ithaka - www.ithaka.org

                               Slide 1
Mike Gangler
• Ann Arbor, Michigan
• Database Specialist at Ithaka.org – aka “JSTOR”
• Multi-platform and Database DBA
• Specializing in Performance, Tuning and management of databases and
  environments
• Working With Oracle since Versus 4 ….
• Board of Directors for SEMOP (Southeast Michigan Oracle Professionals),
  MOUS (Michigan Oracle User Summit, and charter member (1993) of the
  Board of Directors of the International Oracle Users Group(1OUG).



                                                               Slide 2
What this presentation will cover
• EM12 Architecture High Availability options

• Setting up an Level 3 MAA Environment

• Installation of a Standby OMS – Silent Mode

• If Time---Notifications Best Practices


                                                Slide 3
Audience Experience
• Grid Control Experience
  • Used the old 9i / Workgroup version ?

  • 10G Grid Control

  • 11G Grid Control

  • 12C Cloud Control

                                            Slide 4
Other Presentations
• Enterprise Manager SIG – 9:30 – 10:30am
  • Wednesday – MileHigh 4B ?




                                            Slide 5
Presentation Goals
• Goals

 • Learn the Different MAA architectures

 • How to run a installation of a Standby OMS in
   silent Mode

 • Notifications Tips / Best Practices

                                           Slide 6
Ithaka’s Mission
                  aka JSTOR
ITHAKA is a not-for-profit organization dedicated to
helping the academic community take full advantage
of rapidly advancing information and networking
technologies. We serve scholars, researchers, and
students by providing the content, tools, and services
needed to preserve the scholarly record and to
advance research and teaching in sustainable ways.
We are committed to working in collaboration with
other organizations to maximize benefits to our
stakeholders.

www.ithaka.org - www.jstor.org
                                               Slide 7
EM12C Architecture
                 Options
• Level 1 – Single Site – No Failover

• Level 2 – Single Site
   • 2 OMS – Active / Passive Mode
   • VIP Failover


                                        Slide 8
EM12C Architecture
              Options
• Level 3
   • Single Site
   • Multiple OMS (Active/Active)
   • Load Balancer
   • RAC Data Guard Primary / Standby DB


                                       Slide 9
EM12C Architecture
              Options
• Level 4
    • Multiple Sites
    • Primary OMS (Active/Active) w/ Load Balancer
    • Load Balancer
    • RAC Data Guard Primary / Standby DB
    • Multiple Standby OMS w/ Load Balancer
• .
                                         Slide 10
EM12C Architecture
               Suggestions
• Setup an additional / separate OMS for Dev/QA targets

• Setup Level 3.5 Maximum Availability Architecture (MAA)
   • Multiple Sites
   • Multiple OMS (Primary / Standby)
   • Oracle Data Guard with Fast Start Failover setup.
   • Using VIP’s / DNS alias help with data Guard rollover.
   • Use Load Balancer (optional) for OMS.


                                                   Slide 11
EM12C Standby OMS
Architecture (level 3 MAA)




                         Slide 12
Monitoring at JSTOR
•   Production Oracle Databases – 25 – 24x7
•   Development Oracle Databases – 27 – 14x5
•   Microsoft SqlServer Databases - 21 – 14x5
•   MySql Databases – 9 – 14x5

• Total 240 Production Targets (Agents, hosts, db, etc.)
• Total 276 Dev/QA Targets (Agents, hosts, db, etc.)



                                                Slide 13
Ithaka OMS Layout
 DATACENTER          DATA                DATACENTER
     1              CENTER 2                 3



                        DNS

     DataCenter 1        DataCenter 2
      Secondary          Primary OMS
         OMS




                    DNS
  Datacenter1
   Standby                     Primary
  DataGuard                       DB
      DB

                                                  Slide 14
Pre-requisites for High Availability
 • Setup VIP or DNS Alias for multiple
     sites.
 •   Create Data Guard database on other
     site.
 •   NFS Shared Library
 •   Primary OMS Setup
 •   Plugins Installed on Primary Site


                                         Slide 15
Pre-requisites for High Availability
  • Optional setups:
   • Fast Start Failover Data guard
      Setup(Requires additional hardware)

   • Load Balancer NOT REQUIRED –
      Although the documentation disagrees

   • RAC DB’s– Great if you can afford it.
                                      Slide 16
Requirements Standby OMS
  • Machine Required for Standby OMS/DB
    (Linux, Sparc, HPUX, Microsoft)

  • Plugins need to be the same version
  • Exact same directory structure
  • Load Balancer (optional)
  • NFS Drive ….                      Slide 17
Configuration Steps:
 1.Install Primary OMS / DB
 2.Add targets to Primary
 3.Setup NFS Drive – Software Library
 4.Add All Plugins required to Primary OMS
 5.Add Dataguard - Standby Database
 6.Create Standby OMS
 7.Convert Standby Database to RAC (IF Available)
                                                    Slide 18
Installation Steps: (Silent)


   Verify NFS Software Library on Standby Machine
   Install a “Software-Only” Installation on Standby OMS
     Machine


     (Do NOT run the ConfigureGC.sh or ConfigureGC.bat)


                                                            Slide 19
Installation Steps: (Silent)
  •   Ensure same Plugins as the source OMS
       SOURCE_OMS_pgrid01>l
        1 SELECT epv.plugin_id, epv.version, epv.rev_version
        2 FROM em_plugin_version epv, em_current_deployed_plugin ecp
        3 WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE','INSTALL_HOME')
        4 AND ecp.dest_type='2'
        5* AND epv.plugin_version_id = ecp.plugin_version_id
       SYSMAN_pgrid01>/

       PLUGIN_ID              VERSION         REV_VERSION
       --------------------    ------------    -----------
       oracle.sysman.mos      12.1.0.2.0            0
       oracle.em.satc         12.1.0.1.0            0
       oracle.em.smss         12.1.0.2.0            0
       oracle.sysman.emct     12.1.0.3.0            0
       oracle.sysman.ssa      12.1.0.4.0            0
       oracle.sysman.db       12.1.0.2.0        20120804
       oracle.sysman.xa       12.1.0.3.0            0
       oracle.sysman.emas     12.1.0.3.0            0
       oracle.sysman.vt       12.1.0.3.0            0
       pythian.mysql.prod     12.1.0.1.2            0



                                                                         Slide 20
Installation Steps: (Silent)
  Plug-in Definitions:
      PLUGIN_ID              DEFINITION
      --------------------   -----------------------
      oracle.sysman.mos      My Oracle Support
      oracle.em.satc         Apache Tomcat
      oracle.em.smss         Microsoft SQL Server
      oracle.sysman.emct     Charge Back
      oracle.sysman.ssa      Cloud Application
      oracle.sysman.db       Oracle Database
      oracle.sysman.xa       Exadata
      oracle.sysman.emas     Fusion Middleware
      oracle.sysman.vt       Virtualisation
      pythian.mysql.prod     MySQL
                                                  Slide 21
Installation Steps: (Silent)
   Plug-ins (Continued)

     1. To Install Additional Plug-ins – Manually download the
        plug-ins from OTN to Accessible location (i.e., /tmp)

       a. Verify that they are the same version as primary OMS
     2. http://www.oracle.com/technetwork/oem/grid-
        control/downloads/index.html

     3. Run $OMS_HOME/sysman/install/PluginInstall.sh –
        pluginLocation /tmp - Choose the plugins you want to
        install (screen print – next page)
                                                         Slide 22
PluginInstall.sh




                   Slide 23
Installation Steps: (Silent)
  1. Apply all patches that were applied to the first OMS
    a. To Check - $OMS_HOME/OPatch/opatch lsinventory
  2. Export the Configuration Details from the First OMS
    a. $OMS_HOME/bin/emctl exportconfig oms –dir <Path
        Name>

    b. Creates bka file – i.e. opf_ADMIN_<DATE>_<TIME>.bka
    c. Copy bka file to standby OMS Machine
                                                            Slide 24
Installation Steps: (Silent)
  oracle@pr2prodora01 ~]$ cd /u01/app/oracle/OracleHomes2/oms/bin/
  [oracle@pr2prodora01 bin]$ ./emctl exportconfig oms -dir $HOME
  Oracle Enterprise Manager Cloud Control 12c Release 2
  Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
  Enter Enterprise Manager Root (SYSMAN) Password :
  ExportConfig started...
  Machine is Admin Server host. Performing Admin Server backup...
  Exporting emoms properties...
  Exporting secure properties...

  Export has determined that the OMS is not fronted
  by an SLB. The local hostname was NOT exported.
  The exported data can be imported on any host but
  resecure of all agents will be required. Please
  see the EM Advanced Configuration Guide for more
  details.

  Exporting configuration for pluggable modules...
  Preparing archive file...
  Backup has been written to file: /home/oracle/opf_ADMIN_20130125_101524.bka

  The export file contains sensitive data.
  Please ensure that it is kept secure.

  ExportConfig completed successfully!
  [oracle@pr2prodora01 bin]$ cd $HOME
  [oracle@pr2prodora01 ~]$ scp opf_ADMIN_20130125_101524.bka oracle@aa2prodora11.jstor.org:/home/oracle
  oracle@aa2prodora11.jstor.org's password:
  opf_ADMIN_20130125_101524.bka 100% 45MB 5.0MB/s 00:09
  [oracle@pr2prodora01 ~]$

                                                                                                          Slide 25
Installation Steps: (Silent)
  OMSCA Breakdown on Standby database using the
  backup file created

    •   Uses the backup file just generated = bka file
    •   Admin Server HTTPS Port = 7101
    •   Managed Server Port=7202
    •   Managed Server HTTPS Port=7301
    •   EM Note Manager Port = 7403
    •   EM Upload Port = 4889
    •   EM Upload HTTPS Port=4900
    •   EM Console Port = 7788
    •   EM Console HTTPS Port= 7799
    •   Config_home = /u01/oracle/product/OracleHomes2/gc_inst
    •   EM_INSTANCE_HOST aa2prodora11.jstor.org (Standby machine)

                                                           Slide 26
Installation Steps: (Silent)

  Of all these ports the following are relevant to the load
  Balancer:



  •EM Upload HTTP Port: 4889
  •EM UPLOAD HTTPS Port: 4900
  •EM Central Console HTTP Port: 7788
  •EM Central Console HTTPS Port: 7799

                                                       Slide 27
Installation Steps: (Silent)
  •   Run OMSCA on Standby database using the backup
      file created

  <OMS_HOME>/bin/omsca recover -ms -backup_file
     /opt/oracle/product/backup/opf_ADMIN_20120504_031016.bka
     -AS_HTTPS_PORT 7101 -MSPORT 7202 -MS_HTTPS_PORT 7301
     -EM_NODEMGR_PORT 7403 -EM_ UPLOAD_PORT 4889
     -EM_UPLOAD_HTTPS_PORT 4900 -EM_CONSOLE_PORT 7788 -EM_
     CONSOLE_HTTPS_PORT 7799 -config_home
     /opt/oracle/product/omsmdw/gc_inst -EM_INSTANCE_HOST
     example.com


  !!! Many errors at this stage, due Failed attempts

                                                       Slide 28
Installation Issues:
  MSEVERE: OMSCA-ERR:Post "Deploy and Repos
  Setup" operations failed

  Check the following items :
  -- Check RAM on Standby Machine – needs 6GB – to Install

  -- If using 32 Bit Linux –
        Modify – create_domain.py (under omsca/scripts/wls
  directory)
        Change - -Xmx1524m to –Xmx1324m
        Change –XX:MaxPermsize=612M to –X:MaxPermsize=412M

  -- Check for Lock in repository

                                                       Slide 29
Installation Issues
  •   Check for lock in the repository

  •   Due to multiple tries there was a lock in the
      repository during the plugin deployment stage.


       •   select host_url, oms_id, status, is_first_oms,
           em_release_version from gc_oms_info;

       •   select mp_guid,file_id,file_name,file_type from
           mgmt_mp_files;

       •   select * from em_plugin_lock_track;


                                                       Slide 30
Installation Issues:

  •   First Issue – wrong version of the database plugin



      •   I upgraded the database plugin but didn’t include it
          in the standby installation

      •   Copied latest “DB” plugin and reran
          pluginInstall.sh




                                                       Slide 31
Installation Issues:
  •   2nd Issue – Plugin Issue and multiple attempts caused
      lock in EM database (SYSMAN)



      •   Removed record from lock table
          em_plugin_lock_track

      •   Ran $OMS_HOME/bin - ./pluginca –action deploy
          –isFirstOMS false –plugins

      •   Started OMS


                                                    Slide 32
Final Installation Steps

  If you haven’t installed the agent on the standby
      host:

  $<AGENT_HOME>/sysman/install/agentDeploy.sh
     AGENT_BASE_DIR=<middleware_ home>/agent
     OMS_HOST=<second_oms_host_name> EM_UPLOAD_PORT=<second_oms_
     port> AGENT_REGISTRATION_PASSWORD=<password> -configOnly


  Note: If you have Server Load Balancer (SLB) configured, then directly
     enter the host name and port number of the SLB. If SLB is not
     configured, then enter the secure upload port of the first OMS.




                                                                   Slide 33
Final Installation Steps
   After agent is installed and deployed
     •       Deploy the required plug-ins on the management agent


   Secure the agent - $AGENT_HOME/bin/emctl secure
      add_trust_cert_to_jks (If applicable)


   In the new OMS Grid control (standby) – discover the Oracle Weblogic
      services


     •   Targets, Middleware, click EMGC_DOMAIN
     •   From EMGC_DOMAIN home page
     •   Select Farm Menu, Refresh Weblogic Domain, click continue till 2nd
         management server is discovered on the standby OMS Host.


                                                                   Slide 34
Installation logs / help
  •    Plugin Logs:
          •    $OMS_HOME/cfgtoollogs/pluginca/*
          •    $OMS_HOME/cfgw/CfmLogger_<timestamp>.log


      •    Repository Configuration logs
          •    $OMS_HOME/cfgw/CfmLogger_<timestamp>.log
          •    $OMS_HOME/sysman/log/schemamanager/m_<timestamp>/*



      •    MDS Schema Configuration Assistant
          •    $OMS_HOME/sysman/log/schemamanager/m_<timestamp>/*
          •    $OMS_HOME/cfgtoollogs/cfgfw/emdcreate*.log


      •    More Log file and Troubleshooting tips:
  http://docs.oracle.com/cd/E24628_01/install.121/e24080/appdx_troubleshooting.htm#CEGFAHIG
                                                                                    Slide 35
What We Covered

 •   High Available architecture options

 •   Silent Standby OMS installations and
     gotchas




                                            Slide 50
References
Deploying a Highly Available EM12C Cloud Control – An Oracle
White Paper – May 2012 - # 1631423.pdf

www.dbakevlar.com – Kellyn Pot’Vin’s blog contains many great
ideas about EM12c

e24089.pdf – Cloud Control Advanced Installation and
Configuration Guide (section 5 - installing Add'l OMS in silent
mode)
References
https://forums.oracle.com/forums/category.jspa?categoryID=70
– OTN Discussion on EM

www.oracle.com – Documentation is not bad for many of the
basic items with EM12c

Keeping your OMS Tidy – Ray Smith - IOUG Tips and Best
Practices – 7th edition - www.ioug.org

http://oramanageability.wordpress.com - Ray Smiths Blog on
Cloud Control
References

www.pythian.com – MySQL plug-in and instructions

oemgc.wordpress.com – Rob Zoeteweij – Good information on
Incident management and ITL, including a blog

www.slideshare.net/Enkitec/em12c-monitoring-metric-
extensions-and-performance-pages#btnPrevious – Em12C
Monitoring, Metric Extensions and Performance pages – By
Kellyn Pot’Vin – Enkitec

EM12C Log Cleanup – Script to clean up logs and information
Questions?
Comments?
Thank You for Attending

 Please fill out your Evaluations
       www.ioug.org/eval

Cloud Control Recipes for EM12C
        Session # 941
         Mike Gangler

     •Email: michael.gangler@ithaka.org
             • Twitter: @mjgangler
       • Blog: mjgangler.wordpress.com
                                          Slide 55

More Related Content

PPTX
Managing Oracle Enterprise Manager Cloud Control 12c with Oracle Clusterware
PPT
Configuring Oracle Enterprise Manager Cloud Control 12c for High Availability
PDF
High Availability Options for Oracle Enterprise Manager 12c Cloud Control
PDF
Migrating from Oracle Enterprise Manager 10g to 12c Cloud Control
PPTX
Where did my day go?: Oracle Enterprise Manager 12c/13c Administration
PPTX
Where did my day go?: OEM 12c Administration #em12c #C15LV @IOUG
PDF
Oracle Enterprise Manager Cloud Control 13c for DBAs
PPTX
Oracle grid control setup and usage challenges version5
Managing Oracle Enterprise Manager Cloud Control 12c with Oracle Clusterware
Configuring Oracle Enterprise Manager Cloud Control 12c for High Availability
High Availability Options for Oracle Enterprise Manager 12c Cloud Control
Migrating from Oracle Enterprise Manager 10g to 12c Cloud Control
Where did my day go?: Oracle Enterprise Manager 12c/13c Administration
Where did my day go?: OEM 12c Administration #em12c #C15LV @IOUG
Oracle Enterprise Manager Cloud Control 13c for DBAs
Oracle grid control setup and usage challenges version5

What's hot (20)

PDF
Exadata SMART Monitoring - OEM 13c
PDF
Oracle Enterprise Manager Cloud Control 13c for DBAs
PDF
Enterprise manager 13c
PPTX
Oracle Enterprise Manager
DOC
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
PPTX
Database as a Service, Collaborate 2016
PPT
Less05 asm instance
PPTX
vmware_site_recovery_manager_and_net_app_fas_v-series_se_technical_presentati...
PDF
Reducing Your E-Business Suite Storage Footprint Using Oracle Advanced Compre...
PDF
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
PPTX
Upgrading Em13c Collaborate 2016
PDF
Essential Linux Commands for DBAs
PDF
Oracle Enterprise Manager 11g
PDF
EM13c: Write Powerful Scripts with EMCLI
PDF
Advanced Shell Scripting for Oracle professionals
PPT
Less18 support
PPT
Less02 installation
PDF
10 Problems with your RMAN backup script - whitepaper
PPTX
10 ways to improve your rman script
PDF
Double the Performance of Oracle SOA Suite 11g? Absolutely!
Exadata SMART Monitoring - OEM 13c
Oracle Enterprise Manager Cloud Control 13c for DBAs
Enterprise manager 13c
Oracle Enterprise Manager
Configuring Oracle Enterprise Manager Cloud Control 12c for HA White Paper
Database as a Service, Collaborate 2016
Less05 asm instance
vmware_site_recovery_manager_and_net_app_fas_v-series_se_technical_presentati...
Reducing Your E-Business Suite Storage Footprint Using Oracle Advanced Compre...
Oracle Flex ASM - What’s New and Best Practices by Jim Williams
Upgrading Em13c Collaborate 2016
Essential Linux Commands for DBAs
Oracle Enterprise Manager 11g
EM13c: Write Powerful Scripts with EMCLI
Advanced Shell Scripting for Oracle professionals
Less18 support
Less02 installation
10 Problems with your RMAN backup script - whitepaper
10 ways to improve your rman script
Double the Performance of Oracle SOA Suite 11g? Absolutely!
Ad

Similar to EM12C High Availability without SLB and RAC (20)

PDF
UKOUG Tech15 - Deploying Oracle 12c Cloud Control in Maximum Availability Arc...
PDF
New Not Your Father's Enterprise Manager
PDF
Zero to Manageability in 60 Minutes: Building a Solid Foundation for Oracle E...
PPTX
CON5451_Brydon-OOW2014_Brydon_CON5451 (1).pptx
PDF
UPGRADING FROM ORACLE ENTERPRISE MANAGER 10G TO CLOUD CONTROL 12C WITH ZERO D...
PPT
Grid Control
PDF
Oracle Enterprise Manager 12c - OEM12c Presentation
PPT
D17316 gc20 l03_broker_em
PDF
Apouc 2014-enterprise-manager-12c
PDF
Oracle RAC 12c Best Practices Sanger OOW13 [CON8805]
PDF
real-application-clusters-installation-guide-linux-and-unix.pdf
DOC
Migrating from Single Instance to RAC Data guard
PPT
01_Architecture_JFV14_01_Architecture_JFV14.ppt
PDF
Em13c features- HotSos 2016
DOC
Configuring oracle enterprise manager cloud control 12 c for high availability
PDF
Monitor Engineered Systems from a Single Pane of Glass: Oracle Enterprise Man...
PDF
Database Provisioning in EM12c: Provision me a Database Now!
PDF
Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version
PDF
Oracle RAC 12c Best Practices with Appendices DOAG2013
PDF
Maa wp sun_apps11i_db10g_r2-2
UKOUG Tech15 - Deploying Oracle 12c Cloud Control in Maximum Availability Arc...
New Not Your Father's Enterprise Manager
Zero to Manageability in 60 Minutes: Building a Solid Foundation for Oracle E...
CON5451_Brydon-OOW2014_Brydon_CON5451 (1).pptx
UPGRADING FROM ORACLE ENTERPRISE MANAGER 10G TO CLOUD CONTROL 12C WITH ZERO D...
Grid Control
Oracle Enterprise Manager 12c - OEM12c Presentation
D17316 gc20 l03_broker_em
Apouc 2014-enterprise-manager-12c
Oracle RAC 12c Best Practices Sanger OOW13 [CON8805]
real-application-clusters-installation-guide-linux-and-unix.pdf
Migrating from Single Instance to RAC Data guard
01_Architecture_JFV14_01_Architecture_JFV14.ppt
Em13c features- HotSos 2016
Configuring oracle enterprise manager cloud control 12 c for high availability
Monitor Engineered Systems from a Single Pane of Glass: Oracle Enterprise Man...
Database Provisioning in EM12c: Provision me a Database Now!
Oracle RAC 12c Collaborate Best Practices - IOUG 2014 version
Oracle RAC 12c Best Practices with Appendices DOAG2013
Maa wp sun_apps11i_db10g_r2-2
Ad

More from Secure-24 (10)

PDF
2019 odtug webinar-20190129
PDF
Gloc gangler 2018._v4
PDF
2016_1201_gangler_ppt
PDF
Granting Oracle Schema Permissions when Objects not created Yet
PDF
Using RPM's to Automagically to install Oracle Agents to EM12C
PDF
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
PDF
2015 582 gangler_ppt - ms
PPTX
Using oracle12c pluggable databases to archive
PDF
2008 2086 Gangler
PPTX
Mgangler Virtualization
2019 odtug webinar-20190129
Gloc gangler 2018._v4
2016_1201_gangler_ppt
Granting Oracle Schema Permissions when Objects not created Yet
Using RPM's to Automagically to install Oracle Agents to EM12C
Optimize and Simplify Oracle 12C RAC using dNFS, ZFS and OISP
2015 582 gangler_ppt - ms
Using oracle12c pluggable databases to archive
2008 2086 Gangler
Mgangler Virtualization

Recently uploaded (20)

PDF
Unlocking AI with Model Context Protocol (MCP)
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Machine learning based COVID-19 study performance prediction
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PPT
Teaching material agriculture food technology
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
PPTX
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
PPTX
A Presentation on Artificial Intelligence
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
Modernizing your data center with Dell and AMD
PDF
Chapter 3 Spatial Domain Image Processing.pdf
PDF
Review of recent advances in non-invasive hemoglobin estimation
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Unlocking AI with Model Context Protocol (MCP)
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Machine learning based COVID-19 study performance prediction
Digital-Transformation-Roadmap-for-Companies.pptx
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Teaching material agriculture food technology
The Rise and Fall of 3GPP – Time for a Sabbatical?
PA Analog/Digital System: The Backbone of Modern Surveillance and Communication
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
CIFDAQ's Market Insight: SEC Turns Pro Crypto
20250228 LYD VKU AI Blended-Learning.pptx
“AI and Expert System Decision Support & Business Intelligence Systems”
A Presentation on Artificial Intelligence
Advanced methodologies resolving dimensionality complications for autism neur...
Modernizing your data center with Dell and AMD
Chapter 3 Spatial Domain Image Processing.pdf
Review of recent advances in non-invasive hemoglobin estimation
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Reach Out and Touch Someone: Haptics and Empathic Computing

EM12C High Availability without SLB and RAC

  • 1. High Availability Cloud Control Recipes - EM12C Session ID# 941 Mike Gangler Ithaka - www.ithaka.org Slide 1
  • 2. Mike Gangler • Ann Arbor, Michigan • Database Specialist at Ithaka.org – aka “JSTOR” • Multi-platform and Database DBA • Specializing in Performance, Tuning and management of databases and environments • Working With Oracle since Versus 4 …. • Board of Directors for SEMOP (Southeast Michigan Oracle Professionals), MOUS (Michigan Oracle User Summit, and charter member (1993) of the Board of Directors of the International Oracle Users Group(1OUG). Slide 2
  • 3. What this presentation will cover • EM12 Architecture High Availability options • Setting up an Level 3 MAA Environment • Installation of a Standby OMS – Silent Mode • If Time---Notifications Best Practices Slide 3
  • 4. Audience Experience • Grid Control Experience • Used the old 9i / Workgroup version ? • 10G Grid Control • 11G Grid Control • 12C Cloud Control Slide 4
  • 5. Other Presentations • Enterprise Manager SIG – 9:30 – 10:30am • Wednesday – MileHigh 4B ? Slide 5
  • 6. Presentation Goals • Goals • Learn the Different MAA architectures • How to run a installation of a Standby OMS in silent Mode • Notifications Tips / Best Practices Slide 6
  • 7. Ithaka’s Mission aka JSTOR ITHAKA is a not-for-profit organization dedicated to helping the academic community take full advantage of rapidly advancing information and networking technologies. We serve scholars, researchers, and students by providing the content, tools, and services needed to preserve the scholarly record and to advance research and teaching in sustainable ways. We are committed to working in collaboration with other organizations to maximize benefits to our stakeholders. www.ithaka.org - www.jstor.org Slide 7
  • 8. EM12C Architecture Options • Level 1 – Single Site – No Failover • Level 2 – Single Site • 2 OMS – Active / Passive Mode • VIP Failover Slide 8
  • 9. EM12C Architecture Options • Level 3 • Single Site • Multiple OMS (Active/Active) • Load Balancer • RAC Data Guard Primary / Standby DB Slide 9
  • 10. EM12C Architecture Options • Level 4 • Multiple Sites • Primary OMS (Active/Active) w/ Load Balancer • Load Balancer • RAC Data Guard Primary / Standby DB • Multiple Standby OMS w/ Load Balancer • . Slide 10
  • 11. EM12C Architecture Suggestions • Setup an additional / separate OMS for Dev/QA targets • Setup Level 3.5 Maximum Availability Architecture (MAA) • Multiple Sites • Multiple OMS (Primary / Standby) • Oracle Data Guard with Fast Start Failover setup. • Using VIP’s / DNS alias help with data Guard rollover. • Use Load Balancer (optional) for OMS. Slide 11
  • 12. EM12C Standby OMS Architecture (level 3 MAA) Slide 12
  • 13. Monitoring at JSTOR • Production Oracle Databases – 25 – 24x7 • Development Oracle Databases – 27 – 14x5 • Microsoft SqlServer Databases - 21 – 14x5 • MySql Databases – 9 – 14x5 • Total 240 Production Targets (Agents, hosts, db, etc.) • Total 276 Dev/QA Targets (Agents, hosts, db, etc.) Slide 13
  • 14. Ithaka OMS Layout DATACENTER DATA DATACENTER 1 CENTER 2 3 DNS DataCenter 1 DataCenter 2 Secondary Primary OMS OMS DNS Datacenter1 Standby Primary DataGuard DB DB Slide 14
  • 15. Pre-requisites for High Availability • Setup VIP or DNS Alias for multiple sites. • Create Data Guard database on other site. • NFS Shared Library • Primary OMS Setup • Plugins Installed on Primary Site Slide 15
  • 16. Pre-requisites for High Availability • Optional setups: • Fast Start Failover Data guard Setup(Requires additional hardware) • Load Balancer NOT REQUIRED – Although the documentation disagrees • RAC DB’s– Great if you can afford it. Slide 16
  • 17. Requirements Standby OMS • Machine Required for Standby OMS/DB (Linux, Sparc, HPUX, Microsoft) • Plugins need to be the same version • Exact same directory structure • Load Balancer (optional) • NFS Drive …. Slide 17
  • 18. Configuration Steps: 1.Install Primary OMS / DB 2.Add targets to Primary 3.Setup NFS Drive – Software Library 4.Add All Plugins required to Primary OMS 5.Add Dataguard - Standby Database 6.Create Standby OMS 7.Convert Standby Database to RAC (IF Available) Slide 18
  • 19. Installation Steps: (Silent)  Verify NFS Software Library on Standby Machine  Install a “Software-Only” Installation on Standby OMS Machine  (Do NOT run the ConfigureGC.sh or ConfigureGC.bat) Slide 19
  • 20. Installation Steps: (Silent) • Ensure same Plugins as the source OMS SOURCE_OMS_pgrid01>l 1 SELECT epv.plugin_id, epv.version, epv.rev_version 2 FROM em_plugin_version epv, em_current_deployed_plugin ecp 3 WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE','INSTALL_HOME') 4 AND ecp.dest_type='2' 5* AND epv.plugin_version_id = ecp.plugin_version_id SYSMAN_pgrid01>/ PLUGIN_ID VERSION REV_VERSION -------------------- ------------ ----------- oracle.sysman.mos 12.1.0.2.0 0 oracle.em.satc 12.1.0.1.0 0 oracle.em.smss 12.1.0.2.0 0 oracle.sysman.emct 12.1.0.3.0 0 oracle.sysman.ssa 12.1.0.4.0 0 oracle.sysman.db 12.1.0.2.0 20120804 oracle.sysman.xa 12.1.0.3.0 0 oracle.sysman.emas 12.1.0.3.0 0 oracle.sysman.vt 12.1.0.3.0 0 pythian.mysql.prod 12.1.0.1.2 0 Slide 20
  • 21. Installation Steps: (Silent) Plug-in Definitions: PLUGIN_ID DEFINITION -------------------- ----------------------- oracle.sysman.mos My Oracle Support oracle.em.satc Apache Tomcat oracle.em.smss Microsoft SQL Server oracle.sysman.emct Charge Back oracle.sysman.ssa Cloud Application oracle.sysman.db Oracle Database oracle.sysman.xa Exadata oracle.sysman.emas Fusion Middleware oracle.sysman.vt Virtualisation pythian.mysql.prod MySQL Slide 21
  • 22. Installation Steps: (Silent) Plug-ins (Continued) 1. To Install Additional Plug-ins – Manually download the plug-ins from OTN to Accessible location (i.e., /tmp) a. Verify that they are the same version as primary OMS 2. http://www.oracle.com/technetwork/oem/grid- control/downloads/index.html 3. Run $OMS_HOME/sysman/install/PluginInstall.sh – pluginLocation /tmp - Choose the plugins you want to install (screen print – next page) Slide 22
  • 23. PluginInstall.sh Slide 23
  • 24. Installation Steps: (Silent) 1. Apply all patches that were applied to the first OMS a. To Check - $OMS_HOME/OPatch/opatch lsinventory 2. Export the Configuration Details from the First OMS a. $OMS_HOME/bin/emctl exportconfig oms –dir <Path Name> b. Creates bka file – i.e. opf_ADMIN_<DATE>_<TIME>.bka c. Copy bka file to standby OMS Machine Slide 24
  • 25. Installation Steps: (Silent) oracle@pr2prodora01 ~]$ cd /u01/app/oracle/OracleHomes2/oms/bin/ [oracle@pr2prodora01 bin]$ ./emctl exportconfig oms -dir $HOME Oracle Enterprise Manager Cloud Control 12c Release 2 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : ExportConfig started... Machine is Admin Server host. Performing Admin Server backup... Exporting emoms properties... Exporting secure properties... Export has determined that the OMS is not fronted by an SLB. The local hostname was NOT exported. The exported data can be imported on any host but resecure of all agents will be required. Please see the EM Advanced Configuration Guide for more details. Exporting configuration for pluggable modules... Preparing archive file... Backup has been written to file: /home/oracle/opf_ADMIN_20130125_101524.bka The export file contains sensitive data. Please ensure that it is kept secure. ExportConfig completed successfully! [oracle@pr2prodora01 bin]$ cd $HOME [oracle@pr2prodora01 ~]$ scp opf_ADMIN_20130125_101524.bka oracle@aa2prodora11.jstor.org:/home/oracle oracle@aa2prodora11.jstor.org's password: opf_ADMIN_20130125_101524.bka 100% 45MB 5.0MB/s 00:09 [oracle@pr2prodora01 ~]$ Slide 25
  • 26. Installation Steps: (Silent) OMSCA Breakdown on Standby database using the backup file created • Uses the backup file just generated = bka file • Admin Server HTTPS Port = 7101 • Managed Server Port=7202 • Managed Server HTTPS Port=7301 • EM Note Manager Port = 7403 • EM Upload Port = 4889 • EM Upload HTTPS Port=4900 • EM Console Port = 7788 • EM Console HTTPS Port= 7799 • Config_home = /u01/oracle/product/OracleHomes2/gc_inst • EM_INSTANCE_HOST aa2prodora11.jstor.org (Standby machine) Slide 26
  • 27. Installation Steps: (Silent) Of all these ports the following are relevant to the load Balancer: •EM Upload HTTP Port: 4889 •EM UPLOAD HTTPS Port: 4900 •EM Central Console HTTP Port: 7788 •EM Central Console HTTPS Port: 7799 Slide 27
  • 28. Installation Steps: (Silent) • Run OMSCA on Standby database using the backup file created <OMS_HOME>/bin/omsca recover -ms -backup_file /opt/oracle/product/backup/opf_ADMIN_20120504_031016.bka -AS_HTTPS_PORT 7101 -MSPORT 7202 -MS_HTTPS_PORT 7301 -EM_NODEMGR_PORT 7403 -EM_ UPLOAD_PORT 4889 -EM_UPLOAD_HTTPS_PORT 4900 -EM_CONSOLE_PORT 7788 -EM_ CONSOLE_HTTPS_PORT 7799 -config_home /opt/oracle/product/omsmdw/gc_inst -EM_INSTANCE_HOST example.com !!! Many errors at this stage, due Failed attempts Slide 28
  • 29. Installation Issues: MSEVERE: OMSCA-ERR:Post "Deploy and Repos Setup" operations failed Check the following items : -- Check RAM on Standby Machine – needs 6GB – to Install -- If using 32 Bit Linux – Modify – create_domain.py (under omsca/scripts/wls directory) Change - -Xmx1524m to –Xmx1324m Change –XX:MaxPermsize=612M to –X:MaxPermsize=412M -- Check for Lock in repository Slide 29
  • 30. Installation Issues • Check for lock in the repository • Due to multiple tries there was a lock in the repository during the plugin deployment stage. • select host_url, oms_id, status, is_first_oms, em_release_version from gc_oms_info; • select mp_guid,file_id,file_name,file_type from mgmt_mp_files; • select * from em_plugin_lock_track; Slide 30
  • 31. Installation Issues: • First Issue – wrong version of the database plugin • I upgraded the database plugin but didn’t include it in the standby installation • Copied latest “DB” plugin and reran pluginInstall.sh Slide 31
  • 32. Installation Issues: • 2nd Issue – Plugin Issue and multiple attempts caused lock in EM database (SYSMAN) • Removed record from lock table em_plugin_lock_track • Ran $OMS_HOME/bin - ./pluginca –action deploy –isFirstOMS false –plugins • Started OMS Slide 32
  • 33. Final Installation Steps If you haven’t installed the agent on the standby host: $<AGENT_HOME>/sysman/install/agentDeploy.sh AGENT_BASE_DIR=<middleware_ home>/agent OMS_HOST=<second_oms_host_name> EM_UPLOAD_PORT=<second_oms_ port> AGENT_REGISTRATION_PASSWORD=<password> -configOnly Note: If you have Server Load Balancer (SLB) configured, then directly enter the host name and port number of the SLB. If SLB is not configured, then enter the secure upload port of the first OMS. Slide 33
  • 34. Final Installation Steps  After agent is installed and deployed • Deploy the required plug-ins on the management agent  Secure the agent - $AGENT_HOME/bin/emctl secure add_trust_cert_to_jks (If applicable)  In the new OMS Grid control (standby) – discover the Oracle Weblogic services • Targets, Middleware, click EMGC_DOMAIN • From EMGC_DOMAIN home page • Select Farm Menu, Refresh Weblogic Domain, click continue till 2nd management server is discovered on the standby OMS Host. Slide 34
  • 35. Installation logs / help • Plugin Logs: • $OMS_HOME/cfgtoollogs/pluginca/* • $OMS_HOME/cfgw/CfmLogger_<timestamp>.log • Repository Configuration logs • $OMS_HOME/cfgw/CfmLogger_<timestamp>.log • $OMS_HOME/sysman/log/schemamanager/m_<timestamp>/* • MDS Schema Configuration Assistant • $OMS_HOME/sysman/log/schemamanager/m_<timestamp>/* • $OMS_HOME/cfgtoollogs/cfgfw/emdcreate*.log • More Log file and Troubleshooting tips: http://docs.oracle.com/cd/E24628_01/install.121/e24080/appdx_troubleshooting.htm#CEGFAHIG Slide 35
  • 36. What We Covered • High Available architecture options • Silent Standby OMS installations and gotchas Slide 50
  • 37. References Deploying a Highly Available EM12C Cloud Control – An Oracle White Paper – May 2012 - # 1631423.pdf www.dbakevlar.com – Kellyn Pot’Vin’s blog contains many great ideas about EM12c e24089.pdf – Cloud Control Advanced Installation and Configuration Guide (section 5 - installing Add'l OMS in silent mode)
  • 38. References https://forums.oracle.com/forums/category.jspa?categoryID=70 – OTN Discussion on EM www.oracle.com – Documentation is not bad for many of the basic items with EM12c Keeping your OMS Tidy – Ray Smith - IOUG Tips and Best Practices – 7th edition - www.ioug.org http://oramanageability.wordpress.com - Ray Smiths Blog on Cloud Control
  • 39. References www.pythian.com – MySQL plug-in and instructions oemgc.wordpress.com – Rob Zoeteweij – Good information on Incident management and ITL, including a blog www.slideshare.net/Enkitec/em12c-monitoring-metric- extensions-and-performance-pages#btnPrevious – Em12C Monitoring, Metric Extensions and Performance pages – By Kellyn Pot’Vin – Enkitec EM12C Log Cleanup – Script to clean up logs and information
  • 41. Thank You for Attending Please fill out your Evaluations www.ioug.org/eval Cloud Control Recipes for EM12C Session # 941 Mike Gangler •Email: michael.gangler@ithaka.org • Twitter: @mjgangler • Blog: mjgangler.wordpress.com Slide 55

Editor's Notes

  • #2: ----- Meeting Notes (3/12/13 19:40) ----- Ask Questions regarding using CLoud control How many used 11g Grid Control? 10G 9 ? Earlier - ?
  • #3: I have worked for Oracle a long time since version 4 and 5.1c and have seen the growth of Enterprise manager / Grid Control / Cloud control since the early days of the “Work Group Stations”, to Version 10 and now Version 12c. I never used Version 11g of Grid Control and havent heard good things about that
  • #5: Used 9i or less (i.e., workgroup) ? 10g Grid Control ? 11G Grid Control ? 12G Grid Control ? Additional OMS ?
  • #6: Used 9i or less (i.e., workgroup) ? 10g Grid Control ? 11G Grid Control ? 12G Grid Control ? Additional OMS ?
  • #7: Used 9i or less (i.e., workgroup) ? 10g Grid Control ? 11G Grid Control ? 12G Grid Control ? Additional OMS ?
  • #9: Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- Level 1 is all contained on 1 site/machine and if that machine fails your OMS is not monitoring. Level 2 is is the same as level 1 but you have 2 OMS setup in an active / Passive mode - failover is via VIP or Load balancer
  • #10: Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- THis level 3 definition is the standard definition, in our situation i created a 3.5 level without RAC or load balancer.
  • #11: Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- you could have multiple Standby OMS in active mode with the load balancers
  • #12: One of the other reasons for setting this up is to consolated our job scheduler and jobs (Out of cron). This would help create a high available job scheduler as well. Level 3 utilizes multiple Mangement Services (Primary and Secondary) accessed through a local load-balncer and a databases that uses RAC. Also they recommend that the database and OMS are on the same host or close proximity to reduce latency. Level 3 provides continuous availabilyt when a database host or OMS host fails. Furthermore, to inenhance the the availability, we added a Dataguard setup. Note – Level 3 doesn’ t protect against site failure. Level 4 would put the Standby OMS and dataguard on athnterr rac site. ADDING RAC Gives you additioinal protection – Level 4 ----- Meeting Notes (3/12/13 15:45) ----- The dev/qa environments allows you to test patches, plugins, etc prior to going to produciton ----- Meeting Notes (3/12/13 15:48) ----- This will require at least 5 machines - 1 for the oms, 1 for standby OMS, 1 for dg observer, 1 standby db, and 1 primary db.
  • #14: Produciton – 24 hours a day – 7 days a week Development/QA – 14 hours a day – 6:00am – 8:00pm – Monday – Friday SqlServer Databases – 14 Hours a day – 6:00am – 8:00pm – Monday – Friday MySql Databases – 14 hours a day – 6:00am – 8:00pm – Monday - Friday
  • #15: NOTE – The diagram should have a DNS in between the Datacenters and the OMS and 1 between the databases. ----- Meeting Notes (3/12/13 15:55) ----- This presentation wont be focusing on the data guard, NFS Shared library or the load balancer, but on setting up the standby OMS, which currently is very buggy and hopefully will be fixed in the next release.
  • #16: ----- Meeting Notes (3/12/13 15:55) ----- NFS Shared is a
  • #17: ----- Meeting Notes (3/12/13 15:55) ----- additional hardware is talking about the dg observer.
  • #18: If you want to put your Primary and Standby DB’s on separate machines, that will also be required. This application is not a friendly application for shariing (i.e. database farms) and will use the entire instance for users, etc.
  • #20: ----- Meeting Notes (3/12/13 15:55) ----- Silent install is currently required due to Buggy GUI interface ----- Meeting Notes (3/12/13 20:58) ----- after root.sh DO NOT RUN - caps
  • #22: ----- Meeting Notes (3/12/13 20:58) ----- Definitiion
  • #23: Note this is where one of the problems occurred – There was version mismatch ----- Meeting Notes (3/12/13 20:58) ----- i.e.,
  • #25: ----- Meeting Notes (3/12/13 20:58) ----- $OMS_OME ==? $OMS_HOME
  • #26: ----- Meeting Notes (3/12/13 16:04) ----- Exports the OMS and security properties as well - this output is saying that the Local host name was not export
  • #27: The command is OMSCA in the OMS HOME/bin directory. This is all the settings
  • #28: ----- Meeting Notes (3/12/13 20:58) ----- UPLD - Slide 24 caps - UPLAD
  • #31: ----- Meeting Notes (3/12/13 20:58) ----- Open paranethesis
  • #34: ----- Meeting Notes (3/12/13 20:58) ----- havent
  • #35: ----- Meeting Notes (3/12/13 20:58) ----- add --&gt; add space after comma
  • #37: Events - The Entity Could be a target, config file, job, etc. - Example is database down, job failed, host cpu Percentage exceeded. Incident – Might be as simple as the relation with a singe event – Alert Log ora-0600 error or more completex as an server running out of resources (CPU, memory, io
  • #39: Example of our production database incident rule set Broken Down into : DB Critical Events, DB Up/Down event and Streams Only rule sets
  • #42: These alerts / rules will automaticallly clear out events based on the
  • #43: I found most of them didn ’t pertain to our environment or making my own provided better alerts and notifications.
  • #44: I found most of them didn ’t pertain to our environment or making my own provided better alerts and notifications. The only ones we use are Targets Unreachable and target error. Note most are based on Events
  • #45: Example of some of the criteria and Actions for each out of the box rule sets Service Level Agreements – Any / All Service Level agreement events that are critical will create an incident but NOT EMAIL Targets Unreachible – Any Host or Agent event that indicate unreachible – Create An Incident and send an Email to the sysman users Targets Down – Any Target availability Event will just create an incident – NOT EMAIL – (I created my own for hosts, databases, etc) Target Error - Any Errors regarding a target availability Event – will send an email to the sysman users and create an incident Clear ADP &gt; 7 days- if an event has been open for 7 days and not associated with an incident - clear
  • #47: This is an example of our Host Critical rule set: If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.
  • #48: This is an example of our database rule set: We have 3 rules : - If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.
  • #49: This is an example of our database rule set: We have 3 rules : - If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.
  • #50: This is an example of our database rule set: We have 3 rules : - If any host metric Alerts match the following – Severity in Critical, fatal and category in Availability and capacity) Do the following : Email to the syssman users Create an incident – assign to sysman user Update incident (If duplicate) and set it to sysman And then clear the event if no longer an issue.