Pandora FMS: Oracle Enterprise Plugin

Pandora FMS
Administrator Manual
Oracle Monitoring on Unix environments

Administrator Manual Oracle Monitoring on Unix environments
© Artica Soluciones Tecnológicas 20052012
Index
1Changelog...........................................................................................................................................3
2Introduction........................................................................................................................................4
3Compatibility matrix..........................................................................................................................5
4Compulsory documentation to hand in by the area that requires the monitoring .............................6
5Plugin modules...................................................................................................................................7
5.1.Multi instance Monitoring.........................................................................................................8
5.2.Monitoring Automatic Pre-conditions ......................................................................................8
5.3.Plugin parametrization...............................................................................................................9
5.3.1.Location of the Oratab file.................................................................................................9
5.3.2.Information and Error Log ................................................................................................9
5.3.3.“Skip” Checks by default.................................................................................................10
5.3.4.Configuration block of one instance................................................................................10
5.3.4.1.Configuring the listener and services.......................................................................10
5.3.4.2.Monitoring Preconditions ........................................................................................11
5.3.5.Restarting of the listener..................................................................................................11
5.3.6.Monitoring Specifics SID................................................................................................11
5.3.6.1.Monitoring an Specific SID(sid)..............................................................................12
5.3.6.2.SID Specific Monitoring(only_sid)..........................................................................12
5.3.6.3.Monitoring Specific SID (skip_sid).........................................................................12

1 CHANGELOG
Date Author Change Version
29/09/11 Juanma First Version v1r1
13/12/11 Juanma New features v1v4
22/12/11 Juanma New features v1r5
13/03/12 Juanma New features V1r7
18/06/12 Juanma New modules V2r1
25/07/12 Juanma New features v2r2
17/09/13 Mario New features v2r3
Page 3

2 INTRODUCTION
The main aim of this document is to describe the Oracle systems monitoring based on Unix. Some
“base” modules have been chosen according with our experience in system monitoring and the
necesities of some of our clients. Besides, all the specifications collected in different real production
environments have been added, taking real specifications of database administrators.
For extracting information we use:
• An external configuration file where all the plugin parameterization is defined. This
configuration file could do calls (includes) to other files.
• One file of environment variables parameterized by instance.This file is located at
$ORACLE_HOME/dbs/orauser_$SID directory.
• We use the software that is already installed in the system (SQLplus, lsnrctl,Oracle alert
files, etc), for the monitoring done by the plugin without having to install libraries or third
parts utilities. Optionally, the ORATAB file could be used if the instances to monitor are not
configured directly in the plugin configuration file.
• An already existing log parser is used (the one from Pandora) to process Oracle alert
logs . The model to search could be parameterized through one token in the plugin
configuration file. The detection of the alert log files will be automatic for each instance.
• Autodetection of SID, instances, and system tablespaces are done, but the administrator
could be deleted, and/or force specific SID , configuring different tokens in the plugin
configuration file .
• Some basic checks “by default” are done, but they could be deleted or customized.
• An “open” interface is available to specify free SQL queries, allowing to modelate all
kind of SQL queries that are done with other tools or in a manual way for the
administrators. We also provide some queries usually used by administrators in all the
world, and that offer important imformation about the performance.
• The system is integrated with the Unix agent and it could distribute file colections, so it is
posible to distribute the plugin by one hand and the file collections in an individual way by
agentand/or by policy.
We need to mention that as with the rest of monitoring with Pandora FMS, the Oracle monitoring
plugin could be used to collect information type “text string”(to process it as events) or type
numeric (to do performance management).
Page 4

3 COMPATIBILITY MATRIX
The compatibility matrix for the plugin is shown here:
Systems where it has been tested
• Solaris 10 Sparc, Oracle 11g
• Linux RHEL 5, Oracle 10.x, Oracle 11.x
Systems where it should work
• Solaris 10 and higher, Oracle 11.x and
higher
• Linux RHEL 5 and higher, Oracle 10.x,
Oracle 11.x and higher
Page 5

4 COMPULSORY DOCUMENTATION TO HAND IN BY THE AREA THAT REQUIRES
THE MONITORING
In order that the Oracle monitoring could be done correctly, it is necessary that the technical Area
sends same specific information that will be included in the configuration files. This information is
the following:
• User and password with access to the Oracle engine via SQLplus. This user should have
permission to read in all the tables of the system (see following point) and be in the Oracle
system group.
• Name of the DDBB instance (Oracle SID).
• In case of using the ORATAB file: Verify that there is a file /var/opt/oracle/oratab(or in
other path as /etc/oratab) that is correct and also with the paths and SID corrects.
• If you want to monitor other component that does not come defined “by default” it will be
necessary that you gives the SQL code to do this monitoring, and also an exit data example,
specifying if it is numeric, kind string, etc).
• If the DBA environment variables (like PATH to the SQLplus and lsnrctl binaries,
TNS_ADMIN, ORACLE_HOME, ORACLE_BASE, etc.) are not exported as environment
variables, the plugin will need access to the environment variables file:
$ORACLE_HOME/dbs/orauser_$ORACLE_SID of each instance and then you could show it
specifically in the plugin configuration file.
Page 6

5 PLUGIN MODULES
The plugin does“by default” the following things:
• Scanning of the tablespace to get free space.
• Scanning of the tablespace to get status.
• Scanning of Oracle filesystems working on the system (in base of one prefix and on the
name of the instance (SID).
• Verification of the connection to the DB (through one query (Select) and getting one date).
• Verification of the services with a listener (only at instance level).
• Verification of Pmon process of each SID .
• Verification of the general status of the listener (for all instances).
• Error parsing looking for a regular expression in the alert log. The alert log depends on each
instance and its exact location is auto detected.The plugin is necessary to parse Pandora logs
in order to process these logs.
Besides it also monitors the following performance parameters, such as is recommended by many
administrators who are experts in Oracle for different kinds of environments:
• Dictionary Cache Hit Ratio
• Library Cache Hit Ratio
• DB Block Buffer Cache Hit Ratio
• Latch Hit Ratio
• Disk Sort Ratio
• Rollback Segment Waits
• Dispatcher Workload
• Active sessions
• Concurrent parallel sessions
• Redo log write time
• Redo log buffer space wait
Page 7

• Enqueue waits
• Free buffer waits
NOTE: Its important to say that if the database to monitor does not have configured some of the
statistics before mentioned, then the corrresponding modules will figure as not initialized in the
Pandora FMS console.
None of the performance parameters has associated by default any kind of event, and it is posible to
assign thresolds to assign them alerts.
All these modules come parameterized in the “oracle.conf” file that comes in the plugin package.
These modules are SQL, that could be checked in the file and that could be deleted, modified or
extended by an Oracle administrator.
You can find more information about these queries and other, recommended by several Oracle
administrators communities in the following link:
http://www.hoopoes.com/cs/oracle_tune.shtml
5.1. Multi instance Monitoring
The plugin, besides doing a monitoring in a general level of all the instances, allows to monitor
several instances in a more specific way.
This way the plugin allows a more flexible configuration:
• Monitoring at a general level: By one part, the monitoring by default will be done, as is
explained in the previous point.
• Monitoring at an instance level: You could configure the monitoring by specific instance.
NOTE : It is important to mention that the monitoring at an instance level has priority over the
general monitoring, so it is considered as a more specific one. This will avoid that duplicated modules
will be generated in case that the same instance will be configured for general checks and in a block of
instance configuration(see in the following point) of the configuration file.
In order that the modules of the different instances dont delete themselves, the notation that the
modules use will be
ORA_{SID}_{Comprobación}
5.2. Monitoring Automatic Preconditions
To avoid that duplicated modules would be generated, some conditions have been fixed that should
be fulfilled before checking. If these conditions are not satified, then, the monitoring will be
aborted. These conditions are at two levels:
Page 8

• At a general level: the access to the configuration file will be verified, and also that the SQL
plus binary would be available, the listener status (if the listener verification is not available)
and that the pmon process is running (if it is not disabled)
• At an instance level: It will be check if there is an script that determines the monitoring of the
corresponding instance, the pmon process is verified for that instance (if it is not disabled),
then the listener status is verified and also their services (if the listener verification is not
disabled).
5.3. Plugin parametrization
The plugin is used through the configuration in an external file of the configuration
NOTE: It is extremely important to consider that the configuration files thought for the UNIX plugin should
be edited and stored with carriage returns kind “UNIX” and that if “WINDOWS” carriage returns are used,
then the plugin will not work correctly.
There are some specific checks that have their own configuration “tokens” that are described next:
5.3.1. Location of the Oratab file
In case of using it. Remember that the oratab file stores the instance information and the home directory of
that instance. By default, the script understand that the oratab file is located at:
/var/opt/oracle/oratab
If the oratab file would not in that location, it is posible to specify an alternative in the
configuration file.
oratab /etc/oratab
This will do that it looks for the oratab file in an alternative location. In this example the /etc.
directory.
5.3.2. Information and Error Log
All the actions done by the plugin will be registered in a log file which name will be parameterized in the
following way:
/tmp/pandora_ora_{SID}
Page 9

5.3.3. “Skip” Checks by default
It is posible to “skip” the checks by default using the correct configuration token:
skip_listener_status
skip_tablespace_free
skip_tablespace_status
skip_oracle_fs
skip_connect_check
skip_alert_log
skip_pmon
These tokens will be applied at a general level or at an instance level. This will allow to do the
monitoring flexible. For example, supposing that in oratab we have configured the “hpsacert”
instance and in an instance configuration block we have it configured also (see next). If we want to
do an specific parse on the alert log of this instance and not on the other ones, then we should leave
out the skip_alert_log token (see next) in the instance configuration block . As we mentioned before, the
monitoring at an instance level has priority over the general one, so an specific parse will be executed for
this instance.
5.3.4. Configuration block of one instance
To do the specific configuration of one instance, you should use the following tokens:
instance_begin
instance_name xxxxxx
<tokens de configuración>
instance_end
On this block, it is posible to use the tokens that have been explained before, and this way to chose
if the corresponding check would be done or not.
Besides, the following tokens could be used only in this block:
5.3.4.1. Configuring the listener and services
In order that the plugin monitors the service status through the listener, you should configure both values, the
listener name and the name of each of the services . For this we will use the following configuration
syntax:
listener_name LISTENER_RHGE0208
This will define the name of the listener, in this example “LISTENER_RHGE0208”. It should be the
Page 10

exact name of the listener that we want to monitor.
By other hand, it is necessary to specify the service or the services that we want to check that are listening in
this listener.This is done in the following configuration tokens:
service OIM
service OIMXA
service OIMXDB
service OIM_XPT
NOTE : In case there were several listeners in a machine, we should configure these tokens in each of the
configuration blocks of the instance.
5.3.4.2. Monitoring Preconditions
It is posible to determine the monitoring of one instance if one scrip returns 1. For this, it is posible to use the
token conditional_monitoring in the instance block.
For example:
conditional_monitoring /var/plugin oracle/conditional script
NOTE: You should write the complete path of the script. In case that the path or the name of the script have
spaces, they should be masked.
5.3.5. Restarting of the listener
In case that the listener checking (would be it like global check or an instance check) does not answer, it is
posible to enable one scrip for it could be executed and try to restart the listener. Each time that this script
would be executed, the number of the listener restarts should be stored in the file o /tmp/
{sid}_listener_restart. This counter will be reset every 24 hours. The format of this file is the following:
RefDate: yyyy-mm-dd hh:mi:ss
Reinicios totales: xx
RefDate will have the date in which the listener restart counter was reseted.
5.3.6. Monitoring Specifics SID
It is posible to limit the number of instances to which we are going to do the checks in a general level.For
doing this we have the following tokens: sid, only_sid y skip_sid.
Page 11

5.3.6.1. Monitoring an Specific SID(sid)
In the common parameters for all the instances, it is posible to configure this parmeter:
Example:
sid <nombre instancia>
With this, all checks at a general level will be done only on this instance, without consider the rest
of the configured instances.
5.3.6.2. SID Specific Monitoring(only_sid)
This token is useful to limit the number of instances to monitor. Through the only_sid token the system is
forced to monitor that SID (or several if we introduce several lines).
Example:
only_sid pandora01
With this call, of all SIDS that the DDBB contains, it will only monitor the “pandora01” instance.
5.3.6.3. Monitoring Specific SID (skip_sid)
If with the token only_sid xxxx we limit the number of objective instances, the token skip_sid is useful just
for the contrary. With it, the instance (or instances) selected will be not included in the global checks.
Example:
skip_sid pandora01
So, then we will not include the instance “pandora01” in the global checks.
5.3.6.4. Assignment of these tokens
Due to that these tokens are not compatible between them, the way to assign them will be the following and
Page 12

one of them will prevail over the other:
1. sid
2. only_sid
3. skip_sid
5.3.7. Monitoring the Disk volumes of one Instance
The system will try to “auto detect” the volumes in base of one “prefix” and to the name of each monitored
instance. For this, we use the token volume_preffix to define the prefix of all the Oracle volumes.
For example:
volume_preffix /aplicaciones/oracle
If one of the SID is “pandora01”, then automatically will detect all the disk volumes contained in one path
that has “oracle/applications” and that has “pandora01” in its name, as for example:
/aplicaciones/oracle/arc_pandora01
/aplicaciones/oracle/rdbms_pandora01
5.3.8. Not using the ORATAB file
In some of the Oracle installations (for example the RAC ones) there is not available one ORATAB file that
reflects the different instances that are going to be monitored, so this file will not be used. This way you
should show specifically through tokens only_sid in the configuration file the Sids of the instances to
monitor and the variable $ORACLE_HOME.
For example:
skip_oratab
only_sid pandora01
only_sid AST1
oracle_home /oracle/product/11.0.1
5.3.9. Monitoring “Separated” disk volumes
The system tries to “auto detect” the Oracle volumes, according ot the previous point. If it is not able to
detect it, or the administrator want to include “separated” volumes, then it is posible to monitor using the
volume token.
For example:
volume /var
Page 13

volume /oracle
5.3.10. Access Credentials
They are necessary to use the plugin
user sys
password pandoraA01 AS SYSDBA
Note: The use of the connection as SYSDBA is optional. The same line could be like this (if the user has
permissions for the tables and special views that are required).
user oracle
password oracle
Please, consider that here the “full”mode is not used, and that a kind of numerical data is used.
5.3.11. Log parser
By default, we use the log parse plugin that is located in the path /etc/pandora/plugins/grep_log but it is
posible to modify the path through the logparser token. For example:
logparser /var/opt/PandoraFMS/etc/pandora/plugins/grep_log
5.3.12. Processing the Alert logs
To process the Oracle alert logs, you should omit the skip_alert_log token and configure the log_pattern
token as a regular expression. For example, to search the message ORA- in the alert log:
#skip_alert_log
log_pattern ORA-*
5.3.13. Monitoring vía SQL
One of the most powerful features of the plugin is the posibility of specify its own SQL order to get the
value. Let's see some examples:
Page 14

custom_query_begin
custom_query_name NUM_MAX_FICHEROS
custom_query_sid xxxx
custom_query_type generic_data
custom_query_sql_begin
select round ((ficheros_act/ficheros_max)*100) ratio
from (select count(*) ficheros_act from dba_data_files),
(select value ficheros_max from v$parameter where name='db_files');
custom_query_sql_end
custom_query_end
Custom_query_sid (Optional):Defines one SID for this SQL query, and it will only execute it for this
instance. If it is not specified it will be executed for all instances.
Custom_query_type: It should be of a valid type in Pandora, for example generic_data for numeric data or
async_string for text strings .
Custom_query_mode: It will be “full” when we want to get all the exit (for example one list). When we use
the “full” mode we always should use a kind of data kind string (async_string).
Custom_query_description It is optional and is useful to send a description with the module.
Custom_query_condition <script> If this script returns 1, the query of this block will be executed. On the
contrary, it will not. You should write the complete path and mask the spaces.
Lets see an example that uses an SQL query to get discrete values:
custom_query_begin
custom_query_name SQL_SESS_CONCURRENTES
custom_query_type generic_data
custom_query_condition /var/oracle/conditional script
custom_query_description Devuelve el porcentaje de sesiones concurrentes.
custom_query_sql_begin
select round ( (sesiones*100)/value )
from (select count(*) sesiones from v$session), v$parameter ;
custom_query_sql_end
custom_query_end
Please consider that here we do not use the “full” mode, and that we use a kind of numeric data
instead.
In the plugin package is distributed one script (pmon_ASM_test) that allows to check if the process
pmon_+ASM is running and that could be used with the token custom_query_condition. If so it will give
back 1. On the contrary, it will return 0.
Page 15

5.3.14. Creating modules for logs by hand
So as the error log module does not send information until there is an error, the module should be
created manually. The name of the module always depends on the DDBB instance. The DDBB
instance names will be shown in the agent monitoring in serveral places, for example:
In this case, the DDBB instance is called “hpsacert”. For it, we will create manually a new
module, like this:
And after we should create an alert. Before assigning the alert templatea we should be sure that we
have an alert template that sends an event when we get something in the module. This is done with
a template similar to this:
The fire conditions are special, so we will be interested in get a short timethreshold, just in case that
more events come, that will be always different and that could be sent. However, we will put it a
minimum time to avoid an storm of repeated events.
Page 16

And finally, we will asign the alert module log to the error template in logs:
6 REQUISITES
The requisites for this monitoring could work properly are these:
Page 17

• To install the Pandora FMS agent.
• To have one Oracle installed in the machine where it is going to be monitored, with the
basic tools (SQLplus, lsnrctl).
• To specify the name of the Oracle listener that you want to monitor, and also the name of
each service that you want to monitor in the listener for each of the instances reflected in
the configuration file. Both things are necessary to monitor the listener at an instance level.
• It is necessary that the user with which the Pandora FMS agent is executed, that is the user
that will execute the plugin, has access to the following resources of Oracle:
◦ SQLplus
◦ lsnrctl (correctly configured).
◦ All DBA environment variables exported to the user that executes the plugin. This
user also needs to belong to the Oracle system group.
◦ Alert log files (dinamically got for each instance).
◦ The PATH to the SQLplus and lsnrctl binaries should be available for the pluguin:
▪ If a monitoring is done on a single instance it could be exported as environment
variable:
▪ The plugin will search this variable in the environment variable files
$ORACLE_HOME/dbs/orauser_$ORACLE_SID and in case that you find them it
will load them.
◦ The variables $ORACLE_HOME y $ORACLE_SID are available for the plugin. In case
of using the ORATAB file, they will be extracted from this file. In case that you dont
use this file, you could show it in the plugin configuration file or they could be
charged as environment variables in case of monoinstance monitoring.
• To do the monitoring, the plugin will call to SQLplus to do different SQL queries getting the
information.
• The plugin should be able to write temporal files in the path /tmp
• It should be necessary to give the list of disk volumes to monitor and in case of doing the
listeners monitoring, the listener and services list associated.
• The user that executes the Pandora FMS agent should belong to the database explotation
group to could get access to the the tnsnames.ora file
Page 18

6.1. Requisites to could have access to the DDBB
• The plugin needs an user and a password to connect with the DDBB. This user should have
enough privileges to check some of the system tables, in order to get information.
• The user that is provided could be “normal” or use a connection like SYSDBA, in any case
this is specified in the “password”parameter of the plugin configuration file. And belong to
the Oracle system group.
• Preparation of the database. It is necessary to have an user with access priviledges for some
of the tables. In this example is detailed how to give access to same tables that the plugin
use by default, for the user “pandora” with password "pandora". The plugin will always do
the connections to the local machine (localhost).
CREATE USER pandora IDENTIFIED BY pandora;
GRANT CREATE SESSION TO pandora;
GRANT SELECT any dictionary TO pandora;
GRANT SELECT ON V_$SYSSTAT TO pandora;
GRANT SELECT ON V_$STATNAME TO pandora;
GRANT SELECT ON gv$sysstat TO pandora;
GRANT SELECT ON v$sesstat TO pandora;
GRANT SELECT ON V_$INSTANCE TO pandora;
GRANT SELECT ON V_$LOG TO pandora;
GRANT SELECT ON SYS.DBA_DATA_FILES TO pandora;
GRANT SELECT ON SYS.DBA_FREE_SPACE TO pandora;
GRANT SELECT ON V_$parameter TO pandora;
GRANT SELECT ON dba_tablespaces TO pandora;
GRANT SELECT ON dba_data_files TO pandora;
GRANT SELECT ON dba_free_space TO pandora;
.
.
(others GRANTs necessary, for all tables used in the plugin configuration file)
.
.
--
-- if somebody still uses Oracle 8.1.7...
GRANT SELECT ON sys.dba_tablespaces TO pandora;
GRANT SELECT ON dba_temp_files TO pandora;
GRANT SELECT ON sys.v_$Temp_extent_pool TO pandora;
GRANT SELECT ON sys.v_$TEMP_SPACE_HEADER TO pandora;
GRANT SELECT ON sys.v_$session TO pandora;
Page 19

7 INSTALLATION
Copy the plugin to the agent plugin directory, or distribute it with file collections. Do the same with
the conf.file . The call from the agent will be similar to this, but using the paths where the plugin
and the conf are installed.
module_plugin perl /var/opt/PandoraFMS/etc/pandora/plugins/oracle.pl
/var/opt/PandoraFMS/etc/pandora/collections/fc_23/oracle.conf
Page 20

8 USING THE PLUGIN IN POLICIES
The use of this script with policies is very easy. It is based on creating a policy that contains the
plugin (pandora_oracle.pl) and a configuration file for each agent. This way, the call to the policy
plugin would use the $HOSTNAME variable to replace it in time of execution by the hostname
(complete) of the machine. This is an example:
If the machine is called ilp0x068.tsm.inet, we should upload a file named ilp0x068.tsm.inet
oracle.conf as we can see in the following screenshot.
The content of the .conf is specific for each system, but all the .conf are copies to all the systems
associated to the policy. This way, although with permissions only the user that executes the
pandora agent could read these .conf (where the access password to the oracle are), it is
recommended to use user of only reading and that these users will have only local access.
Page 21

Pandora FMS: Oracle Enterprise Plugin

More Related Content

Similar to Pandora FMS: Oracle Enterprise Plugin (20)

More from Pandora FMS (20)

Recently uploaded (20)

Pandora FMS: Oracle Enterprise Plugin