Soa config con8627_pdf_8627_0001

CON8627- Administration and
Management Essentials for
Oracle SOA Suite 11g

Ramkumar Menon
Oracle SOA Suite Product Management

Francis Ip
Oracle SOA Suite A-Team


The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material,
code, or functionality, and should not be relied upon in making
purchasing decisions. The development, release, and timing of any
features or functionality described for Oracle s products remains at
the sole discretion of Oracle.


Program Agenda

§  Overview

§  Provisioning and Configuration Management

§  Managing a System in Production

§  Transactions and Fault Handling

§  Troubleshooting

§  Q&A


Overview
Mediation
BPEL Rules

Human
Task
BPEL
orchestration
Human
Tasks
Web
services SCA Composite
Business Rules and
adapters

SOA Composite Editor
BPMN BPEL
Human
Task
…
Policy
Management Policy Manager
Optimized Service Infrastructure
Service & Eventing
Infrastructure SOAP
B2B
HTTP
JCA
EJB
…
Common Connectivity
Infrastructure


Program Agenda

§  Overview





§  Q&A


SOA Domain
Overview

§  Overview Domain home Oracle home Typical SOA HA install
(configuration) (binaries)

EM Administration
Server
Managed
Server
… Managed
Server
SOA Managed EM
Server Cluster 1: SOA
Administration
SOA Server
BAM Managed
Server
BAM Managed
Server
… Managed
Server
Cluster 2: BAM
SOA_Domain Cluster_Domain


SOA Configuration
3 Types of Properties

§  WLS Properties
–  Persisted in domain configuration files, changes are propagated across the cluster by the Admin
Server.

§  Service Infrastructure (soa-infra) Properties
–  Properties are persisted in MDS (database), changes are automatically propagated across a
cluster.

§  Composite Properties
–  Meta-data associated with a composite is stored in MDS (database)
–  These properties can be customized using a “Configuration Plan” at the time of deployment.
–  Composite properties are automatically propagated across a cluster as part of the composite
deployment process.


SOA Domain
Directory Structure domain home

start/stop & env scripts

§  Configuration: configuration root

–  Stored under “config” directory
–  The config directory contains a file called config.xml which is
the primary source of all configuration information within a
domain.
–  SOA configuration is stored in MDS since PS1.

§  Scripts: local cache of deployed
composites

–  Stored under “bin” directory.
–  Start / stop Admin & Managed server
–  Set environment and JVM properties

§  Logs: sub-directory for each
managed server (logs /
–  Server logs are placed under servers/<server_name>/logs cache)


SOA 11g MDS Store

§  Runtime Store
–  Composite meta-data
–  Security policies
–  Error policies
–  B2B Agreements
–  Shared WSDLs and XSDs
–  ADF / Human Workflow

§  Can be accessed from JDeveloper
and Ant.
§  Configuration customization is stored
in MDS.


Configuration Propagation in a Cluster
WLS + Coherence Clusters
SOA 1

Admin SOA Cluster MDS

WLS Configuration SOA 2
SOA Configuration

WLS Configuration SOA Configuration
§  File based §  DB (MDS) based.
§  AdminServer has “gold” copy §  Database has “gold” copy
§  Configuration propagated by WLS JMX
§  Configuration propagated by soa-infra
framework.
using the Coherence cluster.
§  During startup, a Managed Server copies
configuration from AdminServer and §  During startup, a SOA server loads its
overwrites local copy. configuration from MDS.


SOA 11g HA Architecture
2 Levels of Clustering
§  WLS Cluster
–  Defined in config.xml
–  Can be viewed from WLS Console and
EM

§  SOA Cluster
–  Coherence based cluster
–  Defined as system parameters in
startup scripts
–  Can be validated through deployment /
configuration changes.


Tools for Configuring SOA Applications

§  SOA container configuration done through EM console /
WLST

§  Endpoint properties modified during deployment through a
Configuration Plan
–  Created
–  Adapter properties

§  WLS Console / WLST for managing WLS configuration
–  Domain creation, adding / changing resources within a domain.


Test to Production (T2P) Utility
Minimize Manual Steps
§  What is moved?
–  Installed binaries and patches
–  Configuration and metadata
–  Security config e.g. OPSS policies

§  Supports MDS meta-data, JCA
connection factories
§  MovePlan – Rewiring external
references to match target
–  Data-sources, endpoint addresses


Program Agenda

§  Overview





§  Q&A


Production Readiness

The Readiness Checklist

Sign off on Qualifying Pre-Production Tests
Clearly defined and documented processes
Trained personnel lined up for Go-live and Post-Production
Availability of a Production Replica Environment
Identify SLAs and Key Transactions


Tools for Middleware Management

For Farm with a single WLS Domain. Composite lifecycle management, Instance Tracking, Performance
EM FMW Control
Metrics, Fault and Message Recovery
EM Grid Control
Extends FMW Control for multiple farms. SOA/WLS Management Pack. Historical data, Alerts and
11g
Notifications
WLS Admin
Console Monitor and manage the WLS domain and Managed Servers. Configure JDBC, JMS, JTA, Security,
Deploy Applications etc.
JR Mission Control
JVM monitoring and management. Flight recorder for troubleshooting and diagnosis.

WLDF Generate and Analyze diagnostic data from WLS instances and your applications. Watches and
Notifications. Integrate with JFR for troubleshooting.

WLST Create/manage/monitor WLS Domains via Jython based WLI. WLS Admin Console/Configuration Wizard
on a command line.

BAM Real-Time Business dashboards. Correlate Business Indicators with System Metrics by integrating
with EM GC 11g


EM FMW Control
• SOA Composite Lifecycle
Management

• Transaction Search and Drill Down

•  Fault and Message Recovery

•  Service Engine Configuration

• Live In-flight DMS Performance
BPEL
Metrics
Ext Drill-down
• System MBean Browser for
ern
Hu into
component advanced configuration and metrics
ma
al
Rou
s

Ser
n
Bus
ting • Use FMW Control for searching and
tracking your composite instances
Wo
vic
ine
e
rkfl
ss
18
ow
Rul
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 13

FMW Control - Key Performance Tuning Tasks
§  Configure SOA-Infra Properties
•  Audit/Instance Tracking(CPU,DB),
Payload Validation (CPU)

§  Configuring Service Engine threads
•  BPEL - Threads(Concurrency), Audit
(DB), Validation (CPU)
•  Mediator - Parallel/Resequencer Threads
(Concurrency), Audit (DB), Sleep
Intervals/Metrics Level (CPU)

§  Configure Adapter Properties
•  MaxRaise/TxnSize(DB), Number of
Threads (Concurrency)

§  All this Accessible as MBeans - Update
via WLST


SOA Management Pack Enterprise Edition

§  Business Transaction Management
–  Discovery of transaction flows across the enterprise
–  Leverages transaction content/payload for business visibility Development
–  Exception Management Business Performance
Transaction &
§  Performance & Diagnostics Management Diagnostics
–  Enterprise scale multi-domain management
–  Specific framework/metadata knowledge
Production Test
–  SOA 10g (BPEL/ESB), SOA 11g (BPEL/Mediator/DS/HW)
Lifecycle
–  OSB (Business/Proxy Svc, Pipelines) Management
§  Lifecycle Management
–  Configuration Management: SOA infrastructure and applications
–  SOA domain and application provisioning
–  SOA Patching support


JRockit Mission Control
§  Negligible performance overhead.
§  Comprised of three toolkits.
–  Management Console - for managing and
monitoring one or more JVMs.
–  Runtime Analyzer – Application and VM
Profiler. Record and Playback for Root-
cause analysis.
§  Analyze patterns around GC, CPU, Latency, Locks etc

–  Memory Leak Detector

§  Integrates with WLDF for analysis of
diagnostic dumps
§  Use JRMC to root-cause performance
issues by analyzing memory, CPU, thread
contention, locks etc via Flight
Recordings. (Use for Support
engagements)

WLDF and DFW

§  Enable Just-in-time Proactive diagnostics
§  Reduce time and effort for collecting diagnostic information.
§  Support for SOA Suite Diagnostic dumps -
adf.ADFConfigDiagnosticDump,soa.composite, soa.composite.trail, soa.config, soa.db
soa.edn, soa.env, soa.wsdl
§  SOA specific DMS Metrics – Message Processing metrics
§  Manually trigger dumps


WLST
q .Jython based CLI
WLST q Create, monitor and manage WLS
Domains
q  Runs in Interactive,Batch or
Embedded Mode
Online Offline

Create
Composite MBean WLDF Integration Domain
Server Deployment access and to export
Lifecycle Templates/
and Lifecycle Updates Diagnostic Data
Domains
from
Templates,
Extend
DomainRunti ServerRunti Custom(For Domains etc
DomainMBean meMBean meMBean SOA)

•  Command line automation .
•  Hierarchical navigation via domain and server Mbeans:
•  retrieve and set domain metrics and properties
•  start/stop servers etc
•  WLST recording available

Capacity Planning

§  Hardware Resources (Memory, CPU)
§  Database Capacity
§  HA Requirements (Shared Storage)
§  Methodology
§  Retention policy
§  Data Archival


Purging
Via EM Console [For Targeted Deletion], or via Purge Scripts and/or dropping Partitions[For Bulk]
Scripts located at RCU_HOME/rcu/integration/soainfra/sql/soa_purge

§  Looping §  Parallel §  Partitioning
§  Dropping partitions is the fastest way
to purge.
§  Time-Range Partitioning Done at the
component level.
§  Decide time-range based on
§  Useful for small installations [~100k §  Useful for medium [ ~500 GB total to ~10 average and max composite lifetime.
rows] GB daily] to large installations §  Run this in conjunction with parallel
§  Loops and runs until max_runtime §  Resource intensive – hence run during purging for a mixed load.
loop parameter is exceeded. off hours.DOP indicates number of
§  Not resource intensive, but slow parallel threads.
performance. §  For large installs [>~500 GB total or >
§  All purgeable instances are deleted. ~10 GB daily], recommended to run a
hybrid of parallel purging + dropping
partitions, using optimal thread counts.


Program Agenda

§  Overview





§  Q&A


Fault Handling – Key Factors

§  Handled Faults
–  Handled programmatically – Component / Policy
–  Can specify manual intervention through EM
–  Principal transaction not rolled back

§  Un-handled Faults
–  Not handled programmatically
–  Typically leads to transaction rollback
–  Messages can be re-tried when they are retrievable
§  JMS queue, DB table, Dehydration storeetc.


BPEL – Threads, Transactions and Faults

Client
Services Client Services
Receive Client Receive Receive

Commit Commit

Invoke Invoke Invoke

Commit Commit

Reply
Partner
Receive
Commit
Commit

Invoke
Asynchronous
Synchronous Asynchronous Durable Commit
Process Process Process



Client Fault
Services Client Services
Receive Client Receive Receive

Commit Commit

Rollback

Rollback
Invoke Invoke Invoke

Commit Commit

Reply
Partner
Receive
Commit Invoke, Callback, Activity
Type Commit

Rollback
Invoke
Asynchronous
Recover, Mark Cancelled, Reset
Synchronous Asynchronous
Action Durable Commit
Process Process Process


Breakpoint Activities
Receive

“Breakpoint” activity is
transaction boundary
•  Receive
•  Wait
<Activity> •  Pick - onMessage
•  Pick - onAlarm
•  Mid-process Reply
•  Idempotent Invoke
•  Non-blocking Invoke
… •  Fault Policy Retries
•  …
exception or system crash
rolls message back into DB,
available to BPEL SE for
manual (activity) recovery


Mediator (3) Worker threads
process messages
Sequential vs. Parallel Routing
Mediator Mediator

…
Caller thread
executes routing

(1) Caller thread (2) A single locker thread
saves message retrieves messages in
batch

Tune the locker thread
batch size and # of worker
threads for parallel routing
rules


Mediator Service Engine (3) Worker threads
processes messages

Sequential vs. Parallel Routing
Mediator Mediator

…
returns
Caller thread exception (or
executes routing timeout) to
caller
(1) Caller thread (2) A single locker thread
saves message retrieves messages in
batch

Recover Mediator parallel
instances via EM console –
Transaction rolls back,
Faults page for soa-infra,
available for manual instance
composite or Mediator SE
recovery


Programmatic Fault Handling
Fault Policies
§  Fault Policies
–  Covers BPEL invoke, adapter rejection message
–  Are externalized fault handling rules (XML files)

§  Fault bindings
–  Bind policies to Reference, Service (inbound adapter), Component, Composite
–  Can define a conditions chain of actions

§  Actions
–  Retry, Abort, Replay, Re-throw
–  Custom Java actions
–  Human Intervention – recover from EM


Manual / Scripted Fault Handling - Recovery
Faults &
exceptions

§  Recovery Options for
Administrators:
–  Human intervention actions
defined by fault policies Recovery
screen log

–  BPEL message recovery for
unhandled faults
§  Auto-recovery configured via
MBean
§  Manual Recovery from BPEL
recovery console. Action
–  Rejected messages from
Edit payload
inbound adapters


Program Agenda

§  Overview





§  Q&A


Troubleshoot SOA

§  Oracle Fusion Middleware Diagnostic
–  Diagnostic Framework
–  Selective Tracing
–  SOA Service engine statistics
§  JDK Diagnostic
§  Operating System Diagnostic
§  Database and external resources diagnostic


Troubleshooting SOA
Oracle Fusion Middleware Diagnostic Framework

Log
Diagnostic Files
Diagnostic
WLDF MBeans
Dumps
data

WLST
Command Line
Automatic Diagnostic Repository Utilities
(ADR)

Troubleshooting SOA
§  “Problem”
–  listProblems(), …
§  “Incident”
–  listIncidents()
–  showIncident(id=‘1’)
–  createIncident()

§  “Dump”
–  listDumps(appName=‘soa-infra’)
–  describeDump(name=‘soa.env’, appName=‘soa-infra’)
–  executeDump(name=‘soa.env’, appName=‘soa-ifnra’)

§  “Diagnostic Rule”


Troubleshooting SOA
§  Configure WLDF Watch and Notification


Troubleshooting SOA


Troubleshooting SOA
Anatomy of a Log File 3
2
1
[2011-09-22T06:50:07.986-07:00] [AdminServer] [NOTIFICATION] []
4 [oracle.integration.platform.blocks.deploy] [tid: [ACTIVE].ExecuteThread: '3' for 5
6
queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: 7
3c584fd020bdf1ba:-65df75b4:132916143aa:-8000-0000000000000166,1:23265:2]
9
8 [APP: soa-infra] [Deployment] Component: SendMail was successfully deployed to
service engine:implementation.bpel, for composite:SendMailApp

1.  Timestamp 5. Thread ID 9. Message Data
2.  Server Name 6. User ID
3.  Message Type 7. Execution Context ID
4.  Component ID 8. Supplemental Attribute


Troubleshooting SOA
Viewing Log Files in EM

Message Pane


Troubleshooting SOA
Use of ECID
§  Globally unique identifier
§  Enables you to track a message flow that crosses
instances of different composites.
§  Usually generated at the time of receiving the incoming
message


Troubleshooting SOA
Editing Log Configuration


Troubleshooting SOA
Setting Logging Levels
Logging Levels

Active Loggers
Saved
Loggers

Logger for SOA
Adapters

Logger for BPEL


Troubleshooting SOA
Selective Tracing
Increase log levels for
chosen context,
Context: Composite Name, Client Address etc
loggers, and servers

Trace Level

Loggers

Under
WebLogic
Domain


Trouble Shooting SOA
BPEL Engine Statistics
§  Requests: pending and
active
Requests
§  Threads
§  Request break down show
BPEL activities

Threads

Request
Breakdown


Troubleshooting SOA
Debugging URLs
§  http://soa-server:port/soa-infra: Sanity check
§  http://soa-server:port/soa-infra/cluster/info: Cluster info
§  http://soa-server:port/wls-cat : Class loader analysis tool


Troubleshoot SOA
JDK Monitoring: Verbose GC
§  What it may show
–  JVM heap memory utilization
–  Memory leaks
–  Major and minor garbage collection frequency and time
–  JVM health in general

§  How to produce:
–  JVM command line argument

§  How to analyze:
–  GCViewer

Troubleshooting SOA
JDK Monitoring: Thread Dumps
§  What it shows:
–  Application or product deficiency
–  External resource contentions
–  JVM health

§  How to produce:
§  Console / WLST / Kill -3 <pid> / jstack <pid> / jrcmd <pid> print_threads

§  Taken repetitively, during the period when symptom occurs

§  How to analyze
–  ThreadLogic

Troubleshooting SOA
Operating System Monitoring
§  What to look for
–  High CPU system time
–  High CPU wait time
–  High CPU idle time along with poor throughput


Troubleshoot SOA
Operating System Monitoring

OS Tools
Linux top, netstat, iostat
Solaris prstat, vmstat, mpstat, netstat,
iostat
AIX, Linux nmon
Windows Task Manager, Resource
Monitor


Common Problems and Solutions
Resolving common startup problems with SOA server
§  Problem: ClassNotFound •  Problem: SOA Server (soa_infra)
exceptions when SOA does not start
server is started by a Node
•  Solutions:
Manager
•  Check port conflicts
§  Solution: Add property
StartScriptEnabled=true to •  Make sure MDS schema is
nodemanager.properties. reachable
•  Check coherence configuration
in clustered setup. Make sure
that you’re using command line
instead of active cache
configuration


Resolving OS level problems
§  Problem: ‘Too many open files’ during compilation or runtime.
§  Solution: /etc/security/limits.conf to increase the value of both file
descriptors to at least 4096.


Resolving transaction or connection timeouts
§  Problem: Transaction or connection timeouts
§  Solution:
–  Figure out why the transaction takes so long!
–  Increase transaction timeout for at Domain and soa-infra EJBs


Common application-related Problems
§  Problem: Changes to connection factories JNDI (Configuration -> Outbound
Connection Pool) for SOA adapters do not take effect . Click ‘Update’ after
selecting the adapter

Click ‘Save’ after the
connection factory properties
have been updated

Hit ‘Enter’ Key
after changing


Common adapter-related Problems
§  Problem: Mis-configured •  Problem: Duplicate message
Queues/Datasource in processing for non-transactional
JMSAdapter and DBAdapter. adapters (File/Ftp) in clustered
§  Solution: Make sure that the setup.
runtime resources such as •  Solution: Use ‘eis/HAFileAdapter’
JMS connection factories, or ‘eis/ftp/HAFtpAdapter’ in jca
JMS Queues, DB configuration. Alternatively
datasources are configured configure active-passive failover in
properly in the WLS console the composite.xml file by
configuring “standalone” property
with ”true”


EM Performance Issue

§  Issue: EM
dashboard loads
slowly / hangs on
login.

§  Step 1: Set default
time duration for
aggregate queries.
§  Step 2: Disable
auto-fetch of
aggregated metrics.


Q&A


APPENDIX


BPEL Service Engine (to be removed)
Threads and Persistence
Msg persisted (DB)

Sync Async
Proces
Receive Proce Receive De/ (1st) process hydration

s ss hydr
(2nd) callback msg
ation persisted (DB),
triggers dehydration
sync process … …
invoke executed
Receive
by caller thread,
e.g. servlet
After hydration (e.g
mid-process
…
Reply receive activity for
async callback)
async process
process executed
invoke executed
by BPEL engine
by BPEL engine
engine thread
invoker thread


BPEL Service Engine (to be removed)
Faults and Transactions
msg persisted (DB)
Async

Sync Proc De/ (1st) process hydration

Proc Receive
ess Receive hydr (2nd) callback msg
ess ation persisted (DB),
triggers dehydration

Receive
… …

exception or exception or system
system crash crash rolls message
exception or system crash rolls … back into DB,
returns exception message back into DB, available
(or timeout) to available to BPEL
to BPEL SE for manual (invoke)
client SE for manual
recovery.
(callback) recovery.

* – A 3rd type of recovery is Activity composite running but BPEL composite and BPEL
recovery. See next slide. instance faulted; new instance
instance running
initiated on recovery.


Soa config con8627_pdf_8627_0001

More Related Content

What's hot (13)

Similar to Soa config con8627_pdf_8627_0001 (20)

More from jucaab (20)

Soa config con8627_pdf_8627_0001