CSS Alarm Handling

Managed by UT-Battelle
for the Department of Energy
Kay Kasemir, Ph.D.
ORNL/SNS
kasemirk@ornl.gov
June 2011 at KEK
Control System Studio
- CSS -
Alarm Handling

2 Managed by UT-Battelle
Previous Attempts at SNS
ALH; manual “summary” displays; generated soft-IOCs +
displays
Issues
–  GUI
•  Static Layouts
•  N clicks to see active alarms
–  Configuration
•  .. was bad  Always too many alarms
•  Changes required contacting one of the 2 experts, wait
~days, restart CA gateway, hope that nothing else broke
–  Information
•  Operator guidance?
•  Related displays?
•  Most frequent alarm?
•  Timeline of alarm?

Now: Best Ever Alarm System Tool
Yes, alarms are always a little scary…

Alarm System Components
Control SystemAlarm Server
Cool UI
Configuration
B. Hollifield, E. Habibi, "Alarm Management: Seven Effective
Methods for Optimum Performance", ISA, 2007
1.  What you see
2.  Technical details
3.  How to use it

1. What you see
Alarm GUI used by Operators

What you see: Alarm Table
•  All current
alarms
–  new, ack’ed
•  Sort by PV,
Descr., Time,
Severity, …
•  Optional:
Annunciate
•  Acknowledge one or multiple alarms
–  Select by PV or description
–  BNL/RHIC type un-ack’

Another View: Alarm Tree
•  All alarms
–  Disabled, inactive, new, ack’ed
•  Hierarchical
–  Optionally only show
active alarms
–  Ack’/Un-ack’ PVs or sub-tree

Guidance, Related Displays, Commands
  Basic Text
  Open EDM/OPI screen
  Open web page
  Run ext. command
Hierarchical:
Including info of parent
entries
Merges Guidance etc. from
all selected alarms

Integrated with other CSS Tools
 Alarms
 History of PV
 EPICS Config.

CSS Context Menus Connect the Tools
Send alarm
PV to any
other CSS
PV tool

E-Log Entries
•  “Logbook”
from context menu
creates text w/
basic info about
selected alarms.
Edit, submit.
•  Pluggable implementation, not limited to
Oracle-based SNS ELog

.. may require Authentication/Authorization
 Log in/out while CSS is running
Online Configuration Changes

Add PV or Subsystem
1.  Right-click on ‘parent’
2.  “Add …”
3.  Enter name
Online. No search for config files, no restarts.

Configure PV
•  Again online
•  Especially useful
for operators to
update guidance
and related
screens.

2. Technical details
Behind the GUI;
Tools to monitor performance

Technical View
Alarm Cfg & State
RDB
IOCs
Alarm Server
Current Alarms: Acknowledged? Transient? Annunciated?
LOG
Message
RDB
JMS
2
Speech
JMS
2
RDB
Tomcat
- Reports
CSS Applications
Alarm Client GUI
JMS
Alarm Updates Ack’; Config UpdatesAnnunciationsLog Messages
TALK ALARM_CLIENTALARM_SERVER
PV Updates (Channel Access, …)

General Alarm Server Behavior
•  Latch highest severity, or non-latching
–  like ALH “ack. transient”
•  Annunciate
•  Chatter filter ala ALH
•  Alarm only if severity persists some minimum time
•  .. or alarm happens >=N times within period
•  Optional formula-based alarm enablement:
–  Enable if “(pv_x > 5 && pv_y < 7) || pv_z==1”
–  … but we prefer to move that logic into IOC
•  When acknowledging MAJOR alarm, subsequent MINOR
alarms not annunciated
–  ALH would again blink/require ack’

Logging
•  ..into generic CSS log also used for error/warn/
info/debug messages
•  Alarm Server: State transitions, Annunciations
•  Alarm GUI: Ack/Un-Ack requests, Config changes
•  Generic Message History Viewer
–  Example w/ Filter on TEXT=CONFIG

Logging: Get timeline
•  Example: Filter on TYPE, PV
1. PV triggers,
clears,
triggers again
2. Alarm Server
latches alarm
4. Problem fixed
3. Alarm Server
annunciates
5. Ack’ed by operator
6. All OK

All Sorts of Web Reports

3. How to use it
This may be more important
than the tools!

Best Ever Alarm System Tools, Indeed
.. but Tools are only half the issue
Good configuration requires plan & follow-up.
B. Hollifield, E. Habibi,
"Alarm Management:
Seven (??) Effective Methods
for Optimum Performance", ISA, 2007

Alarm Philosophy
Goal:
Help operators take correct actions
–  Alarms with guidance, related displays
–  Manageable alarm rate (<150/day)
–  Operators will respond to every alarm
(corollary to manageable rate)

•  DOES IT REQUIRE IMMEDIATE OPERATOR
ACTION?
–  What action? Alarm guidance!
•  Not “make elog entry”, “tell next shift”, …
•  Consider consequence of no action
•  Is it the best alarm?
–  Would other subsystems, with better PVs, alarm at the
same time?
What’s a valid alarm?

How are alarms added?
•  Alarm triggers: PVs on IOCs
–  But more than just setting HIGH, HIHI, HSV, HHSV
–  HYST is good idea
–  Dynamic limits, enable based on machine state,...
Requires thought, communication, documentation
•  Added to alarm server with
–  Guidance: How to respond
–  Related screen: Reason for alarm (limits, …), link
to screens mentioned in guidance
–  Link to rationalization info (wiki)

Impact/Consequence Grid
Category So What Minor Consequence Major Consequence
Personnel Safety PPS independent
from EPICS?
Environment,
Public
Can EPICS cause
contained spill of
mercury?
Uncontained spill??
Cost:
Beam
Production,
Downtime,
Beam Quality
No effect
Beam off
< 1 sec?
Beam off <10 min
<$10000
Beam off >10min
>$10000
•  Mostly: How long will beam be off?

.. combined with Response Time
Time to Respond Minor Consequence Major Consequence
>30 Minutes NO_ALARM MINOR
10..30 minutes MINOR MAJOR
<10 minutes MAJOR MAJOR + Annunciate
–  This part is still evolving…

Example: Elevated Temp/Press/Res.Err./…
•  Immediate action required?
–  Do something to prevent interlock trip
•  Impact, Consequence?
–  Beam off: Reset & OK, 5 minutes?
–  Cryo cold box trip: Off for a day?
•  Time to respond?
–  10 minutes to prevent interlock?
  
•  MINOR? MAJOR?
•  Guidance: “Open Valve 47 a bit, …”
•  Related Displays: Screen that shows Temp, Valve, …

“Safety System” Alarms
•  Protection Systems not per se high priority
–  Action is required, but we’re safe for now, it won’t
get worse if we wait
•  Pick One
 “Mommy, I need to gooo!”
 “Mommy, I went”
(Does it require operator action?
How much time is there?)

Avoid Multiple Alarm Levels
•  Analog PVs for Temp/Press/Res.Err./…:
–  Easy to set LOLO, LOW, HIGH, HIHI
•  Consider:
•  In most cases, HIGH & HIHI only double the
alarm traffic
–  Set only HSV to generate single, early alarm
–  Adding HHSV alarm assuming that the first one is
ignored only worsens the problem

Bad Example: Old SNS ‘MEBT’ Alarms
•  Each amplifier trip:
≥ 3 ~identical
alarms, no guidance
•  Rethought w/ subsystem
engineer, IOC programmer
and operators: 1 better alarm

Alarms for Redundant Pumps

Alarm Generation: Redundant Pumps
the wrong way
•  Control System
–  Pump1 on/off status
•  Simple Config setting: Pump Off => Alarm:
–  It’s normal for the ‘backup’ to be off
–  Both running is usually bad as well
•  Except during tests or switchover
–  During maintenance, both can be off

Redundant Pumps
•  Control System
–  Number of running pumps
–  Configurable number of desired pumps
•  Alarm System: Running == Desired?
–  … with delay to handle tests, switchover
•  Same applies to devices that are only needed
on-demand
1Required Pumps:

Weekly Review: How Many? Top 10?

A lot of information available
•  How often did PV trigger?
•  For how long?
•  When?
•  Temporary issue?
Or need HYST,
alarm delay,
fix to hardware?

Weekly Check: Stale, Forgotten?

Summary
•  BEAST operational since Feb’09
–  Needs a logo
–  For now without BEAUtY
–  DESY AMS is similar and has been
operational for longer
•  Pick either, but good configuration requires work in
any case
–  Started with previous “annunciated” alarms
•  ~300, no guidance, no related displays
•  Now ~330, all with guidance, rel. displays
–  “Philosophy” helps decide what gets added and how
•  Immediate Operator Action? Consequence?
Response Time?
–  Weekly review spots troubles and tries to improve
configuration

CSS Alarm Handling

More Related Content

Viewers also liked (11)

Similar to CSS Alarm Handling (20)

More from Robert-Emmanuel Mayssat (20)

Recently uploaded (20)

CSS Alarm Handling