The New Soft-IOC Based alarm handler at the SNS
0 likes475 views
1) Previously at the Spallation Neutron Source (SNS), different alarm handling approaches were used that did not integrate well. To address this, a soft-IOC based alarm handler was developed that runs in Linux processes to better integrate alarms. 2) The new alarm handler was built using scripts to generate EPICS databases, display screens, and configuration from XML files for standard Linux soft-IOCs. This allows alarm summaries and controls to be incorporated into display screens. 3) The soft-IOC based alarm handler is now used across multiple systems at SNS, handling over 10,000 process variables and 700 alarm summaries. Operators use the integrated alarm displays and controls daily.
![P. Gurd, G. Law 37831, USA
E. Williams, SLAC, Menlo Park, CA 94025, USA
A
re that alarms are defined in the
control system.
DEFINI W TO
way to
integrate everything into a single system.
THE SOLU C-BASE
ft IOCs to be built
onfiguration files.
de all the
alarms hierarchically,
larms included in
ation about
n; in others, to a web page.
d with audible signals that
ppen, such as by output to a
appearance and functionality of the existing
op new
stem added a few requirements:
phone dialers. This is important to achieve
er
ting components.
ms when the
se, the IOC heartbeat) has
e maskable.
It should be possible to see the alarm status of
y are masked.
S shared area contains
THE NEW SOFT-IOC-BASED ALARM HANDLER AT THE SPALLATION
NEUTRON SOURCE*
son, J. Munro, W.H. Strong, ORNL, Oak Ridge, TN
bstract
The standard EPICS alarm handler tool (ALH) does not
integrate well with other EPICS client applications. At
SNS, we wanted the ability to incorporate alarm
summaries and alarm controls such as masks and resets
into screens in the display manager as well as the ability
to call display screens from alarm screens. To achieve
these aims, we built a soft-IOC-based alarm handler that
runs in Linux soft IOCs. A set of scripts builds EPICS
databases, display manager screens, and startup scripts for
standard Linux soft IOCs from old EPICS Alarm Handler
(ALH) or extensible markup language (XML)
configuration files. With this new tool the summaries,
masks and latch status can be incorporated into other
EPICS client applications. In this paper we describe our
experience building and using the soft-IOC-based alarm
handler everywhe SNS identify alarms as they ha
NG THE PROBLEM: HO
INTEGRATE ALARMS
Previous to the development of the soft-IOC-based
alarm handler, the Spallation Neutron Source (SNS)
controls group had tried a number of different approaches
to alarm handling. The cryogenic systems, the IOC
(input-output controller) summary and the operator top-
level summary screens used a number of EPICS database
calculation records to summarize machine status which
was then displayed using the extensible display manager
(edm). Some systems such as the target system used the
standard EPICS alarm handler (ALH) [3]. The
calculation records were hard to maintain, operators
resisted the use of ALH, and there was no easy
TION: THE SOFT-IO D The IOC snapshot, which displays the status of system
processors, added this requirement:
Device heartbeats need to raise alarALARM HANDLER
Some of the calculation records used for alarm
summaries were already running in soft IOCs – Linux
processes sharing Linux servers with other such processes
[1]. It seemed reasonable to replace those soft IOCs using
scripts to build EPICS databases, displays, logging
sequences and start-up scripts from XML configuration
files [4]. The new XML files are much easier to maintain
than the EPICS databases they replaced. The scripts were
then extended to allow the alarm so
from ALH-style c
Requirements
The new alarm handler needed to provi
functionality provided by the systems it was to replace.
Some important features are given below.
It must be possible to group
with arbitrary numbers of a
arbitrary numbers of levels.
Each alarm should be maskable.
It should be possible to find more inform
any alarm. In some cases, this means getting to a
display scree
Alarms should latch so that momentary glitches can
be tracked.
Operators must be presente
voice synthesizer.
Alarms must be logged.
In addition, when changing the software behind existing
operator interfaces, it is very important to ensure that
operators are comfortable with the new system. Interfaces
with the
erator interfaces should be created from the
system.
The cryogenic sy
A delay before latching some alarms must be
configurable.
It must be possible to interface alarms to devices
such as
unmanned operation of the Central Helium Liquifi
(CHL.)
It must be possible to add logic to alarm elements.
It must be possible to use templates and substitutions
to configure systems with repea
process variable (in this ca
not updated in a specified time period.
Operators had a few wishes:
Each alarm group should b
Alarms only relevant in some machine modes could
be masked automatically.
devices even if the
Software Structure
for the U.S. Department of Energy.
___________________________________________
* Work supported by Oak Ridge National Laboratory
for UT-Battelle, LLC, under contract DE-AC05-00OR22725 A soft alarm area in the EPIC
Proceedings of ICALEPCS07, Knoxville, Tennessee, USA RPPB26
Operational Tools
665](https://image.slidesharecdn.com/rppb26-140627160957-phpapp01/85/The-New-Soft-IOC-Based-alarm-handler-at-the-SNS-1-320.jpg)
![Figure 4: The Annunciated Alarm Summary and the soft
IOC that provides its data and functionality is built
automatically using the Soft Alarm scripts.
Figure 5: Another automatically created screen, the Ring
Cooling Alarm Screen, shows individual alarm status bits.
EXPERIENCE WITH THE SOFT-IOC-
BASED ALARMS AT SNS
The alarms are in constant use, both in the Central
Control Room and in the CHL Control Room. Operators
use the top-level alarm screen, the automatically created
lower-level alarm screens and the masking facilities all
the time. Occasionally, they refer to the alarm logs to find
out the sequence of alarm events.
The main objection the operators have to the new soft-
IOC-based alarm system is the many screens they have to
drill down through to get to the actual alarm. This might
be improved somewhat by more rational alarm
configuration.
Most of the operators and control system engineers who
have configured alarms have used the traditional EPICS
ALH-style configuration files rather than the new XML-
style configuration files [2]. This might be because of
familiarity or because of a lack of clear documentation for
the new XML format. The soft IOCs created by operators
to summarize machine states have not yet been converted
to the new alarm handler, though operators have said it
would be a good idea.
The Linux server that runs the alarm soft IOCs and
several other soft IOCs – a Hewlett Packard DL360 Intel
32 bit machine – is only about 10% busy, and thus can
easily accommodate all the alarms the SNS needs in the
foreseeable future.
THE FUTURE OF THE SOFT-IOC-BASED
ALARM HANDLER
At the SNS, several systems need to be configured to
complete the alarm systems, notably the low level RF
systems and the power supply systems. The front end
alarm configuration needs improvement to reduce the
number of nuisance alarms before it is ready for full
deployment.
A number of improvements should be made to the
existing configuration files so that operators could click
on any alarm summary or alarm point and get to more
information about the system. Some of the capabilities
have not really been used; for example, the machine mode
could be used to reduce unwanted alarms in many more
areas.
It could be useful to incorporate alarm summary
process variables into more control system screens.
An attempt was made to create a configuration
interface and an operator status display interface using
CSS/Eclipse (Control System Studio) [5], [6]. CSS would
make it possible to enable operators to go directly to the
alarm point, as they have requested. Also, an interface
could be created to make alarm configuration self-
explanatory. This approach seems hopeful, but much
more work is needed to make it useful.
REFERENCES
[1] P. Gurd, G. Lawson, D.H. Thompson, E. Williams,
“
The Application of Linux ‘Soft’ IOCs for Status
Summaries at the Spallation Neutron Source,”
ICALEPCS 2005, Geneva, November 2005.
[2] J. Munro, R.E. Battle, E. Danilova, R.L. Sangrey, E.
Williams, “A Proposed Alarm Handling System
Management Plan for SNS with Application to Target
Control System,” ICALEPCS 2007, Knoxville,
October 2007.
[3] http://www.aps.anl.gov/epics/extensions/alh/index.php
The EPICS Alarm Handler.
[4] http://en.wikipedia.org/wiki/XML XML.
[5] http://css.desy.de/content/index_eng.html CSS.
[6] K. Kasemir, “Control System Studio Applications,”
ICALEPCS 2007, Knoxville, October 2007.
Proceedings of ICALEPCS07, Knoxville, Tennessee, USA RPPB26
Operational Tools
667](https://image.slidesharecdn.com/rppb26-140627160957-phpapp01/85/The-New-Soft-IOC-Based-alarm-handler-at-the-SNS-3-320.jpg)
The New Soft-IOC Based alarm handler at the SNS
- 1. P. Gurd, G. Law 37831, USA E. Williams, SLAC, Menlo Park, CA 94025, USA A re that alarms are defined in the control system. DEFINI W TO way to integrate everything into a single system. THE SOLU C-BASE ft IOCs to be built onfiguration files. de all the alarms hierarchically, larms included in ation about n; in others, to a web page. d with audible signals that ppen, such as by output to a appearance and functionality of the existing op new stem added a few requirements: phone dialers. This is important to achieve er ting components. ms when the se, the IOC heartbeat) has e maskable. It should be possible to see the alarm status of y are masked. S shared area contains THE NEW SOFT-IOC-BASED ALARM HANDLER AT THE SPALLATION NEUTRON SOURCE* son, J. Munro, W.H. Strong, ORNL, Oak Ridge, TN bstract The standard EPICS alarm handler tool (ALH) does not integrate well with other EPICS client applications. At SNS, we wanted the ability to incorporate alarm summaries and alarm controls such as masks and resets into screens in the display manager as well as the ability to call display screens from alarm screens. To achieve these aims, we built a soft-IOC-based alarm handler that runs in Linux soft IOCs. A set of scripts builds EPICS databases, display manager screens, and startup scripts for standard Linux soft IOCs from old EPICS Alarm Handler (ALH) or extensible markup language (XML) configuration files. With this new tool the summaries, masks and latch status can be incorporated into other EPICS client applications. In this paper we describe our experience building and using the soft-IOC-based alarm handler everywhe SNS identify alarms as they ha NG THE PROBLEM: HO INTEGRATE ALARMS Previous to the development of the soft-IOC-based alarm handler, the Spallation Neutron Source (SNS) controls group had tried a number of different approaches to alarm handling. The cryogenic systems, the IOC (input-output controller) summary and the operator top- level summary screens used a number of EPICS database calculation records to summarize machine status which was then displayed using the extensible display manager (edm). Some systems such as the target system used the standard EPICS alarm handler (ALH) [3]. The calculation records were hard to maintain, operators resisted the use of ALH, and there was no easy TION: THE SOFT-IO D The IOC snapshot, which displays the status of system processors, added this requirement: Device heartbeats need to raise alarALARM HANDLER Some of the calculation records used for alarm summaries were already running in soft IOCs – Linux processes sharing Linux servers with other such processes [1]. It seemed reasonable to replace those soft IOCs using scripts to build EPICS databases, displays, logging sequences and start-up scripts from XML configuration files [4]. The new XML files are much easier to maintain than the EPICS databases they replaced. The scripts were then extended to allow the alarm so from ALH-style c Requirements The new alarm handler needed to provi functionality provided by the systems it was to replace. Some important features are given below. It must be possible to group with arbitrary numbers of a arbitrary numbers of levels. Each alarm should be maskable. It should be possible to find more inform any alarm. In some cases, this means getting to a display scree Alarms should latch so that momentary glitches can be tracked. Operators must be presente voice synthesizer. Alarms must be logged. In addition, when changing the software behind existing operator interfaces, it is very important to ensure that operators are comfortable with the new system. Interfaces with the erator interfaces should be created from the system. The cryogenic sy A delay before latching some alarms must be configurable. It must be possible to interface alarms to devices such as unmanned operation of the Central Helium Liquifi (CHL.) It must be possible to add logic to alarm elements. It must be possible to use templates and substitutions to configure systems with repea process variable (in this ca not updated in a specified time period. Operators had a few wishes: Each alarm group should b Alarms only relevant in some machine modes could be masked automatically. devices even if the Software Structure for the U.S. Department of Energy. ___________________________________________ * Work supported by Oak Ridge National Laboratory for UT-Battelle, LLC, under contract DE-AC05-00OR22725 A soft alarm area in the EPIC Proceedings of ICALEPCS07, Knoxville, Tennessee, USA RPPB26 Operational Tools 665
- 2. Database templates Display screen templates t IOCs. of the standard EPICS compone ss variables for both arm points and for riables. a full-size and a miniature – all of the parameters needed so that the alarm masking survives a soft IOC reboot. LER AT SNS oft-IOC-based alarm handlers are operating in ten systems : Soft-IO ed A s maries H at Scripts for building the components of the individual alarm sof The soft IOCs are composed nts: EPICS Run-Time Databases o Summary process variables Mask proceo individual al summaries. o Time stamps. o Logging process va Sequences to create logging records and to send logs to the Oracle relational database. Display screen files – screen for the summary at each level. Soft IOC startup script. Autosave request files Archive request files. THE SOFT-IOC-IMPLEMENTATION OF BASED ALARM HAND System and Channel Counts S at the SNS, as shown in Table 1. Table 1 C-Bas larms at SNS System PV Sum AL Form Accelerator Cooling 599 43 Yes Conventional Facilities 270 72 Yes Cryogenics 540 56 No High Power RF 5611 52 No Integrated Controls 2509 266 No Personnel Protection 723 148 Yes Target 545 84 Yes Target Moderator 567 18 Yes Timing e454 7 Y s Vacuum 218 27 Yes Totals 12036 773 The process variable (PV) counts given in Table 1 are those being monitored by the soft IOCs. The soft IOC logic uses a large number of PVs for its logic; for ex nal database to make it po e of any of a subset of the alarms, a message is sent to a speech synthesizer in the control room. Screen Examples ample, the accelerator cooling system soft IOC uses 8677 PVs to monitor 599 points and serve 43 summaries. All alarms are logged to XML-like files that are sent periodically to the Oracle relatio ssible to search through the alarm histories and to create alarm statistics. Upon the occurrenc Figure 1: The Top Alarm Summary displays the status of all the alarms in the system. Since EPICS process variables provide the status, the information can be used in any EPICS client application. Figure 2: The IOC Status Screen was one of the previous status displays that the new alarm system had to replicate. Figure 3: The Cryogenic Alarm Screens existed prior to the new alarm system, and would continue to be required. RPPB26 Proceedings of ICALEPCS07, Knoxville, Tennessee, USA Operational Tools 666
- 3. Figure 4: The Annunciated Alarm Summary and the soft IOC that provides its data and functionality is built automatically using the Soft Alarm scripts. Figure 5: Another automatically created screen, the Ring Cooling Alarm Screen, shows individual alarm status bits. EXPERIENCE WITH THE SOFT-IOC- BASED ALARMS AT SNS The alarms are in constant use, both in the Central Control Room and in the CHL Control Room. Operators use the top-level alarm screen, the automatically created lower-level alarm screens and the masking facilities all the time. Occasionally, they refer to the alarm logs to find out the sequence of alarm events. The main objection the operators have to the new soft- IOC-based alarm system is the many screens they have to drill down through to get to the actual alarm. This might be improved somewhat by more rational alarm configuration. Most of the operators and control system engineers who have configured alarms have used the traditional EPICS ALH-style configuration files rather than the new XML- style configuration files [2]. This might be because of familiarity or because of a lack of clear documentation for the new XML format. The soft IOCs created by operators to summarize machine states have not yet been converted to the new alarm handler, though operators have said it would be a good idea. The Linux server that runs the alarm soft IOCs and several other soft IOCs – a Hewlett Packard DL360 Intel 32 bit machine – is only about 10% busy, and thus can easily accommodate all the alarms the SNS needs in the foreseeable future. THE FUTURE OF THE SOFT-IOC-BASED ALARM HANDLER At the SNS, several systems need to be configured to complete the alarm systems, notably the low level RF systems and the power supply systems. The front end alarm configuration needs improvement to reduce the number of nuisance alarms before it is ready for full deployment. A number of improvements should be made to the existing configuration files so that operators could click on any alarm summary or alarm point and get to more information about the system. Some of the capabilities have not really been used; for example, the machine mode could be used to reduce unwanted alarms in many more areas. It could be useful to incorporate alarm summary process variables into more control system screens. An attempt was made to create a configuration interface and an operator status display interface using CSS/Eclipse (Control System Studio) [5], [6]. CSS would make it possible to enable operators to go directly to the alarm point, as they have requested. Also, an interface could be created to make alarm configuration self- explanatory. This approach seems hopeful, but much more work is needed to make it useful. REFERENCES [1] P. Gurd, G. Lawson, D.H. Thompson, E. Williams, “ The Application of Linux ‘Soft’ IOCs for Status Summaries at the Spallation Neutron Source,” ICALEPCS 2005, Geneva, November 2005. [2] J. Munro, R.E. Battle, E. Danilova, R.L. Sangrey, E. Williams, “A Proposed Alarm Handling System Management Plan for SNS with Application to Target Control System,” ICALEPCS 2007, Knoxville, October 2007. [3] http://www.aps.anl.gov/epics/extensions/alh/index.php The EPICS Alarm Handler. [4] http://en.wikipedia.org/wiki/XML XML. [5] http://css.desy.de/content/index_eng.html CSS. [6] K. Kasemir, “Control System Studio Applications,” ICALEPCS 2007, Knoxville, October 2007. Proceedings of ICALEPCS07, Knoxville, Tennessee, USA RPPB26 Operational Tools 667