Design Like a Pro: How to Pick the Right System Architecture
Download as PPTX, PDF2 likes699 views
The document details a webinar on selecting the appropriate architecture for implementing Ignition, an industrial application platform for HMI, SCADA, MES, and IIoT. It covers different architecture options such as standard, scale-out, and cloud architectures, along with important considerations for performance and database sizing. Additionally, it discusses the importance of tags, value changes, polling rates, and deadbands in optimizing system performance.
![Tag Paths
Recommendation: Use fully-qualified tag paths in your projects, especially with visualization
templates and screens.
Example of a non-fully-qualified tag path:
Path/to/my/tag
Example of a fully-qualified tag path:
[TagProviderName]Path/to/my/tag
Scale-Out Architecture
← DON’T DO THIS! ❌
← DO THIS! ✔️](https://image.slidesharecdn.com/slidesharemay2021webinar-dlaparchitecture-210528160122/85/Design-Like-a-Pro-How-to-Pick-the-Right-System-Architecture-48-320.jpg)
Design Like a Pro: How to Pick the Right System Architecture
- 2. Presenters Kevin McClusky Co-Director of Sales Engineering Inductive Automation Don Pearson Chief Strategy Officer Inductive Automation
- 3. Today’s Agenda • Introduction: Architecture and Performance • How Do I Pick the Right Architecture? • Standard Architecture • SQL Database Sizing Tips • Scale-Out Architecture • Cloud Architectures • Security • Optimizations • Running Ignition on Virtual Machines • Q&A
- 4. Ignition: The Industrial Application Platform • Unlimited licensing model • Cross-platform compatibility • Based on IT-standard technologies • Scalable server-client architecture • Web-managed • Launch on desktop or mobile • Modular configurability • Rapid development and deployment One Universal Platform for HMI, SCADA, MES & IIoT: One Universal Platform for HMI/SCADA, MES & IIoT: • Unlimited licensing model • Cross-platform compatibility • Based on IT-standard technologies • Scalable server-client architecture • Web-based & web-managed • Web-deployed designer & clients • Modular configurability • Rapid development & deployment Ignition: Industrial Application Platform
- 5. Industry Introduction: Architecture and Performance Ignition is a development toolkit for building solutions of all sizes — as small as a data collector for a few tags or as large as an enterprise solution with hundreds of devices, hundreds of thousands of tags, and hundreds of visualization clients. It is extremely important to find the right architecture and choose the right sizes of servers so your project works as intended and has room to grow. Any architecture needs to be fully tested and verified, so you can observe the server’s performance characteristics and make adjustments to the architecture. There is no guarantee on performance since it is based on your design choices.
- 6. Ignition’s performance and sizing is based on several different factors: • Number of device connections • Types of PLCs (driver) • Number of tags • Frequency of tag polling (1 second, 5 seconds, 1 minute, etc.) • Number of tag value changes per second (% of change) • Number of concurrent visualization clients • SQL database throughput (historian, alarm journal, etc.) • Deployment (physical machine, VM, etc.) • Network & latency • And more ... Introduction: Architecture and Performance
- 7. The features you use in Ignition also have a big effect on performance because they require processing time and memory: Introduction: Architecture and Performance • Device communication • OPC-UA (client and server) • Tags (OPC, Expression, SQL query, etc.) • Historian (storage and retrieval) • Alarming (status, journal, notification pipelines) • Transaction groups (especially large numbers) • Scheduled PDF reports • Sequential Function Charts • Scripting ○ Tag change scripts ○ Scripts running on the server (timer, message handlers, etc.) • Visualization clients ○ Number of tag subscriptions on a screen ○ Polling queries (tag historian, alarming, custom queries, etc.) ○ Number of Gateway requests • Gateway Network (remote services, EAM) • MES functionality • MQTT or cloud injection • And more …
- 8. Ignition is heavily multithreaded and can handle configurations with all of those features at reasonable limits. Some features, such as device communication and tags, have predictable performance. Other features, such as visualization clients, have less predictable performance because of how the user interacts with the system and how the project is configured. Introduction: Architecture and Performance
- 9. In many cases, a single server is sufficient to handle everything. However, when projects get large or when we push the limits of Ignition, we need multiple servers. Ignition is modular and can scale out by separating different modules and features onto dedicated servers. Those separate servers work as one Ignition unit and are completely transparent to the user, thanks to Ignition’s Gateway Network, standards, and more. Introduction: Architecture and Performance
- 10. In this webinar, we’ll look at projects of different sizes to understand their hardware requirements and architecture. Since Ignition has a lot of different features, this webinar will focus on the number of devices, tags, and clients. Based on the Ignition Server Sizing and Architecture Guide Introduction: Architecture and Performance
- 11. Let’s look at the options ... ● Standard ● Scale-Out ● Hub-and-Spoke ● Cloud How Do I Pick the Right Architecture?
- 12. Which of those architectures is best for you? Ask yourself some questions … ● First, think about the physical connections. Are the connections unreliable? Is there geographic separation? Need resilience if part of the system goes down? Choose Hub-and-Spoke. How Do I Pick the Right Architecture?
- 14. ● Need to run in the cloud? Choose a Cloud architecture. How Do I Pick the Right Architecture?
- 16. Which of those architectures is best for you? Ask yourself some questions … ● Have a small system and a reliable network and can run everything from a central office or server room? Choose Standard. How Do I Pick the Right Architecture?
- 18. ● That same setup but the system is large? Choose Scale-Out. How Do I Pick the Right Architecture?
- 21. ● Maybe you need to do a variety of those functions. Can you mix and match architectures? Absolutely! How Do I Pick the Right Architecture?
- 22. Mix & Match Example: Scale-Out with Cloud
- 23. ● What about IIoT, Industry 4.0, and data collection architectures enabling digital transformation? These layer into all of the architectures mentioned. How Do I Pick the Right Architecture?
- 24. How Do I Pick the Right Architecture? MQTT is often used for: ● Resilient Data Collection ● Single Source of Truth ● Company Adoption of Open Standards ● Low Bandwidth Connections ● Modern Industrial and IT Protocol Support ● Industry 4.0 / IIoT / Digital Transformation initiatives This webinar is focused on the Ignition Gateways in the architectures, and every architecture presented can utilize MQTT for data collection and transfer. Find more information at https://inductiveautomation.com/solutions/iiot
- 26. ● The most common starting Ignition architecture ● Consists of a single on-premise Ignition server connected to a SQL database, PLCs, and clients. ● All functionality is configured on the same Ignition server ● Ignition has unlimited licensing but is limited by the size of the hardware. Larger hardware equals more device connections, tags, and clients. Standard Architecture (Single Server)
- 28. Small Ignition Project Perfect for Edge Panel, Edge IIoT, HMI, small SCADA, data collection, alarming. Standard Architecture (Single Server) ARM, 1GB memory, SSD 1-2 devices 500 tags 2 concurrent clients † 2 Cores (2GHz+), 2GB memory, SSD 1-10 devices 2,500 tags 5 concurrent clients † 2 Cores (3 GHz+), 4GB memory, SSD 1-10 devices 5,000 tags 10 concurrent clients †
- 29. Medium Ignition Project Medium full SCADA systems or MES systems. Standard Architecture (Single Server) 4 Cores (3 GHz+), 4GB memory, SSD 1-25 devices 10,000 tags 20 concurrent clients † 4 Cores (4 GHz+), 8GB memory, SSD 1-50 devices 25,000 tags (40% tag history change at max) 25 concurrent clients † 4 Cores (4 GHz+), 16GB memory, SSD 1-100 devices 50,000 tags (20% tag history change at max) 50 concurrent clients †
- 30. Large Ignition Project Large full SCADA systems or MES systems. Standard Architecture (Single Server) 8 Cores (4 GHz+), 8GB memory, SSD 1-100 devices 75,000 tags (10% tag history change at max) 50 concurrent clients † 8 Cores (4 GHz+), 16GB memory, SSD 1-100 devices 100,000 tags (10% tag history change at max) 75 concurrent clients † 16 Cores (4 GHz+), 32GB memory, SSD 1-100 devices 150,000 tags (6% tag history change at max) 100 concurrent clients †
- 31. A Note About Single-Server Architectures: † Results can vary based on design choices. The examples are based on typical usage. Faster polling rates, increased value changes, and utilization of other Ignition features, such as scripts, Transaction Groups, SFCs, and more, can significantly impact the overall performance. Standard Architecture (Single Server)
- 32. SQL Database
- 33. In all of the following scenarios, the SQL database should be on its own dedicated server. If you put the database on the same server as Ignition, you will need more processing power and memory. Here are some basic database sizing tips ... SQL Database
- 34. Small Historian 2 Cores (2 GHz+), 2GB memory, SSD 0-100 value changes per second Requires approximately 300GB disk space/year if 100% of the values are changing every second sustained (Requires approx. 6GB with 2% change, smaller with slower rates) 2 Cores (2 GHz+), 4GB memory, SSD 100-500 value changes per second Requires approximately 3TB disk space/year if 100% of the values are changing every second sustained (Requires approx. 60GB with 2% change, smaller with slower rates) SQL Database Sizing Tips
- 35. Medium Historian 4 Cores (4 GHz+), 8GB memory, SSD 500-2,500 value changes per second Requires approx. 7TB disk space/year if 100% of the values are changing every second sustained (Requires approx. 150GB with 2% change, smaller with slower rates) 4 Cores (4 GHz+), 16GB memory, SSD 2,500-5,000 value changes per second Requires approx. 15TB disk space/year if 100% of the values are changing every second sustained (Requires approx. 300GB with 2% change, smaller with slower rates) SQL Database Sizing Tips
- 36. Large Historian 8 Cores (4 GHz+), 16GB memory, SSD 5,000 - 10,000 value changes per second Requires approx. 30TB/year if 100% of the values are changing every second sustained (Requires approx. 60GB with 2% change, smaller with slower rates) SQL Database Sizing Tips
- 37. A Note About Databases: Some databases on a properly tuned server can max out at 10,000 value changes per second. For example, stay within 5,000 for MySQL. Microsoft SQL Server can handle up to 20,000-30,000 value changes per second with the latest version. Throughput and query speed is dependent on hardware and database setup. Speed of retrieving data in Ignition can also be impacted with higher storage throughput. For higher throughput, install another Ignition instance on the same server as the database and use Ignition’s remote tag history through the Gateway Network. SQL Database Sizing Tips
- 38. You can adjust the amount of memory allocated in Ignition’s ignition.conf configuration file. You need to set the proper amount of memory for each of the scenarios shown here. Typically, you can leave 1-2GB of memory to the OS and use the rest for Ignition. For more details, refer to “Gateway Configuration File Reference” in the Ignition Online User Manual at docs.inductiveautomation.com A Note About Ignition Memory
- 40. Scale-Out Architecture ● In larger systems, it is often easier to have multiple Ignition installations to help split the load between frontend and backend tasks. ● Perfect for single large systems that aren't split up into different sites. ● The Hub and Spoke Architecture is usually a better fit for systems split into different sites. Scale-Out Architecture
- 41. Scale-Out Architecture ● The Scale-Out architecture has at least one Gateway that handles backend communications. ● The backend Gateway deals with all PLC and device communications. ● The frontend Gateway handles all of the Clients, serving up the data pulled from the backend Gateway. ● This is made possible through the Gateway Network connecting Gateways to each other and allowing tag sharing through remote tag providers. Scale-Out Architecture
- 42. Gateway Network – Connects multiple Gateways over a WAN and opens up many distributed features between Gateways Gateway Network features include: ● Dedicated HTTP data channel ● Set up a node to act as a proxy for another node ● Security settings (restrict incoming connections based on a whitelist or on manual approval; or disable incoming connections entirely) ● Available SSL mode Scale-Out Architecture
- 43. More Gateway Network Features: ● Enterprise Administration ○ Enterprise Administration Module (EAM) uses the Gateway Network for message and file transfer ○ Can monitor network connections for availability ○ Reports whenever communications are lost via alarm events and system tags ● Distributed Services ○ Remote Real-time Tag Providers ○ Remote Historical Tag Providers ○ Remote Alarming ○ Remote Alarm Journal ○ Remote Audit Logs ● Security Zones ● Service Security Scale-Out Architecture
- 44. Scale-Out Architecture: One I/O / One Frontend
- 45. Scale-Out Architecture: One OPC-UA Server / One I/O / One Frontend
- 47. Tag Provider Best Practice ● Provide proper names to your tag providers ● Make the names consistent across the I/O and the frontend ● Avoid using the default provider that comes in a standard installation ● It is common to create a new realtime tag provider on the I/O server with a name that describes that I/O server and is distinguished from other I/O servers. On the frontend server, create a remote tag provider with the same name as the I/O server so the two are consistent. Scale-Out Architecture
- 48. Tag Paths Recommendation: Use fully-qualified tag paths in your projects, especially with visualization templates and screens. Example of a non-fully-qualified tag path: Path/to/my/tag Example of a fully-qualified tag path: [TagProviderName]Path/to/my/tag Scale-Out Architecture ← DON’T DO THIS! ❌ ← DO THIS! ✔️
- 52. Security
- 54. Find more best practices for Ignition security in our Security Hardening Guide. inductiveautomation.com/resources/article/ignition-security- hardening-guide Security
- 55. Optimizations
- 56. When it comes to tags, value changes are the most important metric to look at. ● Defined as a change in value that is more than the configured deadband. ● Deadbands can be absolute or percentage-based. Example: Analog value with a deadband of 0.1 A change from 7.89 to 7.88 would not be considered a change. A change from 7.89 to 7.9 would be considered a change. ● Usually, any change with discrete values is considered a change, since we are dealing with a state. ● Deadbands are configurable on a per-tag basis. Optimizations
- 57. ● Value changes have to be processed for alarms, historian, and more. ● That processing requires memory and compute time. The more tags changing per second, the more processing. ● An Ignition server at the high end can handle approx. 50,000 value changes per second processing through the tag system, alarming, historian, and clients. However, some SQL databases have a limit of 10,000 value changes per second on a dedicated instance. ● Recommendation: Try to reduce the number of value changes per second. ● Ignition can handle more devices, tags, and clients through optimization and reducing value changes. Optimizations
- 58. Here are some scenarios that show what a server can handle. (Each scenario represents a number of tags that max out performance on a server.) ● 10,000 tags at 1 second rate with 100% changing = 10,000 values/sec ● 50,000 tags at 5 second rate with 100% changing = 10,000 values/sec ● 100,000 tags at 1 second rate with 10% changing = 10,000 values/sec ● 500,000 tags at 5 second rate with 10% changing = 10,000 values/sec ● 500 devices with 100 tags each = 50,000 tags @ 5 second rate with 100% changing = 10,000 values / sec As you can see, more tags can be handled by optimizing poll rates & deadbands. Optimizations
- 59. Optimize your system by paying close attention to the number of value changes happening every second. Here are some things to consider: ● Polling Rates ● Deadbands ● Leased and Driven Tag Groups ● Event-driven ● Scripts ● Avoid Polling Queries and Use Caching ● Tag Historian Stale Data Detection ● Vision Client Tag Poll Rate ● Gateway Network Optimizations ● Remote Tag Provider Alarm Subscription ● Remote Tag Provider History Mode Optimizations
- 60. Polling rates ● Tag groups (or scan classes) dictate how often Ignition polls data from PLCs. ● Make sure you are using proper polling rates. Not everything has to be at a 1-second rate. Some values can be polled slower than others since they don’t change very often. ● Try to determine all of the possible polling rates you require. Optimizations
- 61. Deadbands ● Deadbands log fewer data points, particularly when value changes are too small to be significant. ● Especially useful in a process that inspects with a high frequency but is most interested in capturing value changes exceeding a defined magnitude or percentage. ● Deadbands can be absolute or percentage-based ● Two deadband modes: digital and analog. ● There are deadbands on both realtime and historical tags. ● Without proper deadbands, systems will log ‘noise’ on analog signals. It should never have more sensitivity for logging than a sensor’s rated precision. ● The better you tune your deadbands, the fewer value changes you will have in the system. You can avoid fluttering on analog values and costly historian storage. Optimizations
- 62. Leased and Driven Tag Groups ● Tag groups tell Ignition how often to poll the PLC, and dictate the execution of tags, which is especially important for large & high-performance systems. ● Recommended: Organize tag groups by polling strategies. It is common to use a fast tag group for status and control, and a slower tag group for history. ● Driven tag groups are a great option for history where faster logging is conditionally required. ● Leased tag groups are a great option when you only want to poll tags when it is needed on a screen. ● When using Ignition’s OPC-UA server and drivers with leased or driven tag groups, we recommend creating two device connections to the PLC if allowed: One connection for the direct tag groups and one connection for leased and driven tag groups. Optimizations
- 63. Event-Driven ● Extremely important: Only execute logic on change, especially with expressions and SQL queries, to avoid additional computation when values haven’t changed. ● Ignition 8.0+ introduced new execution modes on tags, including “Event-Driven” ● Event-driven only fires the expression or SQL query when a dependency changes (i.e., tag value event or alarm event) versus running at a specific rate all the time, thereby reducing a lot of unnecessary computations. ● Recommendations: ○ Use event-driven on expressions, derived tags, and SQL query tags ○ Set a tag’s history mode to “On Change” Optimizations
- 64. Scripts ● It is very important to understand where and when scripts are running. ● Avoid runScript expressions unless absolutely necessary. ● Avoid lots of timer scripts on the Gateway or in the client. ● Avoid lots of tag change scripts. ● Try to reduce the number of scripts on both the server and clients. ● Use expressions instead of scripts wherever possible. ● Use scripting where it makes sense, but be mindful of the processing power required. Optimizations
- 65. Avoid Polling Queries and Use Caching ● Setting up polling queries to the historian, alarm journal, audit logs, or custom SQL queries, in a client places additional load on the Ignition server, especially when lots of clients are open. ● Recommendation: Try to reduce the number of polling queries or just turn them off. ● Instead, put a refresh button on the UI to run the query on-demand (available in both Vision and Perspective) ● Named queries can opt-in to caching results on the Gateway. This uses more Gateway memory but could reduce queries running against the database. ● When polling is necessary or if you have a lot of clients open, use named queries with caching for better performance. Optimizations
- 66. Tag Historian Stale Data Detection ● If enabled, this feature tracks tag group executions to determine the difference between unchanging values and flat values due to the system not running. This feature is on by default. ● Although useful at times, having this feature enabled can lead to lots of rows in the sqlth_sce table in the database, inadvertently causing performance issues querying historical data. ● Recommendation: Unless absolutely necessary, turn this feature off and simply let the system track values as they change. ○ Instructions about how to turn off the feature are in The Ignition Server Sizing and Architecture Guide ● If there are clock drift or system issues, those should also be dealt with as and treated as separate issues to resolve. Optimizations
- 67. Vision Client Tag Poll Rate ● Vision clients and the Ignition designer communicate to the Ignition Gateway to get tag values. You only need to get the data you want to see on screen. In that case, Vision clients and the designer create a subscription on the tags that they need. ● However, the Gateway cannot notify the client/designer when a tag changes so it polls the Gateway. By default, the client poll rate setting is set to 250ms. The rate is quite fast but typically not a problem. However, a high number of Vision clients can cause additional load on the server. You can easily reduce this load by changing the setting to 1 second. The setting for this is in the designer under Project → Properties on the Project → General tab. Optimizations
- 68. Gateway Network Optimizations ● It’s easy to send lots of data unintentionally. Make sure you know what and how much data is going through the network. ● Check the diagnostic page in the Gateway status section to see incoming and outgoing traffic by the Gateway Network service. ● Recommendation: Optimize the communication to ensure Gateway Network health. Remote Tag Provider Alarm Subscription ● Either turn alarming off or set the mode for how you want to retrieve alarms. ● Recommendation: Use the “Subscribed” mode for optimal communication. ○ The state will be subscribed, and updates will be sent asynchronously. ○ This mode provides better performance but uses more memory. ○ Especially important with lots of Gateway Network connections & alarms. Optimizations
- 69. Remote Tag Provider History Mode ● Recommendation for most cases: Connect a frontend Gateway to the SQL database directly and use historical tag paths to query the data. ● Recommendation if using realtime tag paths to query history: Set the remote tag provider to use the “Database” mode for history access. Instead of asking the I/O server for historical data, you can short-circuit and query the database directly, requiring a direct connection to the SQL database. If no SQL connection is available, try to reduce the amount of history queries and avoid costly queries with large time ranges and data. Optimizations
- 70. Running Ignition on Virtual Machines
- 71. Software ● Ignition runs on any hypervisor that properly emulates hardware and an OS. ● When running on a VM, Ignition is most commonly used on VMWare, although Inductive Automation does not recommend VMWare above other hypervisors. ● As long as resource allocation is appropriate, Ignition has not seen hypervisor- specific issues in any recent virtualization software. Resource Allocation ● CPU and memory allocation should be determined by the engineers creating the Ignition projects. ● Small systems may be 4 vCPUs with 8GB of RAM. ● Very large systems could be 32 vCPUs or more with 64+GB RAM. ● When Ignition is running small systems with only a few clients and tags, normal resource allocation is often acceptable. ● When it is running critical applications or any kind of significant load, dedicated resources are required for the system to stay performant. Running Ignition on Virtual Machines
- 72. Dedicated Resources ● When first specing an Ignition Gateway’s needs, always mention the need for dedicated vCPUs and memory. If an Ignition Gateway starts small, it might not be initially needed, but as it grows dedicated resources will be required to avoid issues. If dedicated resources aren’t initially configured, there may later be pushback from IT departments. VMWare-Specific Settings ● In VMWare, this is most commonly configured by setting Latency Sensitivity to High. ● After configuring this, check to make sure the vCPU allocation is 100% dedicated to the Ignition VM. If not, the whole VM Host is over-provisioned, and this will need to be fixed. Running Ignition on Virtual Machines
- 73. VMWare References VMWare 5.5 Latency Sensitivity vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/latency-sensitive- perf-vsphere55-white-paper.pdf VMWare 6.7 Latency Sensitivity www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/v sphere-esxi-vcenter-server-67-performance-best-practices.pdf The Ignition Server Sizing and Architecture Guide inductiveautomation.com/resources/article/ignition-server-sizing-and-architecture-guide (Also available in Handouts on your GoToWebinar control panel) More Information