FME Server 2021.0 at Scale: Advanced Job Control
0 likes708 views
The document discusses the advancements in job control capabilities in FME Server 2021.0, focusing on managing server workloads, optimizing job allocation, and utilizing dynamic engines effectively. Key features include improved job information metrics, challenges with managing job loads, and strategies for maximizing resource efficiency, especially in large and growing deployments.
FME Server 2021.0 at Scale: Advanced Job Control
- 1. FME 2021.0 at Scale Advanced Job Control
- 2. Meet the Presenters Don Murray Rylan Maschak
- 3. Agenda ● Server Workloads Are Changing ● Understanding Server Load ● Job Information in Server 2021 ● Job Management Challenges ● Advanced Job Control in Action ● Q&A This webinar is being recorded and a copy will be emailed to you.
- 4. Server Workloads are Changing Server workloads are changing from running scheduled tasks to integrating data and applications across the enterprise
- 5. Server Workloads Are Changing Fewer workspaces are run on predictable schedules. ● FME Server Apps trigger workflows when a user runs the App. ● Automations workflows are often triggered by external systems. ● Streams mean a whole new set of applications for FME.
- 6. More Workflows are Time Sensitive More and more of the work done by FME Server can be considered real-time. ● FME Server Apps workspaces often return things to the user and so must be run immediately. ● Automations workflows that connect applications also must execute immediately. ● Streams provide a nice interface for IoT and other high velocity data sources.
- 7. Deployments are Growing ● Number of workspaces is exploding. Clients with 1,000 or 10’s of 1000’s is not unheard of. ● Size of the server deployments is increasing, more engines, larger datasets, and applications being connected together. ● Dynamic Engines bring new flexibility but also added complexity in terms of when they should be used. ● Kubernetes and other flexible scaling deployments require better understanding of the workload than ever before to maximize their effectiveness.
- 8. Common Questions ● How can I balance the load on my server? ● How can I ensure that small light jobs don’t get blocked? ● How can I use dynamic engines in the most cost effective way?
- 9. New FME Server 2021 capabilities to support changing deployments.
- 10. Understanding Server Load Maximizing Server ROI requires you be able to easily leverage Job Information. Using this you can ensure: a) Machines don’t get overloaded with simultaneous large jobs. b) Small, time sensitive jobs don’t get blocked. Useful Job Information: a) CPU Time b) % CPU c) Peak Memory Usage d) Elapsed Time
- 11. Why did we build this? ● We are seeing larger and larger deployments of FME Server. ● It was difficult to understand the characteristics of workspaces. The information was buried in the logfile. ● It was difficult to control which workspaces were assigned to which engines. ● It was impossible for organizations to assign jobs to queues based on run-time characteristics.
- 12. Job Information in FME Server 2021.0
- 13. Completed Jobs
- 14. Workspace View
- 15. Schedule View
- 16. Automation Information Same workspace in different spots in same automation!
- 20. Automation Information Metrics are stored on a per workspace run in each Automation. Same workspace can be different because: ● Different input parameters ● Different sizes data being processed
- 22. Challenges ● Prior to FME 2021.0, there was no job information. ● This made it difficult to manage job allocation on FME Server. ● Many effectively let jobs run randomly until there was a problem and then they would try to deal with these “problem” workflows. Let’s look at 3 common challenges...
- 23. Challenge #1 - Overloaded Machines A machine with multiple engines gets overloaded and slows down if large jobs all land on it.
- 24. Show Overloaded Machine Notice the Drastic Difference in Run-Times! 28 mins to > 4 hours. Memory use looks to be the issue! CPU from low of 46% to high of 313%
- 26. Challenge #2 - Jobs Get Blocked Notice Queued Jobs waiting!
- 27. Challenge #3 - Effective Use of Dynamic Engines Dynamic Engines offer you another way of managing FME Server compute resources. The key is that you pay for CPU TIME not run time. So...
- 28. Challenge #3: Dynamic Engines What are Dynamic Engines? ● Same capability as a standard FME Engine ● Users only pay for the CPU time used, not when the engines are idle ● Users can scale engines up and down as needed - there is no per engine cost Video: Introduction to Dynamic Engines in FME
- 29. Challenge #3: Dynamic Engine Pricing
- 30. Challenge #3 - Effective Use of Dynamic Engines Which workspace is the best to assign to Dynamic Engines? What Job seems like the best candidate for Dynamic Engines? Least CPU Intense Job
- 31. Challenge #3 - Effective Use of Dynamic Engines Which workspace is the best to assign to Dynamic Engines? What Job seems like the best candidate for Dynamic Engines? Most Time Sensitive Job
- 32. Q&A
- 34. Challenge #1 - Overloaded Machines Step 1: Create a queue for jobs that use a lot of CPU.
- 35. Challenge #1 - Overloaded Machines Step 2: Assign Jobs to a Queue.
- 36. Challenge #1 - Overloaded Machines Step 3: Assign Engine(s) to the Queue. { Assign Engine(s) to Queue(s)
- 38. Challenge #2 - Jobs get blocked Same Pattern 1) Create a queue for the class of jobs 2) Assign engine(s) to the queue 3) Assign jobs to the queue using Job Information
- 39. Challenge #2 - Jobs get blocked Step 1: Define the queue for light jobs
- 40. Challenge #2 - Jobs get blocked Step 2: Assign Engine(s) to the Queue. { Assign Engine(s) to Queue(s)
- 41. Challenge #2 - Jobs get blocked Step 3: Assign Jobs to a Queue.
- 43. Challenge #3 - Effective Use of Dynamic Engines Let’s assign all of the Fast jobs to Dynamic Engines. This too is very easy and we start the same way. 1) Create Dynamic engines 2) Create a queue for the class of jobs 3) Assign the engines to the Queues 4) Assign jobs to the queue using Job Information
- 44. Challenge #3 - Effective Use of Dynamic Engines Step 1: Create Dynamic Engines Dynamic engines are about cpu. Launch as many as your hardware will support! Job information is key for this too. If you only send small jobs to a dynamic engine you can launch more.
- 45. Challenge #3 - Effective Use of Dynamic Engines Step 2: Create a queue for the class of jobs
- 46. Challenge #3 - Effective Use of Dynamic Engines Step 3 -Assign the engines to the Queues Engine Properties. I am going to use the “Dynamic” property to assign engines to the queue.
- 47. Challenge #3 - Effective Use of Dynamic Engines Step 3 - Assign the engines to the Queues
- 48. Challenge #3 - Effective Use of Dynamic Engines Step 3 - Assign the engines to the Queues This is the weakest part of the user experience in FME 2021.0. Fixed in FME 2021.1
- 49. Challenge #3 - Effective Use of Dynamic Engines Step 3 - Assign the engines to the Queues
- 50. Challenge #3 - Effective Use of Dynamic Engines Step 3 - Assign the engines to the Queues
- 51. Challenge #3 - Effective Use of Dynamic Engines Step 4: Assign jobs to the queue using Job Information
- 53. Summary Advanced Job Control in FME Server gives you ability to: ● Optimize job allocation to resources to increase throughput. ● Identify which Jobs for Dynamic Engines to maximize value. ● Ensure that all tasks get a fair share of resources.
- 54. Come one, come all from May 4 - 14! You won't want to miss these presentations: “FME Train: A Trip through the Land of FME Deployments” “Zipster’s Magic Show: Pushing FME to the Limits” FME World Fair 2021 Get your free ticket: safe.com/world-fair
- 55. Thank you! Try FME for free at safe.com/trial Connect with us for more FME