Parallel Batch Performance Considerations
1 like1,087 views
The document discusses performance considerations for parallel batch processing in computing, highlighting the need to split batch jobs into parallel streams to improve efficiency as traditional speed increases stall. It covers techniques for implementation, such as analyzing and modifying programs for parallel execution, as well as monitoring performance metrics. Case studies are provided to illustrate the successful application of these techniques, emphasizing the importance of data partitioning and resource management.
Parallel Batch Performance Considerations
- 1. © 2009 IBM Corporation Session IK: Parallel Batch Performance Considerations Martin Packer, IBM martin_packer@uk.ibm.com
- 2. Abstract With the laws of physics providing a nice brick wall that chip builders are heading towards for processor clock speed, we are heading into the territory where simply buying a new machine won't necessarily make your batch go faster. So if you can't go short, go wide! This session looks at some of the performance issues and techniques of splitting your batch jobs into parallel streams to do more at once.
- 3. Motivations
- 4. Increased Window Challenge ● Workloads growing in an accelerated fashion: ● Business success ● Mergers and acquisitions – Standardisation of processes and applications ● More processing – Regulation – Analytics – “Just because” ● Shortened Window ● Challenge will outstrip “single actor” speed-up ● For SOME installations and applications – CPU – Disk – Tape – Etc ● Important to assess where on the (possibly) bell-shaped curve you are
- 6. This is still 10 – 15 Years away Maybe more Maybe never
- 7. © 2012 IBM Corporation IBM System z zEC12TLLB7zEC12TLLB7 zEC12 – Overall Attributes Highlights (compared to z196) • 50% more cores in a CP chip – Up to 5.7% faster core running frequency – Up to 25% capacity improvement over z196 uni-processor • Bigger caches and shorter latency – Total L2 per core is 33% bigger – Total on-chip shared L3 is 100% bigger – Unique private L2 designed to reduce L1 miss latency by up to 45% • 3rd Generation High Frequency, 2nd Generation Out of Order Design – Numerous pipeline improvements based on z10 and z196 designs – # of instructions in flight is increased by 25% • New 2nd level Branch Prediction Table for enterprise scale program footprint – 3.5x more branches • Dedicated Co-processor per core with improved performance and additional capability – New hardware support for Unicode UTF8<>UTF16 bulk conversions • Multiple innovative architectural extensions for software exploitation
- 8. Other Motivations ● Take advantage of capacity overnight ● Proactively move work out of the peak ● Resilience ● Consider what happens when some part of the batch fails ● Note: This presentation doesn't deal with the case of concurrent batch ● Some challenges are different ● Some are the same – You just have 24 hours to get stuff done, not e.g. 8
- 9. Issues
- 10. Issues ● Main issue is to break “main loop” down into e.g. 8 copies acting on a subset of the data ● Assuming the program in question has this pattern * ● Not the only issue but necessary to drive cloning ● With this pattern dividing the Master File is the key thing … * This is a simplification of the more general “the whole stream needs cloning” case
- 12. More Issues ● Reworking the “results” ● Probably some kind of merge process ● Handling inter-clone issues ● Locking ● I/O Bottlenecks ● Provisioning resource ● Concurrent use of memory and CPU greatly increased ● Scheduling and choreography ● Streams in lockstep or not ● Recovery boundaries ● Automation of cloning in the schedule
- 13. A Note On Reworking ● Consider the “merge at the end” portion: ● Probably valuable to separate data merge from “presentation” – “Presentation” here means e.g. reports, persistent output ● Consider an “architected” intermediate file – XML or JSON or whatever ● Use the architected intermediate file for other purposes – e.g PDF format reporting – Alongside original purpose
- 14. Implementation
- 15. Implementation ● Implementation consists of three obvious steps: ● Analysis ● Make changes – or Implementation :-) ● Monitoring (and loop back around)
- 16. Analysis ● Look for e.g. ● CPU-intensive steps ● Database-I/O intensive steps ● Prefer other tune-ups ● Be clear whether other tune-ups get you there – Some may effectively do cloning for you ● Take a forward-looking view – Lead time – Keep list of potential jobs to clone later on ● Assess whether “code surgery” will be required
- 17. Making Changes ● Splitting the transaction file ● Changing the program to expect a subset of the data ● Merging the results ● Refactoring JCL ● Changing the Schedule ● Reducing data contention
- 18. Monitoring ● Monitoring isn't terribly different from any other batch monitoring. ● Usual tools, including: ● Scheduler-based monitoring tools - for how the clones are progressing against the planned schedule. ● SMF - for timings, etc. ● Logs ● Need to demonstrate still functions correctly ● Work on “finding the sweet spot”: ● e.g. Is 2 the best, 4 or 8?* ● Work on “balance” – * Note bias to “power of two”
- 19. Case Study
- 20. Case Study ● Our application ● Not meant to be identical to yours ● Scales nicely through iterations ● Process important ● Stepwise progress ● Use e.g. DB2 Accounting Trace to guide ● In the following: ● 0-Up is the original unmodified program ● 1-Up is prepared for 2-Up etc and has – Reporting removed & replaced by writing a report data file ● Report writing and “fan out” and “fan in” minimal elapsed / CPU ● 2 Key metrics: ● Total CPU cost ● Longest Leg Elapsed Time
- 21. Don't Commit: Won't Go Beyond 1-Up 45% DB2 CPU, 45% Non-DB2 CPU 0 -Up 1 -Up 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 Total CPU Max Elapsed Minutes
- 22. Commit Every Update: Scales Nicely Up To 8-Up 50% DB2 CPU, 50% Non-DB2 CPU 0 -Up 1 -Up 2 -Up 4 -Up 8 -Up 16-Up 32-Up 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 Total CPU Max Elapsed Minutes
- 23. 8 Balanced Partitions: 50% Elapsed / CPU Reduction Up To 32-Up Up to 16-Up almost all time is Non-DB2 CPU, 32 is 50% “Queue” 16 and 32 partitions made no difference 0 -Up 1 -Up 2 -Up 4 -Up 8 -Up 16 -Up 32 -Up 0.00 5.00 10.00 15.00 20.00 25.00 30.00 35.00 Total CPU Max Elapsed Minutes
- 24. Case Study Lessons ● Applications mustn't break when you try to clone ● “Sweet spot” in our case is around 8-up ● Might still drive further if CPU increase acceptable ● Elapsed time got better at 16-up ● Data Management work can help ● Partitioning very nice in our case ● Environmental conditions matter ● In our case CPU contention limited scalability ● DB2 Accounting Trace guided us: ● Explained why preloading data into DB2 buffer pools did nothing