Going Deep with MQ
Download as PPTX, PDF0 likes207 views
The document discusses strategies for handling large message backlogs, or "deep queues", that can accumulate in IBM MQ when a getting application experiences an extended outage. It notes that MQ is designed to handle temporary outages but not be used as a long-term database. The maximum queue depths possible on different platforms are estimated, with private queues on z/OS allowing up to around 16.8 million messages and shared queues potentially storing over 600 million messages if configured optimally. Indexing, sizing, and offload strategies are discussed to maximize queue capacity during getting application failures.
Going Deep with MQ
- 1. Going large deep with MQ Matt Leming: Architect, IBM MQ for z/OS lemingma@uk.ibm.com © 2020 IBM Corporation 1
- 2. What does going deep mean? And when should you, and shouldn’t you do it?
- 3. Application Application A good queue is an empty queue MQ is designed to allow applications to asynchronously communicate, acting as a buffer to smooth workload peaks and temporary application outages
- 4. Application Application A good queue is an empty queue, but application failures happen! MQ is designed to allow applications to asynchronously communicate, acting as a buffer to smooth workload peaks and temporary application outages Failure of a putting application has no real effect on MQ Failure of a getting application will result in messages building up on its queues until the application restarts and beings processing again
- 5. Application A good queue is an empty queue, but application failures happen! MQ is designed to allow applications to asynchronously communicate, acting as a buffer to smooth workload peaks and temporary application outages Failure of a putting application has no real effect on MQ Failure of a getting application will result in messages building up on its queues until the application restarts and beings processing again If the getting application suffers an extended outage queues can completely fill up, resulting in the putting application having to have a strategy for how to deal with a full queue Application
- 6. Application Application The aim of this presentation This presentation aims to explore how deep a queue can go, and things you should bear in mind when that happens
- 7. Application Are deep queues a valid use case for MQ? MQ is designed to allow applications to asynchronously communicate, acting as a buffer to smooth workload peaks and temporary application outages These outages might be minutes, hours or even days, but they are always temporary We are not advocating using MQ as a database. I.e. keeping data on queues for “ever”. MQ is not optimised for this use case Use a database instead! Application
- 8. Application Application Are deep queues the only answer? No, but they are often the simplest from an application perspective. But as we will see, really deep queues require some thinking about Some alternatives include: • Putting application detecting MQRC_Q_FULL and pausing putting more messages • Putting application detecting MQRC_Q_FULL and putting new messages to a database, file, another queue • Starting up a temporary getting application to move the messages off somewhere else • Failing! In addition to having to code these solutions, they often come with challenges such as keeping message ordering etc
- 9. Life-cycle of a getting application outage It is up to you to decide the maximum getting application outage duration that you want to tolerate Some customers are now looking at multiple day outages with 10s of millions of messages a day building up However that decision needs to be in the bounds of possibility for MQ. There is a limit to the amount of data a queue manager can store Time Depth Getting application outage Getting application restarts Getting application processing backlog Normal running Normal running Peak queue depth Duration of recovery from outage Duration of getting application outage
- 10. Other important things • When you decide what the queue size limits are, make sure you enforce them via configuration: MAXDEPTH, MAXMSGL • Make sure you monitor for a getting application outage, perhaps via DIS QSTATUS, or by service interval events, so you can restart the getting application as soon as possible • Monitor for the queue filling up, e.g. queue depth full, or high events, for similar reasons • As the queue fills up you might start to see degraded putting application performance, it’s worth understanding what this might be so you can plan for it Time Depth Getting application outage Getting application restarts Getting application processing backlog Normal running Normal running Peak queue depth Duration of recovery from outage Duration of getting application outage
- 11. Other important things • When the getting application starts up it is going to need to catch up. How fast can it do this? E.g. with a 2 day outage and a getting application that can get twice as fast as data being put, it will take a total of 4 days from the start of the application outage to get back to normal • Do you need to allow for a getting application outage occurring during this recovery window? • Can the messages expire? And are you relying on that to keep the queue depth low? • Be particularly aware of potential getting application inefficiencies. For example getting using a message selector is relatively slow, this will be particularly noticeable on a deep queue • Make sure you test the deep queue scenario, including recovery, to make sure everything works in the timeframes you expect! Time Depth Getting application outage Getting application restarts Getting application processing backlog Normal running Normal running Duration of recovery from outage Duration of getting application outage Peak queue depth
- 12. So, how deep can I go? Approximate number of 1 KB messages MQ on z/OS private queue 16.8 million MQ on z/OS shared queue all in CF 613.6 million* ** MQ on z/OS shared queue on SMDS 1.4 billion* ** MQ on distributed 68.4 billion* We will see where these numbers come from later * We haven’t tested to the end of these limits **Going this deep will mean you can’t recover in the case of a structure failure. So non- persistent messages only!
- 13. Going deep with MQ on z/OS Private queues
- 14. Private queues – buffer pools and page sets Messages on private queues are stored in memory in buffer pools and might be moved into a page set on disk Data in buffer pools and page sets is accessed in 4KB pages. A single page might contain the queue spine, message meta data, message data for at most one message, or space usage (space map page) A private queue is associated with a single buffer pool / page set pair so the size of a page set is the constraining factor for private queues. A page set can be a maximum of 64 GB in size, which allows for ~16.8 million 1KB messages. This is a rough calculation, ignoring space maps, spine pages, etc To get a rough idea for the maximum number of messages of a given size that can be stored in a page set, round up the size of the message to a multiple of 4 KB and divide 64 GB by that number Buffer pool Page set
- 15. Private queues – recommendations If you want to allow a private queue to deal with an extensive getting application outage consider putting them in their own page set and ideally their own buffer pool. This will allow for accurate sizing. You don’t want two very deep queues on the same page set at the same time There is obviously a limit to how much separation you can have here given that there can be at most 100 page sets and buffer pools in a single z/OS queue manager Buffer pool Page set
- 16. Indexing your queues On z/OS you can specify the INDXTYPE attribute on local queues to tell the queue manager how to index the queue: no index; by message ID; by correlation ID; by group ID An index makes no difference if just getting the next available message off a queue, but makes a significant difference otherwise, especially if the queue is deep INDXTYPE only supports a single value, so choose it based off the most common approach for getting messages from the queue. However for private queues you can still use any approach regardless of the index, but it will be less efficient. The queue manager will tell you if you should consider indexing your queues MessageID=A MessageID=B MessageID=C
- 17. Indexing your queues The index for private queues is maintained in the queue manager in 64 bit storage Each message on an indexed queue uses 136 bytes to maintain the index. 10 million messages therefore uses 1360MB of 64 bit storage So if you are going to have deep private queues which are indexed make sure you account for it in the MEMLIMIT attribute of your *MSTR JCL MessageID=A MessageID=B MessageID=C
- 18. Going deep with MQ on z/OS Shared queues
- 19. Shared queues – storage Shared queues are stored in a coupling facility (CF) Messages may be entirely stored in the CF if the message size < 63KB, or a pointer to the message can be stored in the CF and the remainder of the message offloaded to Db2 (not recommended, and not discussed further) or in shared message data sets (SMDS) The maximum supported CF structure size is 1TB. This is all real storage, so is relatively expensive If SMDS is used each queue manager gets its own SMDS data set for the structure. The maximum size of a single SMDS is 16 TB pMessage pMessage pMessage
- 20. Shared queues – small messages Shared queues perform best when the message is held entirely in the CF as that minimises both code path length and removes the need to interact with DASD For messages which are < 63KB in size the best approach is therefore to keep them in the CF and only offload them to SMDS when the queue starts getting deep during a getting application outage This gives the best of both worlds, fast message access normally, but the ability to store lots of messages in the worst case If you do decide to keep small messages in the CF and not to offload, the maximum number of 1KB messages that can be stored in a single structure is approximately 613.6 million, based on the max CF size of 1TB pMessage pMessage pMessage
- 21. Shared queues – offload rules Each MQ CFSTRUCT definition has three offload rules associated with it Each rule specifies the maximum size message that can be stored in in the structure, when the structure is over a given percentage full The default rules assume that no messages < 63KB get offloaded until the structure is very full These defaults are not likely to be good for a getting application outage where you want to maximize the number of messages you can store and minimise the amount of CF used. Instead you might want to start offloading all messages when the structure is say 10% full, as shown in the example on the right hand size DEFINE CFSTRUCT(SHALLOWSTRUCT) CFLEVEL(5) … OFFLOAD(SMDS) OFFLD1TH(70) OFFLD1SZ(32K) OFFLD2TH(80) OFFLD1SZ(4K) OFFLD2TH(90) OFFLD1SZ(0K) DEFINE CFSTRUCT(DEEPSTRUCT) CFLEVEL(5) … OFFLOAD(SMDS) OFFLD1TH(10) OFFLD1SZ(0K) OFFLD2TH(10) OFFLD1SZ(0K) OFFLD2TH(10) OFFLD1SZ(0K)
- 22. Shared queues – offloaded messages Each offloaded message requires a message pointer in the CF. The CF is used to maintain queueing semantics, the pointer allows the messages to be located in SMDS Offloaded messages require you to consider both the CF structure size and the space used in SMDS At most a structure can contain 1.4 billion message pointers, you then need to consider how much space those messages will occupy on SMDS A single SMDS data set can be up to 16TB in size. This can easily take 1.4 billion 1KB messages, but only 16 million 1MB messages pMessage pMessage pMessage a message pointer 1 entry = 256 bytes 2 elements = 2 * 256 bytes Total size = 768 bytes pMessage
- 23. Shared queues – back ups CF structures are in memory structures. In the rare cases where they fail, they need to be rebuilt With MQ this is done by periodically taking back ups of the structure. If a structure failure occurs the structure can be recovered from the back up plus the logs of the queue managers that have accessed the structure since the failure Deep queues have important implications for this process: • Size and number of active and archive logs • Backup time • Backup frequency • Recovery time BACKUP CFSTRUCT(DEEPSTRUCT) Active and archive logs RECOVER CFSTRUCT(DEEPSTRUCT)
- 24. Shared queues – back ups – archive and active logs Backing up a structure involves writing the structure contents and the contents of the SMDS to a queue manager’s active logs. Over time the active logs then get written to the archive logs Therefore, as shown above, the back up might be only in the active logs, in a mixture of active and archive logs, or just archive logs As shown above the contents of the active logs normally mainly overlap the contents of the archive logs Therefore the limiting factor for a back up is the number and size of the archive logs. The biggest backup you can have is ~ 4096 GB. NB this is smaller than the maximum size of an SMDS! Current active log (0) Previous active log (-1) Archive log 0 Archiving in process Previous active log (-2) Archive log -1 Archiving complete = Previous active log (-3) Archive log -2 Archiving complete = Archive log -3 Data only in archive Archive log -4 Data only in archive Up to 310 * 4 GB active logs Up to 1000 * 4 GB archive logs
- 25. Shared queues – back ups – archive and active logs In order to recover a backup it needs to be accessible to the queue managers, so you need to make sure that the start of the backup remains in the available archive logs for that queue manager, otherwise you can’t recover it! You don’t just need one backup, you need to be able to safely transition from one backup to the next one, should anything fail while the backup is occurring. The green and yellow “usable backups” above illustrate this The conclusion of all this is that you really don’t want a backup to be more than about a quarter of the available archive logs of a queue manager, i.e. < 1 TB at the most For this extreme case you should also be considering a separate queue manager just to perform the backups. This removes the risks of application messages pushing the backup from the archive logs, and also means the back up process can’t affect applications Archive log 0 (newest available) Archive log -999 (oldest available) Usable backup Unusable backup Messages since backup RECOVER CFSTRUCT works RECOVER CFSTRUCT fails Usable backup Messages since backup Messages since backup
- 26. Shared queues – back ups – time and frequency Backing up a structure takes time. The best backup is one that contains minimal data, as that will be fast. IBM recommends taking a backup of every structure at least every hour to minimise the amount of time recovery will take Care is needed here with deep queues! Backing up a large structure will take a long time, and as discussed it will take a lot of log space If your getting application outage can be several days, continuing with your normal backup strategy while the structure contains lots and lots of messages might not be a good idea BACKUP CFSTRUCT(DEEPSTRUCT) Active and archive logs RECOVER CFSTRUCT(DEEPSTRUCT)
- 27. Shared queues – back ups – time and frequency It is worth considering adjusting your back up strategy during extended getting application outages 1) During normal running back up every hour or so (whatever is normal for your site) 2) When the queue starts to get very deep, pause backups, or make them less frequent, until the getting application restarts and the queue depth has reduced This approach minimises the repeated costs of taking a large back up both in terms of CPU and log usage. However it does mean that any recovery will rely on reading a potentially large amount of log data across the queue managers in your group. Therefore you need to accurately size the active and archive logs across your group too, and consider how long recovery might take A good time to do the pause is when the size of the backup starts to become multiple times the amount of data that would normally get written between backups Time Depth Regular backups Getting application outage Getting application restarts Regular backups resume Pause backups
- 28. Shared queues – back ups – recovery Frequent back ups are recommended to minimise the amount of time it takes to recover from a back up Recovery involves reading the logs in the backup and scanning the logs of all queue managers that used the structure since the point of the backup. This results in reading the active / archive logs backwards which is typically slower than reading them forwards With deep queues this could take a significant period of time. But it does require multiple failures all at the same time (getting application outage for a significant period of time, and then subsequent CF structure failure) It might be worth considering CF duplexing to reduce the chance of a structure failure, but bear in mind this will result in increased CF CPU utilization, and more CF storage being required Time Depth Regular backups Getting application outage Getting application restarts Structure fails here Pause backups
- 29. Going deep with MQ on distributed
- 30. Queue files Distributed takes a different approach to z/OS Each queue gets its own in memory set of buffers for temporarily staging messages. A different set of buffers are used for persistent and non-persistent messages These buffers can be tuned, up to a maximum size of 100MB, they default to 128KB for non-persistent messages and 256 KB for persistent ones. These settings aren’t as fully externalized as buffer pools on z/OS. But similar performance considerations apply The buffers asynchronously get written to the queue file, and there is one queue file per queue Q1 Non-persistent message buffers Persistent message buffers Queue file Q2 Non-persistent message buffers Persistent message buffers Queue file Q3 Non-persistent message buffers Persistent message buffers Queue file
- 31. Queue files Queue file size is the ultimate upper limit for queue depth on distributed. Default max size is ~2TB From 9.2.0 adjusting this is simple as shown above. The maximum value is ~255TB Messages are stored on queue files in blocks. If the queue file < 2TB in size the block size is 512 bytes Above 2 TB the block size is 4KB, which means a 1KB message will use a whole block Maximum number of 1KB messages on a single queue is therefore ~68.4 billion DEFINE QL(NEWQUEUE) MAXFSIZE(500) ALTER QL(EXISTINGQUEUE) MAXFSIZE(1000) Create queue with maximum file size of 500MB Alter existing queue to have a max size of 1000MB DIS QSTATUS(NEWQUEUE) CURMAXFS CURFSIZE Queue is using 39 MB of its 500MB QUEUE(NEWQUEUE) CURFSIZE(39) CURMAXFS(500)
- 32. Logging Distributed supports both linear and circular logging Linear logging is of interest with deep queues as it allows you to periodically backup queue files onto the log, i.e. create a media image As with shared queues, on distributed you need to think carefully as to when you create a media image of a very large queue file as it will consume a lot of log space, which you will have to maintain Queue managers can be configured to create media images automatically based on time, or amount of log usage since the last image. If you use this you might want to switch it off for long getting application outages Depth Regular media images Getting application outage Getting application restarts Regular media images resume Last media image
- 34. Recommended reading For z/OS much of this information, along with some example values, is in the capacity and planning guide. I strongly recommend reading it http://ibm-messaging.github.io/mqperf/mp16.pdf For distributed, take a look at https://ibm-messaging.github.io/mqperf/MQ_Performance_Best_Practices_v1.0.1.pdf
- 35. Thank you Matt Leming Architect, IBM MQ for z/OS lemingma@uk.ibm.com © 2022 IBM Corporation
Editor's Notes
- #2: Author – David Ware