Using Data Queues in Modern Applications
Download as PPT, PDF4 likes6,591 views
The document presents an overview of using data queues for program-to-program communication in modern applications, highlighting their role in separating data processing from presentation. It explains the characteristics and configuration of local and remote data queues, as well as the essential APIs and commands used for programming with data queues. Additionally, it discusses design considerations for creating robust data queue communication layers, focusing on speed, scalability, and error handling.
![RPG & Beyond 2009 : Using Data Queues in Modern Applications – THE END Presentation completed! Questions concerning this presentation, data queues in general or data queue code snippets can be directed to me at: [email_address] Thank you!](https://image.slidesharecdn.com/usingdataqueuesinmodernapplications-13218882748797-phpapp01-111121091230-phpapp01/85/Using-Data-Queues-in-Modern-Applications-51-320.jpg)
Using Data Queues in Modern Applications
- 1. Using Data Queues in Modern Applications Presented by Carsten Flensburg RPG & Beyond Web Conference 2009
- 2. RPG & Beyond 2009 : Using Data Queues in Modern Applications Introduction : Data Queues provide a speedy, versatile, and functional program-to program communication method and also provide a perfect coupling and transaction layer in modern application design models, aiming at separating the data processing and business logic from the presentation layer, and thus achieving independence between the back-end and the front-end of the application. Objectives of this presentation: Discuss data queue objects and attributes as well as the Data Queue APIs and their use Learn how to design and develop applications using data queues Present a number of data queue CL commands missing from the i5/OS native collection Point you to more data queue information, articles and programming examples
- 3. Getting Started : What’s a data queue Data queue object and primary attributes : The data queue object is one of two queue type objects available on the IBM System i and recognized by the symbolic object type *DTAQ. The Create Data Queue (CRTDTAQ) command exposes the data queue attributes, some of which are explained below: The primary queue attribute is type, being either: *STD – Standard local data queue *DDM – Remote data queue Depending on the sequence type assigned to the data queue entries added to the data queue will be received in one of three possible sequences: *FIFO – First In First Out *LIFO – Last In First Out *KEYED – Entry binary key value
- 4. Getting Started : How to define a local data queue Data queue attributes - operational : The operational attributes of a local data queue allow you to configure a data queue to optimize it’s capacity and performance to fit the requirements of your application. - Let’s have a brief look at the attributes available as well as their supported values: Maximum entry length – 1-64512 bytes Key length – 1-256 bytes Include sender ID – *NO, *YES Force to auxiliary storage – *NO, *YES Maximum number of entries – *MAX16MB, *MAX2GB Initial number of entries – Positive number Automatic reclaim – *NO, *YES
- 5. Getting Started : How to define a remote data queue Data queue attributes - connectivity : The data queue connectivity attributes enables you to achieve system-to-system communication by means of defining a remote data queue. For this type of data queue only the attributes required to establish the connection between the two systems are specified. You can use either SNA/APPC or RDB based on either SNA or TCPIP to configure the communications layer. The RDB/TCPIP option is by far the simplest of the available options and the recommended approach. Remote data queue & library – Name, *LIBL and *CURLIB Remote location – Name, *RDB Relational database – RDB Entry name APPC device description – Name, *LOC Local location (APPC) – Positive number Mode (APPC)) – Name, *NETATR Remote network id (APPC) – Name, *LOC, *NETATR, *NONE
- 6. Getting Started : How to define a remote data queue using RDB Using the RDB option to identify the remote location is the easiest way to configure DDM data queues in today's TCP/IP networks. The following steps are involved in performing this type of remote data queue configuration: On the system on which you want to configure the DDM data queue, use the Work with Relational Database Directory Entry (WRKRDBDIRE) command to find the RDB Entry name of the remote system. On the CRTDTAQ command, specify *RDB for the Remote location name and the RDB Entry name of the server found in step 1 for the RDB attribute. To enable the user profile that puts or gets data queue entries on or from the preceding DDM data queue to successfully connect to the remote system, the user profile and password need to be identical on the two systems or alternatively you must add a server authentication entry. If you now add a data queue entry to the newly created DDM data queue on the local system, the result is a data queue entry showing in the corresponding data queue on the remote system.
- 7. Data Queue Programming : The Data Queue APIs and Commands Native Data Queue CL commands: Create Data Queue (CRTDTAQ) Delete Data Queue (DLTDTAQ) Work with Data Queues (WRKDTAQ) Start Journal Object (STRJRNOBJ) End Journal Object (ENDJRNOBJ) i5/OS Data Queue APIs: Clear Data Queue (QCLRDTAQ) Receive Data Queue (QRCVDTAQ) Retrieve Data Queue Description (QMHQRDQD) Retrieve Data Queue Message (QMHRDQM) Send to a Data Queue (QSNDDTAQ) Change Data Queue (QMHQCDQ) – 6.1
- 8. Data Queue Programming : The Send Data Queue API The Send Data Queue (QSNDDTAQ) API Required Parameter Group: 1 Data queue name Input Char(10) 2 Library name Input Char(10) 3 Length of data Input Packed(5,0) 4 Data Input Char(*) Optional Parameter Group 1: 5 Length of key data Input Packed(3,0) 6 Key data Input Char(*) Optional Parameter Group 2: 7 Asynchronous request Input Char(10) Optional Parameter Group 3: 8 Data is from a journal entry Input Char(10)
- 9. Data Queue Programming : The Receive Data Queue API The Receive Data Queue (QRCVDTAQ) API Required Parameter Group: 1 Data queue name Input Char(10) 2 Library name Input Char(10) 3 Length of data Output Packed(5,0) 4 Data Output Char(*) 5 Wait time Input Packed(5,0) Optional Parameter Group 1: 6 Key order Input Char(2) 7 Length of key data Input Packed(3,0) 8 Key data I/O Char(*) 9 Length of sender information Input Packed(3,0) 10 Sender information Output Char(*) Optional Parameter Group 2: 11 Remove message Input Char(10) 12 Size of data receiver Input Packed(5,0) 13 Error code I/O Char(*)
- 10. Data Queue Programming : The Retrieve Data Queue Message API The Retrieve Data Queue Message (QMHRDQM) API Required Parameter Group: 1 Receiver variable Output Char(*) 2 Length of receiver variable Input Binary(4) 3 Format name Input Char(8) 4 Qualified data queue name Input Char(20) 5 Message selection information Input Char(*) 6 Length of message selection info Input Binary(4) 7 Message selection info format name Input Char(8) 8 Error code I/O Char(*)
- 11. Data Queue Programming : Data Queue API Prototypes The QRCVDTAQ API prototype: **-- Receive data queue entry: D RcvDtaQe Pr ExtPgm( 'QRCVDTAQ' ) D DtqNam 10a Const D DtqLib 10a Const D DtaLen 5p 0 D Data 32767a Options( *VarSize ) D Wait 5p 0 Const D KeyOrder 2a Const Options( *NoPass ) D KeyLen 3p 0 Const Options( *NoPass ) D Key 256a Const Options( *VarSize: *NoPass ) D SndInfoLen 3p 0 Const Options( *NoPass) D SndInfo 44a Options( *VarSize: *Nopass ) D RmvMsg 10a Const Options( *Nopass ) D DtaRcvLen 5p 0 Const Options( *Nopass ) D Error 32767a Options( *VarSize: *Nopass )
- 12. Data Queue Programming : Data Queue API Prototypes The QSNDDTAQ API prototype: **-- Send data queue entry: D SndDtaQe Pr ExtPgm( 'QSNDDTAQ' ) D DtqNam 10a Const D DtqLib 10a Const D DtaLen 5p 0 Const D Data 32767a Const Options( *VarSize ) D KeyLen 3p 0 Const Options( *NoPass ) D Key 256a Const Options( *VarSize: *NoPass ) D AscRqs 10a Const Options( *NoPass ) D JrnDta 10a Const Options( *NoPass ) The QCLRDTAQ API prototype: **-- Clear data queue: D ClrDtaQ Pr ExtPgm( 'QCLRDTAQ' ) D DtqNam 10a Const D DtqLib 10a Const D KeyOrder 2a Const Options( *NoPass ) D KeyLen 3p 0 Const Options( *NoPass ) D Key 256a Const Options( *VarSize: *NoPass ) D Error 32767a Options( *VarSize: *Nopass )
- 13. Data Queue Programming : Data Queue API Prototypes The QMHQRDQD API prototype: **-- Retrieve data queue description: D RtvDtqDsc Pr ExtPgm( 'QMHQRDQD' ) D RcvVar 32767a Options( *VarSize ) D RcvVarLen 10i 0 Const D FmtNam 10a Const D DtaQ_q 20a Const The QMHRDQM API prototype: **-- Retrieve data queue message: D RtvDtqMsg Pr ExtPgm( 'QMHRDQM' ) D RcvVar 32767a Options( *VarSize ) D RcvVarLen 10i 0 Const D FmtNam 10a Const D DtaQ_q 20a Const D MsgSel 10a Const D MsgSelLen 10i 0 Const D MsgSelFmt 8a Const D Error 32767a Options( *NoPass: *VarSize)
- 14. Data Queue Programming : Data Queue API Prototypes The QMHQCDQ API prototype: **-- Change data queue: D ChgDtaQ Pr ExtPgm( 'QMHQCDQ' ) D DtaQue_q 20a Const D RqsChg 32767a Const D Error 32767a Options( *VarSize ) The QMHQCDQ API change request data structure: **-- API change request: D RqsChg Ds Qualified D NbrVarRcd 10i 0 Inz( %Elem( RqsChg.VarLenRcd )) D VarLenRcd LikeDs( VarLenRcd ) Dim( 2 ) ** D VarLenRcd Ds Qualified D ChgKey 10i 0 D ValLen 10i 0 D NewVal 1a
- 15. Data Queue Programming : Data Queue API Parameter Structures QRCVDTAQ API – Sender information: **-- Sender information: Qualified D SndInf Ds D BytRtn 7p 0 D BytAvl 7p 0 D JobNam 10a D UsrPrf 10a D JobNbr 6a D CurUsr 10a Note the data type packed 7,0 specified for the Bytes returned and Bytes available subfields.
- 16. Data Queue Programming : Data Queue API Parameter Structures QMHQRDQD API - Local data queue information: **-- QMHQRDQD RDQD0100 format: D RDQD0100 Ds Qualified D BytRtn 10i 0 D BytAvl 10i 0 D MsgLen 10i 0 D KeyLen 10i 0 D Seq 1a D IncSndId 1a D FrcInd 1a D TxtDsc 50a D Type 1a D AutRcl 1a D 1a D NbrCurMsg 10i 0 D NbrMsgAlc 10i 0 D DtaQ_Nam 10a D DtaQ_Lib 10a D MaxMsgAlw 10i 0 D InlNbrMsg 10i 0 D DtaQ_Size 10i 0
- 17. Data Queue Programming : Data Queue API Parameter Structures QMHRQRDQD API - Remote data queue information: **-- RDQD0200 format: D RDQD0200 Ds Qualified D BytRtn 10i 0 D BytAvl 10i 0 D AppcDevD 10a D Mode 8a D RmtLocNam 8a D LclLocNam 8a D RmtNetId 8a D RmtDtaQ_Nam 10a D RmtDtaQ_Lib 10a D DtaQ_Name 10a D DtaQ_Lib 10a D RdbNam 18a
- 18. Data Queue Programming : Data Queue API Parameter Structures QMHRDQM API – Message selection formats: **-- RDQS0100 selection format: D RDQS0100 Ds Qualified D SelTyp 1a D 3a D MsgBytRtv 10i 0 **-- RDQS0200 selection format: D RDQS0200 Ds Qualified D SelTyp 1a D KeySchOrd 2a D 1a D MsgBytRtv 10i 0 D KeyBytRtv 10i 0 D KeyLen 10i 0 D KeyVal 256a
- 19. Making the Case : Data queue employment Data queues used as a communication layer: Data queues provide a powerful method to establish an asynchronous communication layer between two programs. The example utility presented later demonstrating the techniques involved, uses local data queues. The information about data queue setup, the programming of the transaction dialogue as well as the data queue API calls apply however irrespective of where the physical data queues are located. In the perspective of the communicating programs, the data exchange is merely a question of addressing the interface provided by the Send and Receive Data Queue APIs, respectively. Data queues also offer a solution to different types of design and architecture objectives. For example, it could be a need to process a workload from a web application located on a separate server accessible from the Internet, while the production data and business logic are on a back-end server behind the company firewall. Or the situation could involve a CPU-intensive application that should have its system workload moved from expensive interactive CPU cycles to less expensive batch cycles. The latter issue is potentially not only a question of cost, but also of sufficient system CPU capacity and thereby application scalability.
- 20. Making the Case : Data queue employment Data queue applicability: To elaborate a bit further on my opening statement, let me go into a little more detail and explain some of the application design objectives that can be achieved with data queues. In general a variety of programming challenges are well served by the queue functionality provided by data queues but in an application design context the following properties are in focus: Speed – program activation already completed Scalability – more jobs waiting on same data queue Threading – different requests handled in parallel Distribution – dispatch request to different servers
- 21. Making the Case : Application Design Design process and considerations: To build a robust and flexible data queue communication layer, the following data protocol and application design objectives are worth considering in the early phase of your development project: Message Format Message Extensibility Message Size Limit Error and Event Communication Language Support Exception Tolerance Debug Facility and Application Monitoring
- 22. Making the Case : Application Design Message format: What kind of formatting should be applied to the message? For internal applications, mapped data structures might suffice, but if the data exchange involves external partners or programming languages with poor data structure facilities, XML is definitely worth considering. Regardless of the format that you choose, divide the message into at least two sections: A message header section containing the control data of the message, such as protocol ID, request ID and type, reply key, message format/version, list control information, offset to and length of message data, error and event codes, and other similar information, if applicable. A message data section containing the actual request or response data.
- 23. Making the Case : Application Design Message Extensibility : How should we handle changes in or additions to the message format? For XML-formatted messages, this concern is well covered; otherwise, including message format or message version/release/modification level information in the message header provides the ticket to safely modify or extend a message in the future. Here’s an example of how a message header could be designed in order to open up for support of future requirements: Data protocol 6 Char Transaction ID 16 Char Request ID 6 Char Message format 8 Char Message language 3 Char Message offset value 30 Char Number of entries requested 4,0 Zoned
- 24. Making the Case : Application Design Message Size Limit: Some message types are by nature limitless in size. For example, when embedded lists or textual descriptions are involved, it might be impossible or useless to define the maximum message length to cover the longest imaginable message length. Instead, you have to include a mechanism that allows a message to be split into more segments. One way of providing such a mechanism is adding a correlation-ID and an end-of-message flag to the message header information. The correlation-ID contains the sequence number of the message segment, and the end-of-message flag is used to signal when the current message segment is the last in the chain. For some purposes, such as displaying lists, a better approach is to have the client request the additional segments one by one as new pages are required. There's no need to build and return a full list if only the first page is ever displayed. In such a setup, a number-of-list-entries field and a list-offset field is added to the message header. The list-offset field contains a value that uniquely identifies a list entry, such as a customer number or product ID. In some cases, this could also be a composite value to ensure uniqueness.
- 25. Making the Case : Application Design Error and Event Communication: How should error and informational messages be communicated within the data protocol, to ensure a correct and helpful dialogue with the application user? Defining unique error codes and event codes and related messages helps the receiving part of the application correctly identify and properly communicate an error or event. Message files can be of great help in creating and administering error messages and event messages. Language Support: Do you need to support more than one language within the data protocol? If so, you must include information in the message header to define the language that applies to the message dialogue, so that all language-sensitive information can be presented in the correct language.
- 26. Making the Case : Application Design Exception Tolerance: Foreseeable, logical errors, such as a customer record not found, should be handled by the error and event communication facility that I just described. However, it is very important to include in your application design an exception trap mechanism that catches — and communicates back — unforeseen errors. These errors could be any kind of exception that occurs during program execution and that unhandled would lead to an inquiry message being sent to the QSYSOPR message queue and bring the job to a screeching halt. Over time, unhandled errors could lead to all the server jobs hanging in a message wait, and before that suffering performance as the number of message waiting jobs grows. Registering ILE condition handlers, adding RPG/IV *PSSR subroutines, and coding ditto monitor groups and (e)rror opcode extenders are all vital instruments in establishing the defensive programming style (also) required for this type of application.
- 27. Making the Case : Application Design Debug Facility and Application Monitoring: How do you provide for daily monitoring of the application and investigation of problems and errors? One way is to include log files to record the transaction dialogue as well as the application errors and events. If necessary because of performance or storage considerations, a switch could control the transaction logging and activate it only if debugging the transaction flow or message dialogue is necessary. Building a Sample Data Queue Application: The Display TCP/IP Servers (DSPTCPSVR) command. The DSPTCPSVR command displays a list of all or a subset of the TCP/IP servers available on your system. Using the display option in the list panel, you can further display all the selected server's start attributes from the QATOCSTART file.
- 28. Making it Work : The Data Protocol Data Protocol Design: The following are the requirements for my data protocol design as far as request and response types are concerned: Request a list of TCP/IP servers, optionally subset by server type. Return a list of TCP/IP servers, including server file key information. Request TCP/IP server information for a specific server. Return TCP/IP server information for a specific server.
- 29. Making it Work : The Data Protocol Here's what the QATOCSTART file layout looks like: File . . . . . : QATOCSTART Record format . . . : QTOCSTRT Library . . . : QUSRSYS Record length . . . : 240 Field name Field type Buffer Length Key Column heading SERVERTYPE Char 1 1 SVR TYP SERVER Char 2 30 1 U Server AUTOSTART Char 32 4 Auto Start LIBRARY Char 36 10 Library of Program PROGRAM Char 46 10 Program to Call EXTSTRCMD Char 56 64 External Start CMD EXTENDCMD Char 120 64 External End CMD
- 30. Making it Work : The Data Protocol Data protocol: TCPSVR Request ID: SVRLST - Inbound Request header: Data protocol 6 Char Transaction ID 16 Char Request ID 6 Char Message format 8 Char Message language 3 Char Message offset value 30 Char Number of entries requested 4,0 Zoned Valid range 1-24 Request data: Server type 1 Char
- 31. Making it Work : The Data Protocol Data protocol: TCPSVR Request ID: SVRLST - Outbound Response header: Data protocol 6 Char Transaction ID 16 Char Request ID 6 Char Event code 4,0 Zoned Error code 4,0 Zoned Event message offset 4,0 Zoned Error message offset 4,0 Zoned Response data offset 4,0 Zoned Message offset value 30 Char Number of entries returned 4,0 Zoned Entry length 4,0 Zoned Response data: Server type 1 Char Server key name 30 Char Server name 30 Char Auto start 4 Char
- 32. Making it Work : The Data Protocol Data protocol: TCPSVR Request ID: SVRATR - Inbound Request header: Data protocol 6 Char Transaction ID 16 Char Request ID 6 Char Message format 8 Char Message language 3 Char Request data: Server key name 30 Char
- 33. Making it Work : The Data Protocol Data protocol: TCPSVR Request ID: SVRATR - Outbound Response header: Data protocol 6 Char Transaction ID 16 Char Request ID 6 Char Event code 4,0 Zoned Error code 4,0 Zoned Event message offset 4,0 Zoned Error message offset 4,0 Zoned Response data offset 4,0 Zoned Response data: Server type 1 Char Server name 30 Char Auto start 4 Char Program name 10 Char Program library 10 Char Start command 64 Char End command 64 Char
- 34. Making it Work : Logging the Transactions Error and event log file example: File . . . . . : CBX1691F Record format . . . : CBX1691R Library . . . : QGPL Record length . . . : 595 Field name Field type Buffer Length Column LGTSTP Timestamp 1 26 Log timestamp LGTYPE Char 27 6 Log type (LOGERR, LOGEVT) LGTRID Char 33 16 Transaction ID LGPGMN Char 49 12 Program/module name LGFUNC Char 61 12 Function (*PSSR, Monitor) LGDGCD Zoned 73 4 0 Diagnostic code LGDGMS Char 77 7 Diagnostic message ID LGDGDT Char 84 512 Diagnostic data
- 35. Making it Work : Logging the Transactions Transaction log file example: File . . . . . : CBX1692F Record format . . . : CBX1692R Library . . . : QGPL Record length . . . : 4157 Field name Field type Buffer Length Column LGTSTP Timestamp 1 26 Log timestamp LGTYPE Char 27 6 Log type (LOGTRN) LGTRID Char 33 16 Transaction ID LGDTPC Char 49 6 Data protocol LGRQID Char 55 6 Request ID LGRQTP Char 61 1 Request type (I/O) LGDATA Char 62 4096 Log data
- 36. Making it Work : Separating the Data Flows Why separate data flows: Inbound traffic is typically handled first-in-first-out, whereas outbound traffic is keyed to ensure that the server reply is received by the correct request sender. To support this requirement efficiently, the inbound data queue should therefore be created with sequence *FIFO, and the outbound data queue should be created with sequence *KEYED. Separating data flow means that you avoid the risk of data queue object lock conflicts between the sending and receiving processes. Loss of client or server side processing is easier to detect if the inbound and outbound transactions are not mixed. This setup also makes it easier to monitor the inbound workload to ensure that sufficient server jobs are available to process the incoming requests.
- 37. Making it Work : Application Data Queues Here are the commands to create the two data queues needed to establish the program-to-program communication in the preceding setup: CRTDTAQ DTAQ(CBX169I) MAXLEN(8192) FORCE(*NO) SEQ(*FIFO) SENDERID(*NO) SIZE(*MAX2GB 16) AUTORCL(*YES) TEXT('Sample data queue application inbound queue') CRTDTAQ DTAQ(CBX169O) MAXLEN(8192) FORCE(*NO) SEQ(*KEYED) KEYLEN(16) SENDERID(*NO) SIZE(*MAX2GB 16) AUTORCL(*YES) TEXT('Sample data queue application outbound queue')
- 38. Making it Work : The Application Components The Data Queue Sample Application includes the following sources: CBX169 -- RPGLE -- Data Queue Sample App - Service functions CBX169B -- SRVSRC -- Data Queue Sample App - Binder source CBX1691 -- RPGLE -- Run Data Queue Server - TCPSVR Protocol CBX16911 -- RPGLE -- Run Data Queue Server - SVRLST Request CBX16912 -- RPGLE -- Run Data Queue Server - SVRATR Request CBX1691H -- PNLGRP -- Run Data Queue Server - Help CBX1691X -- CMD -- Run Data Queue Server CBX1692 -- RPGLE -- Display TCP/IP Servers - CPP CBX1692E -- RPGLE -- Display TCP/IP Servers - UIM General Exit Pgm CBX1692H -- PNLGRP -- Display TCP/IP Servers - Help CBX1692L -- RPGLE -- Display TCP/IP Servers - UIM List Exit Program CBX1692P -- PNLGRP -- Display TCP/IP Servers - Panel Group CBX1692X -- CMD -- Display TCP/IP Servers CBX1691F -- PF -- Error/event log file CBX1692F -- PF -- Transaction log file CBX169M -- CLP -- Data Queue Sample App – Build Application
- 39. Making it Work : Performing a Test Run After successful creation of all objects, follow these steps to perform a test run: Using the Change Current Library (CHGCURLIB) command, change your job's current library to the one containing the application. Using the Run Data Queue Server (RUNDTAQSVR) command, start the server process: RUNDTAQSVR LOGTRN(*YES). To avoid locking up your interactive session, submit the command to batch. Check that the submitted job has gone active before proceeding. To generate some work for the server job, run the DSPTCPSVR command. The DSPTCPSVR command requests the list of TCP/IP servers, one page at a time, from the server job through the inbound data queue CBX169I and receives its reply from the outbound data queue CBX169O. Every time you page down, the next block of TCP/IP servers is requested. The list request is sent from the CBX1692L UIM List Exit Program. The list request is processed by the CBX1691 and CBX16911 data queue server programs.
- 40. Making it Work : Performing a Test Run Test run steps – continued: The server job logs all transactions to the transaction log file CBX1692F. Using the Display Physical File (DSPPFM) command, you can monitor the request/response dialogue in the transaction log file CBX1692F: DSPPFM FILE(CBX1692F) FROMRCD(*END). From the DSPTCPSVR panel, you can select option 5=Display server start information. Doing so generates one request/response per selected TCP/IP server. This can of course also be monitored in the transaction log file. The server information request is sent from the CBX1692E UIM General Exit Program. The server information request is processed by the CBX1691 and CBX16912 data queue server programs. To complete the test run, end the server job submitted in step 2. If you ran the data queue server job interactively, use the SysRqs (System Request) key in that job and specify option 2, which runs the End Request (ENDRQS) command.
- 41. Data Queue Programming : New CL Commands New data queue CL commands: Display Data Queue Description (DSPDTAQD) - (1) Clear Data Queue (CLRDTAQ) - (1) Send Data Queue Entry (SNDDTAQE) - (1) Copy Data Queue Description (CPYDTAQD) - (2) Display Data Queue Entries (DSPDTAQE) - (3) Work with Data Queues (WRKDTAQ2) - (4) The data queue commands were all published as part of the Programming Tips Newsletter’s APIs by Example article series. Links to the articles explaining the commands in detail as well as links to the zip files containing all the source code are found on slide 44. (1) - APIs by Example: Data Queue APIs and CL Commands, part 1 (2) - APIs by Example: Data Queue APIs and CL Commands, Part 2 (3) - APIs by Example: Data Queue APIs and CL Commands, Part 3 (4) - APIs by Example: Data Queue APIs and CL Commands, Part 4
- 42. Data Queue Programming : Work with Data Queues (WRKDTAQ2) Command Work with Data Queues WYNDHAMW 28-08-09 21:04:20 Type options, press Enter. 1=Send entry 3=Copy 4=Delete 5=Display description 8=Display entries 13=Change description 14=Clear Opt Data queue Library Type Text QPMJ12T017 QMPGDATA *STD QQFEQ04906 QQFTEMP *STD QSNMPSAQ QUSRSYS *STD QSRVCTL QSERVICE *STD QSRVMON QSERVICE *STD QSYSDTAQ QSYS *STD Exit point: QIBM_QWT_JOBNOTIFY -format: NT QS9HDW QSRVAGT *STD QS9PALTIME QSRVAGT *STD QYIVPUBDTQ QUSRSYS *STD Central PublishingAgent Data Queue QYPSJDTAQ QMGTC2 *STD QYPSJSVR data queue QYPSJDTAQ QUSRSYS *STD QYPSJSVR data queue More... Parameters or command ===> F3=Exit F4=Prompt F5=Refresh F6=Create data queue F9=Retrieve F11=Display full text F12=Cancel F17=Top F18=Bottom
- 43. Data Queue Programming: System-to-Program Communication Besides the efficient and flexible program-to-program communication support that data queues provide, they are also useful for enabling system-to-program communication. I've collected information and programming examples at the end of this article, documenting the following operating system usage of data queues: Display File (DSPF): Specifying a data queue for the CRTDSPF, CHGDSPF, or OVRDSPF commands' DTAQ parameter lets your interactive application be event driven (i.e., wait for user input and data queue input at the same time), as well as lets it time out the display file if no action occurs within a user-defined time limit. For further documentation and a programming example, please follow the link provided at the end of this presentation. Intersystem Communications Function File (ICFF): Specifying a data queue for the CRTICFF, CHGICFF, or OVRICFF commands' DTAQ parameter lets your communication application be event driven (i.e., wait for local program input and remote system input to the communications file at the same time), as well as lets it time out the communication if no action occurs within a user-defined time limit. A link to documentation is at the end of this presentation.
- 44. Data Queue Programming: System-to-Program Communication System-to-Program Communication – continued: Spooled file support 1: Specifying a data queue for the CRTOUTQ or CHGOUTQ commands' DTAQ parameter generates a predefined data queue entry to the specified data queue every time a spooled file enters the ready (RDY) status on that output queue. Links to documentation and a programming example are at the end of this presentation. Spooled file support 2: Adding the QIBM_NOTIFY_CRTSPLF environment variable at either system or job level generates a predefined data queue entry to the data queue specified in the environment variable, whenever a spooled file is created on the system or in the job for which the environment variable was added. Links to documentation and a programming example are at the end of this presentation. Job event notification: Registering a data queue for the QIBM_QWT_JOBNOTIFY exit point and specifying a combination of event types and subsystems generates a predefined data queue entry whenever a specified job event (e.g., job placed on job queue, job starts, job ends) takes place in the specified subsystem. The programming example at the link provided at the end of this presentation reveals the details of setting up a job monitor using this exit point, and a link to the exit point documentation is also provided.
- 45. More Information on System iNetwork Previously published APIs by Example articles: APIs by Example: Data Queue APIs and CL Commands, part 1: http://systeminetwork.com/article/apis-example-data-queue-apis-and-cl-commands http://www.pentontech.com/IBMContent/Documents/article/53542_143_DataQueue1.zip APIs by Example: Data Queue APIs and CL Commands, Part 2: http://systeminetwork.com/article/apis-example-data-queue-apis-and-cl-commands-part-2 http://www.pentontech.com/IBMContent/Documents/article/53685_149_DataQueue2.zip APIs by Example: Data Queue APIs and CL Commands, Part 3: http://systeminetwork.com/article/apis-example-data-queue-apis-and-cl-commands-part-3 http://www.pentontech.com/IBMContent/Documents/article/53850_156_DataQueue3.zip APIs by Example: Data Queue APIs and CL Commands, Part 4: http://systeminetwork.com/article/apis-example-data-queue-apis-and-cl-commands-part-4 http://www.pentontech.com/IBMContent/Documents/article/54001_161_DataQueue4.zip APIs by Example: Data Queue APIs and CL Commands, Part 5: http://systeminetwork.com/article/apis-example-data-queue-apis-and-cl-commands-part-5 http://www.pentontech.com/IBMContent/Documents/article/54098_170_DataQueue5.zip NOTE: The Data Queue CL commands presented here are available for download from the above articles. For your convenience the link to the zip file containing the object sources is also included immediately below the article link.
- 46. More Information on System iNetwork Data queue articles and utility examples: Using Data Queue APIs: http://systeminetwork.com/article/using-data-queue-apis Using Data Queues in RPG/IV: http://systeminetwork.com/article/using-data-queue-apis-rpg-iv Data Queues: A PC-to-iSeries Quick Link: http://systeminetwork.com/article/data-queues-pc-iseries-quick-link Automatic Notification When Spool Files Are Created: http://systeminetwork.com/article/club-tech-iseries-systems-management-newsletter-20 Automatic Spool Rerouting: http://systeminetwork.com/article/automatic-spool-rerouting APIs by Example: Realtime Job Monitor: http://systeminetwork.com/article/apis-example-realtime-job-monitor An Introduction to Asynchronous Messages: http://systeminetwork.com/article/introduction-asynchronous-messages User queue articles: APIs by Example: Creating a User Queue: http://www.systeminetwork.com/article.cfm?id=17972 APIs by Example: User Queues, Part 2: http://www.systeminetwork.com/article.cfm?id=18097
- 47. More Information from IBM IBM Data Queue API Documentation: Data Queue APIs: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/obj2.htm Using Data Queues: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/obj2a.htm Clear Data Queue (QCLRDTAQ) API: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qclrdtaq.htm Receive Data Queue (QRCVDTAQ) API: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qrcvdtaq.htm Retrieve Data Queue Description (QMHQRDQD) API: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qmhqrdqd.htm Retrieve Data Queue Message (QMHRDQM) API: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qmhrdqm.htm Send Data Queue (QSNDDTAQ) API: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apis/qsnddtaq.htm
- 48. More Information from IBM IBM Technical Documents: Configuring DDM Data Queue Support over TCP/IP: https://www-912.ibm.com/s_dir/slkbase.NSF/1ac66549a21402188625680b0002037e/e95c5072635554dd86256e980068aada?OpenDocument Assigning a Default User Profile for IBM DRDA over TCP/IP: https://www-912.ibm.com/s_dir/slkbase.NSF/643d2723f2907f0b8625661300765a2a/9e61c67ade70359986256afd007cc9f1?OpenDocument IBM AnyNet Configuration (SNA over TCP/IP) 1: https://www-912.ibm.com/s_dir/slkbase.nsf/1ac66549a21402188625680b0002037e/c5057870085200f8862565c2007cdc47?OpenDocument IBM AnyNet Configuration (SNA over TCP/IP) 2: https://www-912.ibm.com/s_dir/slkbase.nsf/1ac66549a21402188625680b0002037e/c0e073485d35f074862569da0063626f?OpenDocument Source System Security in a TCP/IP Network — Server Authentication - V5R4: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/ddm/rbae5sourcesecurity.htm Configuring EE (Enterprise Extender) between Two IBM System i Systems: http://www-912.ibm.com/s_dir/slkbase.nsf/1ac66549a21402188625680b0002037e/ed75fbb6841d9f718625710a004963a3?OpenDocument
- 49. Cross-Platform Data Queue Support System iNetwork: Exploring the Client Access Data Queue APIs: http://systeminetwork.com/article/exploring-client-access-data-queue-apis Programming with ODBC and Data Queues: http://systeminetwork.com/article/programming-odbc-and-data-queues AS/400 Toolbox: Using Dataqueue, Record, and RecordFormat Classes: http://systeminetwork.com/article/as400-toolbox-using-dataqueue-record-and-recordformat-classes Data Queues: A PC-to-iSeries Quick Link: http://systeminetwork.com/article/data-queues-pc-iseries-quick-link Use .NET to Develop iSeries Data Queue Applications: http://systeminetwork.com/article/use-net-develop-iseries-data-queue-applications Queue Up to Work with Data Queues in .NET Programs: http://systeminetwork.com/node/26092 Work with the Data Queues API in .NET Applications: http://systeminetwork.com/article/work-data-queues-api-net-applications Accessing SQL Server Databases with Java (using data queues): http://systeminetwork.com/article/accessing-sql-server-databases-java
- 50. Cross-Platform Data Queue Support IBM: Using the Client Access for Microsoft Windows 95 and Windows NT OCX Control and OLE Automation Objects with Visual Basic: http://www-912.ibm.com/s_dir/slkbase.NSF/515a7ef1f8deef8c8625680b00020380/def5574d27de318e862565c2007cbc3c?OpenDocument CWBDQ: Q&A for the Optimized Data Queue API: http://www-912.ibm.com/s_dir/slkbase.NSF/515a7ef1f8deef8c8625680b00020380/2202a0b08aebc41a862565c2007cb060?OpenDocument IBM Toolbox for Java: http://www-03.ibm.com/servers/eserver/iseries/toolbox/overview.html Data Queue Host Server Does Not Support DDM Data Queues: http://www-912.ibm.com/s_dir/slkbase.NSF/f5ed8d76fdf9afb88625680b00020384/8bc86c06f162066b86256d91006a6cac?OpenDocument Websphere MQ (licensed product): http:// www-01.ibm.com/software/integration/wmq/
- 51. RPG & Beyond 2009 : Using Data Queues in Modern Applications – THE END Presentation completed! Questions concerning this presentation, data queues in general or data queue code snippets can be directed to me at: [email_address] Thank you!
Editor's Notes
- #2: Welcome to this webcast discussing the employement of data queues in modern applications!
- #3: Over the years I’ve seen data queues being used for a variety of mission critical and performance sensitive applications. Examples of such applications include OTC (On Truck Computer) distribution and billing systems, credit card transaction and monitoring systems, travel reservation systems as well as a number of full scale web business applications. Although I during this time have run into minor issues and challenges using data queues, I’m generally very impressed with the speed, robustness and functionality offered by data queues. Adding to this that the asynchronous program-to-program communication method offered by data queues also very well support contemporary application development and design models is yet another quality distinguishing data queues. When using data queues in application development you’ll sometimes encounter situations calling for the assistance of a data queue CL command in order to quickly progress the task at hand. The i5/OS operating system however only include the very basic CL commands and I’ve therefore developed my own, additional set of data queue CL commands which I will also be presenting today along with links to other on-line data queue resources.
- #4: I’ll briefly talk about the other queue object on the i5/OS, known as a User queue and having a symbolic object name of *USRQ a little later - and also point you to articles and programming examples covering user queues. Standard data queues are data queues residing on the local system as opposed to DDM (Distributed Data Management) data queues pointing to data queues residing on other (remote) systems. The *DDM attribute refers to the original implementation of remote data queues which was established by the DDM architecture and a SNA/APPC (Systems Network Architecture/Advanced Program-to-Program Communication) infrastructure including configuration attributes such as Remote location , APPC device description , Local location , Mode and Remote network identifier and which involved all the challenges of making such a setup work. Luckily, today another, simpler approach of defining the remote location by means of the Relational Data Base (RDB) infrastructure is available – more details to follow. Note that data queue entries are sometimes referred to as messages , but for the context of this presentation I’ll simply be using the term entry . As for the significant data queue Sequence attribute the following brief explanation is intended to serve as a general guideline when defining data queues: A First-in-first-out (FIFO) sequenced data queue ensure that entries are received in the same order as they were added. This is for example useful for an application inbound transaction data queue, causing all transactions to be served in the order they arrive. A Last-in-first-out (LIFO) sequenced data queue ensure that entries are received in the opposite order as they were added. This could be useful to establish a stack type of object, for example preserving the call level order of entries in a system data queue. As call levels are added and current application or runtime values are overwritten, these are saved in the LIFO data queue and retrieved and applied on return from higher call levels. A keyed data queue supports a variety of design objectives such as ensuring that specific data queue entry processing programs receive only the entries that they are supposed to handle - or that a client submitting a data queue entry to a server will receive only the response for which it is specifically waiting.
- #5: By local data queue I’m referring to a data queue of TYPE(*STD) as opposed to a remote data queue of TYPE(*DDM). The Maximum entry length simply defines the longest entry that it will be possible to add to the data queue. You are allowed to add shorter entries, but note that the space occupied by the entry will always be the size specified for this attribute. A value in the range of 1 to 64512 bytes is supported. The Key length of course applies to keyed data queues only and specifies required key size for entries added to the data queue. Note that the key value itself is regarded as a binary value when processed by the data queue, there’s no notion of character set or decimal precision. A value in the range of 1 to 256 bytes is supported. If you choose to Include sender ID , all data queue entries received from the data queue will include a separate data structure containing information about the job adding the entry to the data queue. This information contains the qualified job name plus the current user profile. The Force to auxiliary storage attribute specifies whether entries added to or received from the data queue are immediately forced to auxiliary storage. This ensures that the changes are not lost if a system failure occurs, but of course also adds a price tag in the form of additional system overhead. The Maximum number of entries attribute defines the maximum size that the data queue will be allowed to occupy and consequently also controls how many entries it is possible to add to the message queue at any time. In high volume transaction applications you could very quickly reach the point where it is no longer possible to add entries to a data queue, if the job(s) processing the entries for some reason come to a halt. This point may also be reached if the processing jobs’ are falling behind due to lack of scalability in an application. The Initial number of entries specifies how much space is initially allocated for the data queue. Once this limit is reached the data queue will have to be extended to allow further entries to be added, an operation causing delay and the use of system resources. Since data queues are allowed to grow up to a size of 2 gigabytes it is quite useful easily being able to reclaim this storage. The Automatic reclaim parameter controlling whether the storage allocated to the data queue is released whenever the entry count goes from one to zero. At that point the value *YES indicates that the storage allocated should be reset to reflect the value specified for the Initial number of entries attribute.
- #6: And here it’s the other way round: By remote data queue I’m referring to a data queue of TYPE(*DDM) – as opposed to a local data queue of type (*STD). The Remote Data Queue specifies the name of the existing data queue on the remote system and this is the target of the data queue operation performed against a remote data queue and the actual physical location of data queue entries. The Remote location attribute identifies the system where the remote data queue resides. Setting this attribute to the *RDB special value you only have to also specify the Relational database attribute to complete the data queue description as far as connectivity is concerned. Choosing the SNA/APPC option you’ll further have to specify the correct APPC Local location , Mode as well as Remote Network Identifier attributes for the configuration to be operational. At the end of this presentation I’ve included links to IBM Technical Documents describing in detail how to configure both SNA/APPC and RDB/TCPIP. I clearly recommend the latter option because IBM plans to discontinue SNA at some point in the future and the RDB configuration is much simpler to perform. More about the RDB approach in a minute. As for the application design perspective DDM data queues offer a convenient way to let two applications on different System i servers communicate with each other. One scenario could be two companies that want to integrate their business applications due to a business merger effort. Another scenario could be the desire to integrate a business application running on one server with a new finance application running on yet another server. And a third example could be a web front-end application needing real-time access to a back-end business application running on a central server behind the company firewall.
- #7: If there’s no RDB entry for the system you want to connect to, use the Add Relational Database Directory Entry (ADDRDBDIRE) command to add it. Follow the help text of the ADDRDBDIRE command or the IBM Technical Document (link at the end of this presentation) to create the RDB directory entry. I keep the names of the local and remote data queues the same while the libraries of the respective data queues might vary from system to system. In the following example, the RDB server name is CPHDB01, and the data queue name is WEBINBDQ in library QGPL on both systems: CRTDTAQ DTAQ(QGPL/WEBINBDQ) TYPE(*DDM) RMTDTAQ(QGPL/WEBINBDQ) RMTLOCNAME(*RDB) RDB(CPHDB01) TEXT('Web application inbound data queue') To avoid the user profile and password synchronization requirement you can use a Server Authentication Entry to store the remote user profile name and password. In the following example, I use the data queue application user profile WEBAPP as the local user profile and the RDB Entry name as the server name to achieve a successful connection with the remote user profile CPHWEBUSR: ADDSVRAUTE USRPRF(WEBAPP) SERVER(CPHDB01) USRID(CPHWEBUSR) PASSWORD(remote password). Before adding server authentication entries on your system, please refer to "Source System Security in a TCP/IP Network — Server Authentication" (link at the end of this presentation), which includes a thorough discussion of the Retain Server Security (QRETSVRSEC) system value. The WEBINDBDQ data queue must exist in library QGPL on the remote system and the connection to the remote system be operational. Otherwise, the Send Data Queue (QSNDDTAQ) API call adding the data queue entry immediately will return an exception specifically indicating the cause of failure. Performing the same drill for the WEBOUTDQ data queue on the two systems should lead to the existence of an inbound data queue and an outbound data queue on both systems. I usually keep the local data queue on the system on which the data queue entries are being processed. So I put data queue entries on a DDM data queue, and I get data queue entries from a local data queue. That way, when sending the data queue entry, I immediately know whether the communication was successful, and I also consistently know where to perform recovery and error processing.
- #8: The Create Data Queue (CRTDTAQ) command creates a data queue description. The Delete Data Queue (DLTDTAQ) command deletes a data queue and implicitly all entries within it. The Work with Data Queue (WRKDTAQ) command displays a panel listing all selected data queues. From the panel you can run the CRTDTAQ, DLTDTAQ as well as Change Object Description (CHGOBJD) command. The Clear Data Queue (QCLRDTAQ) API clears all or, if the data queue is keyed, a selected range of entries from a data queue. The Receive Data Queue (QRCVDTAQ) API receives a single entry from the specified data queue. The Retrieve Data Queue Description (QMHQRDQD) API retrieves information about a data queue. The Retrieve Data Queue Message (QMHRDQM) API retrieves one or more entries from a data queue without removing the retrieved entries. The Send to a Data Queue (QSNDDTAQ) API sends data to the specified data queue. The Change Data Queue (QMHQCDQ) API allows you to change selected data queue attributes. Introduced with release 6.1 It is also worth noting that you can journal local data queues. Issuing the Start Journaling Object (STRJRNOBJ) command against a data queue causes all actions, including entry data being added or removed, against the data queue to be logged to the journal specified. The End Journal Object (ENDJRNOBJ) command terminates the journal logging against the data queue. As mentioned earlier the offering of native Data Queue CL commands available is far from impressive. Developing, testing and debugging data queue applications quickly prompts the need of CL commands offering functionality similar to what is provided by the listed APIs. In an attempt to fill that gap I’ve created 6 additional CL commands allowing you to display and copy data queue description, send, display and clear data queue entries as well as an enhanced work with data queue command providing an interface to all relevant data queue commands. More about this later. When designing and building applications relying on the use of data queues, the core data queue functionality is mainly delivered by the Send to a Data Queue and the Receive Data Queue APIs, respectively – so let us have a closer look at that dynamic duo.
- #9: For those of you being used to API programming it’s obvious that the QSNDDTAQ and QRCVDTAQ APIs do not follow the regular API convention for declaring API parameters. Seeing parameters of type packed decimal is not an everyday experience in an API programmer’s life and in this case points back to these APIs history as belonging to the first set of APIs made available by IBM all the way back in the days of the System/38. In fact you’ll find early documentation of the small group of historic APIs in the System/38 Environment Programming manual, Version 3; page 38, at the following location: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/books/sc413735.pdf#page=38 QSNDDTAQ error handling: The general API error data structure parameter is missing, so in order to catch exceptions following a call to the QSNDDTAQ API you have to resort to the appropriate facilities provided by your programming language. For RPG/IV that usually resolves into an CALLP(E) opcode extender or a MONITOR group, but other options are also available as documented and discussed by the ILE RPG Programmer's Guide section on Handling Exceptions: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/index.jsp?topic=/books/c092507602.htm The Asynchronous request parameter defines whether a send data queue request to a DDM data queue is processed asynchrounously. This parameter consequently only applies to DDM data queues. The eight parameter Data is from a journal entry applies to a scenario where the data queue object itself is journaled. Check out the QSNDDTAQ API documentation for more information.
- #10: If optional parameter group 1 is employed you must ensure that a program variable is specified for parameter 10, Sender information , even if parameter 9, Length of sender information , specifies a length of zero. Failing to do so will cause the call to QRCVDTAQ to return the exception message CPF24B4, Severe error while addressing parameter list. Another caveat is based on a data queue object cache maintained by the operating system: Internally, when a job uses the QSNDDTAQ, QRCVDTAQ, QMHQRDQD, or the QMHRDQM API, a cache is created to allow faster access to the data queue. An entry in the cache means a user is authorized to the data queue. An entry is added to the cache when a user calling one of the APIs has the required authority to the data queue. An entry is also added to the cache when QSNDDTAQ is called to handle a journal entry for a data queue created with the sender ID attribute set to *YES, and the user requesting the send function has the required authority to the current profile name in the sender ID information of the journal entry. The data in the cache is used until the job ends, so if you need to immediately change a user's authority to one of these objects, you may need to end that user's jobs. The cache also implies that if you rename or move a data queue which already has been accessed by a program, the program will continue to address the original data queue, irrespective of an identical named data queue being placed in the same library. Check out the API data queue library documentation for the mentioned APIs for more details.
- #11: The data queue entry retrieval performed by the QMHRDQM API is non-destructive. That implies that all entries retrieved remain in the data queue. This behavior makes the QMHRDQM API an ideal candidate to perform data queue monitoring duties. This allows you for example to detect if entries with a specific key value are not being processed or are queueing up and then perform the necessary recovery action of either reporting the problem or starting the appropriate server job.
- #12: The Receive Data Queue (QRCVDTAQ) API receives data (i.e. an entry) from the specified data queue. When more than one program has a receive pending on a data queue at one time, a data entry sent to the data queue is received by only one of the programs. The program with the highest run priority receives the entry. The next entry sent to the queue is given to the job with the next highest priority. This is very practical in a testing or debugging scenario. By changing the job run priority of your job running the program that calls the QRCVDTAQ API to a value lower (higher priority) than other jobs accessing the same data queue, you ensure that your job will receive the next data queue entry arriving at the data queue. Whenever the last entry on the queue is received, the storage allocated to the data queue will be reduced to the storage needed for the initial number of entries defined for the data queue, given that the data queue was created with the AUTORCL keyword on the Create Data Queue (CRTDTAQ) command set to *YES. Distributed data management (DDM) data queues are supported using this API. This means that you can use this API to receive a message from a data queue that exists on a remote IBM System i. However, using this API to receive messages without removing them from the data queue is not supported for DDM data queues.
- #13: The Send Data Queue (QSNDDTAQ) API sends data (i.e. an entry) to the specified data queue. Note that the storage allocated for each entry will be the value specified for the maximum entry length on the Create Data Queue (CRTDTAQ) command, irrespective of the value specified for the data length on the QSNDDTAQ call. Distributed data management (DDM) data queues are supported using this API. This implies that you can use the QSNDDTAQ API to send data to a data queue that exists on a remote IBM System i. The Clear Data Queue (QCLRDTAQ) API clears all data from the specified data queue, or clears messages that match the key specification from a keyed data queue. If the data queue was created with the AUTORCL keyword on the Create Data Queue (CRTDTAQ) command set to *YES, the clear operation will cause storage allocated to the data queue will be reduced to the storage needed for the initial number of entries defined for the data queue. Distributed data management (DDM) data queues are supported using this API. This means that you can use this API to clear a data queue that exists on a remote iSeries. Clearing messages by key is however not supported for DDM data queues.
- #14: The Retrieve Data Queue Description (QMHQRDQD) API retrieves the description and attributes of a data queue. Examples include the number of entries currently on the data queue, the text description of the data queue, whether the queue includes sender ID information, and whether the data queue is keyed. The attributes of a distributed data management (DDM) data queue can be retrieved with this API. The Retrieve Data Queue Message (QMHRDQM) API retrieves one or more messages from a data queue, thereby enabling the retrieval of multiple messages per call. The message selection information parameter allows you to have some control over which messages are returned and allows you to specifically retrieve one of the following: The first or last message of a data queue All messages of a data queue Selected messages from a keyed data queue The QMHRDQM API is similar in function to the QRCVDTAQ API. However, the QRCVDTAQ API can remove the received message from the data queue; QMHRDQM API does not remove received messages. Distributed data management (DDM) data queues are not supported using this API.
- #15: The QMHQCDQ API introduced with release 6.1 allows you to alter a data queue objects auto reclaim storage attribute as well as a new lock enforcement attribute controlling whether object locks issued against the data queue should be enforced or not. Here’s a brief example demonstrating how to set up the call to the QMHQCDQ API: **-- Global constants: D AUT_RCL c 100 D ENF_DTQ_LCK c 200 /Free RqsChg.VarLenRcd(1).ChgKey = AUT_RCL; RqsChg.VarLenRcd(1).ValLen = %Size( VarLenRcd.NewVal ); RqsChg.VarLenRcd(1).NewVal = '1'; RqsChg.VarLenRcd(2).ChgKey = ENF_DTQ_LCK; RqsChg.VarLenRcd(2).ValLen = %Size( VarLenRcd.NewVal ); RqsChg.VarLenRcd(2).NewVal = '1'; ChgDtaQ( PxDtaQue_q : RqsChg : ERRC0100 ); /End-Free
- #16: The Sender information data structure used when calling the QRCVDTAQ API with optional parameter group 1. A total length of 44 bytes in its current incarnation.
- #17: The Retrieve Data Queue Description (QMHQRDQD) APIs return data structure format for local data queues – RDQD0100.
- #18: The Retrieve Data Queue Description (QMHQRDQD) APIs return data structure format for remote data queues – RDQD0200.
- #19: The Retrieve Data Queue Message (QMHRDQM) API offers two message selection formats: The RDQS0100 format to select messages when using non-keyed data queues. The RDQS0200 format to select messages when using keyed data queues.
- #20: In the following I’m going to demonstrate how to use data queues to establish a communication layer between two programs, and in that context discuss some of the problems and considerations involved in designing and developing data queue applications. Using DDM data queues, the programs communicating through data queues can reside on different servers if required. They can even be on different platforms. For example, both Visual Basic and Java offer programmable interfaces to data queues. Look up the links at the end of this article for more information about DDM data queues as well as VB and Java data queue support.
- #21: The receiving program will already have been activated at the point at which it is waiting on the data queue for messages to arrive. The call level has been established, initiation activities performed, and the data queue has been located and authority has been verified. More than on e process or job can be waiting on the same inbound data queue, so that as many messages can be processed in parallel, as server jobs have been submitted. The sending job includes a reply key to uniquely identify the request, and that key is used when the request processor returns the reply message to an outbound keyed data queue. Specifying that reply key on the receive data queue API call ensures that the receiving job gets the correct reply from the outbound data queue. Using a keyed inbound data queue, you can have different server jobs waiting against the same data queue for different types of request to process. One client could issue three different request messages, have them processed in parallel by three different server jobs, and then continue processing when all reply messages have been returned. One request could also be distributed to different applications or servers by sending the same request to different data queues and then having the reply messages consolidated into a single reply upon return. I should add to point number one that if you are chasing the last milliseconds, only user queues are more efficient in performing queue services than data queues. This speed, however, has its price in that user queues are a bit more complex and difficult to program. At the end of this presentation I provide a link to an article that explains and demonstrates how to program user queues – just in case you're interested in pursuing or evaluating that option.
- #22: Message Format: What kind of formatting should be applied to the message? Message Extensibility: How should we handle changes in or additions to the message format? Message Size Limit: Some message types are by nature limitless in size. For example, when embedded lists or textual descriptions are involved, it might be impossible or senseless to define the maximum message length to cover the longest imaginable message length. Error and Event Communication: How should error and informational messages be communicated within the data protocol, to ensure a correct and helpful dialogue with the application user? Language Support: Do you need to support more than one language within the data protocol? Exception Tolerance: It is very important to include in your application design an exception trap mechanism that catches — and communicates back — unforeseen errors. Debug Facility and Application Monitoring: How do you provide for daily monitoring of the application and investigation of problems and errors?
- #23: If you stick with data structures, your design should allow for varying-length subfields, to avoid space constraint problems. To achieve this objective, keep a fixed-length part of the data structure, which includes offset and length fields defining the location and length of the varying-length subfields. This concept is similar to what is used by many of the APIs that return information in a receiver variable defined by a data structure.
- #24: The Data protocol identifies the exchange format in general and the same value is used for all types of data exchange within the application. If the same data queue is used for different servers this value determines the server to which the request is routed. The Transaction ID uniquely identifies each inbound transaction and also ties the request and the reply message together. The Request ID defines the type of request that is performed by the client and in turn routes the request to the appropriate server function in charge of handling this type of request. The Message format specifies the layout and content of the message exchanged and allows you to vary both elements for the same request ID, depending on the actual circumstances and change in requirements.
- #25: The request message uses the number-of-list-entries field to indicate how many list entries are needed to build one page, and it uses the list-offset field to define where the list should start. If the latter field is empty, the list begins with the first entry. The response message uses the number-of-list-entries field to indicate how many entries are returned, and it uses the list-offset field to indicate whether there are more entries to request. If this field is empty, the list is complete; otherwise, the value is to be used in the subsequent request message to retrieve the next page, thereby ensuring that this page begins at the correct list position. Whereas these methods of preserving information between client and server job are stateless by nature, other methods involve establishing this persistence by means of a stateful message dialogue, typically by assigning and exchanging a session ID or session handle. The session ID is then used to identify the storage location needed to preserve the persistent information. This could be a record in a file, a data area, a user space, a user index, or something similar. If such a design is chosen, it is important to include a mechanism in the protocol to signal when the use of the session ID has completed, so that the associated allocated storage can be released — and the session ID safely reused.
- #26: In general, it is much easier and straight forward to add support for error and event communication as well as language considerations up front, as opposed to establishing such mechanisms further down the road. Again the time spent early in the project getting the application design and the individual components right before embarking on the implementation path clearly offers a substantial pay-back.
- #27: A useful approach to covering this angle is to develop a general model for the catching, registration and communication of this type of event for your application in its entirety. Once the model is devised and facilitated it’s simple to apply to your modules and programs as you move along, and ensures and supports a consistent and well documented behavior of your application. This also makes investigation and debugging of incidents at a later point in time much easier and faster to perform.
- #28: To sum up all these considerations, let me continue with the practical implementation in a sample application context. In a real-life business application, the data exchange would include customer, product, pricing, availability, order, accounting, payment, and many other types of information and transactions, but to avoid the problem of generating test data, for this purpose I've simply chosen to use some information already available on your system to demonstrate a data queue–driven application: The TCP/IP server start information in the system file QATOCSTART. Your part of this demonstration is therefore to envision all the types of information being distributed and exchanged in real business applications of similar design and architecture. For this purpose, I've created the Display TCP/IP Servers (DSPTCPSVR) command, which displays a list of all or a subset of the TCP/IP servers available on your system. Using the display option in the list panel, you can further display all the selected server's start attributes from the QATOCSTART file.
- #29: Using the QATOCSTART file layout, I then continue the design efforts by deciding which file fields to include in the TCP/IP server list. The server name, being the primary key as well, is a good start. The server type and autostart information completes the list. The remaining fields are then included in the full server attribute message.
- #30: The next step is to decide what information my request and response message headers should include and, as part of that consideration, how the server list data exchange should work.
- #34: Given the limited number of file records and the limited amount of server information, simply returning all server information in the list would be no problem. The reason for not doing that is of course the intention of demonstrating how to build the two different types of data exchange. To complete the picture, I have also created the following two examples of log files to capture errors and events as well as the transaction dialogue:
- #37: Now it's time to have a look at the data queue setup. For this application, there is a separate data queue for inbound and outbound transactions, respectively. Separating the data flows makes great sense for a number of reasons, including the above mentioned.
- #38: Both commands are part of the CL program to build the sample data queue application found following the link below: http://www.pentontech.com/IBMContent/Documents/article/54098_170_DataQueue5.zip
- #39: Here’s a list of all sources involved in this data queue sample application. To create all the above objects, copy all sources to their respective source files in your library, then compile and run CBX169M. Compilation instructions are found in the source headers as well.
- #41: To get a closer look at the transaction dialogue and how the various parts of the sample application play together, you could also step through the programs as they run in a source debug session. This concludes the program-to-program data queue communication sample application. Besides program-to-program communication, data queues are also employed by the IBM System i operating system for system-to-program communication. I’ve gathered a list of such uses as well as links to programming examples demonstrating how to take advantage of this facility.
- #42: Data Queue CL commands source file specifications: CBX165 -- Display Data Queue Description - CPP CBX165H -- Display Data Queue Description - Help CBX165P -- Display Data Queue Description - Panel Group CBX165V -- Display Data Queue Description - VCP CBX165X -- Display Data Queue Description CBX1652H -- Clear Data Queue - Help CBX1652X -- Clear Data Queue CBX1653H -- Send Data Queue Entry - Help CBX1653X -- Send Data Queue Entry CBX165M -- Data Queue Commands - Build commands CBX166 -- Copy Data Queue Description - CPP CBX166H -- Copy Data Queue Description - Help CBX166O -- Copy Data Queue Description - POP CBX166V -- Copy Data Queue Description - VCP CBX166X -- Copy Data Queue Description CBX166M -- Copy Data Queue Description - Build command CBX167 -- Display Data Queue Entries - CPP CBX167E -- Display Data Queue Entries - UIM General Exit Program CBX167H -- Display Data Queue Entries - Help CBX167L -- Display Data Queue Entries - UIM List Exit Program CBX167P -- Display Data Queue Entries - Panel Group CBX167V -- Display Data Queue Entries - VCP CBX167X -- Display Data Queue Entries CBX167M -- Display Data Queue Entries - Build command CBX168 -- Work with Data Queues - CCP CBX168E -- Work with Data Queues - UIM Exit Program CBX168H -- Work with Data Queues - Help CBX168P -- Work with Data Queues - Panel Group CBX168V -- Work with Data Queues - VCP CBX168X -- Work with Data Queues CBX168M -- Work with Data Queues - Build command
- #43: Running the command WRKDTAQ2 DTAQ(*ALL/Q*) shows the above list panel on my system. For each listed data queue you have the option of running the SNDDTAQE, CPYDTAQD, DLTDTAQ, DSPDTAQD, DSPDTAQE, CHGOBJD and CLRDTAQ commands.
- #44: IBM documentation for data queue support: Use Data Queues: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/rbam6/qcomm.htm Examples: Using Data Queues or User Queues: http://publib.boulder.ibm.com/infocenter/iseries/v5r4/topic/apiref/ExTaskDataque.htm ICFF & Display File — Example Using a Data Queue: http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/apis/apiexusque.htm Data Queue Support for Spooled Files: http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/rzalu/rzaludataq.htm Job Notification Exit Point: http://publib.boulder.ibm.com/infocenter/iseries/v5r3/topic/apis/xjobntfy.htm
- #45: Programming examples of data queue usage: Mystifying Timeout Messages — Tech Corner, July 1996: http://www.systeminetwork.com/article.cfm?id=1539 INVITE a Timeout — Tech Corner, May 1996: http://www.systeminetwork.com/article.cfm?id=1592 Automatic Spool Rerouting — March 31, 2004: http://www.systeminetwork.com/article.cfm?id=18333 Automatic Notification When Spool Files Created — March 17, 2004: http://www.systeminetwork.com/article.cfm?id=18244 Realtime Job Monitor — August 19, 2004: http://www.systeminetwork.com/article.cfm?id=19252 Synchronizing Passwords Between Systems — July 21, 2004: http://www.systeminetwork.com/article.cfm?id=18986