WebSphere MQ Managed File Transfer V8 - Capitalware MQTC Conference

WWeebbSSpphheerree MMQQ
MMaannaaggeedd FFiillee TTrraannssffeerr
MMaatttthheeww WWhhiitteehheeaadd
WWeebbSSpphheerree MMQQ DDeevveellooppmmeenntt
mmwwhhiitteehheeaadd@@uukk..iibbmm..ccoomm
Capitalware's © Copyright IBM 2014 MQ Technical Conference v2.0.1.4

Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change
or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general
product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a
commitment, promise, or legal obligation to deliver any material, code or
functionality. Information about potential future products may not be incorporated
into any contract. The development, release, and timing of any future features or
functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM
benchmarks in a controlled environment. The actual throughput or performance
that any user will experience will vary depending upon many factors, including
considerations such as the amount of multiprogramming in the user’s job stream,
the I/O configuration, the storage configuration, and the workload processed.
Therefore, no assurance can be given that an individual user will achieve results
similar to those stated here.

Please Note
• IBM and the IBM logo are trademarks of International Business Machines Corporation,
registered in many jurisdictions. Other marks may be trademarks or registered trademarks
of their respective owners.
• Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft
Corporation in the United States, other countries, or both.
• Java and all Java-based trademarks and logos are trademarks or registered trademarks of
Oracle and/or its affiliates.
• Other company, product and service names may be trademarks, registered marks or service
marks of their respective owners.
• References in this publication to IBM products and services do not imply that IBM intends to
make them available in all countries in which IBM operates.

Agenda
 Common problems transferring file data
 Introduction to MQ Managed File Transfer
 IBM’s Managed File Transfer Portfolio
– Introducing IBM Sterling Commerce products
 Key MQ Managed File Transfer concepts
 Usage scenarios for MQ Managed File Transfer

IBM WebSphere MQ family
Portfolio of messaging capabilities optimized for a range
of connectivity challenges
WebSphere MQ
WebSphere MQ for z/OS
WebSphere MQ Managed File Transfer
WebSphere MQ Adv. Message Security
WebSphere MQ Telemetry
WebSphere MQ Low Latency
for mission critical data
for System z investment
for managed file transfer
for maximum security
for sensors and devices
for high speed delivery

Historical Roadmap
FTE V7
 Reliable, managed file transfer
 Remote management and audit
 Core platforms including z/OS
 Command line and GUI
interfaces
 File auditing across backbone
Q4 2008
WebSphere MQ V7.5
WebSphere MQ File Transfer Edition
became
WebSphere MQ Managed File Transfer
FTE V7.0.2
 Ability to bridge to FTP
networks
 iSeries® support
 Initial DataPower XB60
integration
 Security enhancements
Q2 2009
WMB 7.0.0.1
 FTE input and
output nodes
Q4 2009
FTE V7.0.1
 Archive transfer audit log to external DB
 Enhanced directory monitoring
 ANT XML Scripting of multi-step transfer
jobs
 Support for zLinux
 Enhanced z/OS Performance, Tape, GDG
4Q 2010
MQ MFT V8
Support for z/OS and IBM i
Z/OS enhancements
Monitoring enhancements
Transfer enhancements
2Q 2011
1Q 2014
2Q 2012
FTE V7.0.4
 Integration with existing IBM
Sterling Connect:Direct
networks
 Bridge to transfer files into out
of existing C:D networks
 Enhanced Explorer tooling with
visibility of file transfers sent
into/out of C:D networks
FTE V7.0.3
 Web browser (ad hoc) file transfers
 Convert payloads between files and messages
 Automatically start agents and DB logger on Windows
 End-to-end encryption using WebSphere MQ AMS
2Q 2014
IBM MQ MFT 7.5 +
IBM Sterling
Managed File
Transfer
 Bundles MQ MFT,
Connect:Direct and
Sterling Control
Center into a single
bundle

Shortcomings of Basic FTP
Limited
Reliability
Unreliable delivery – Lacking
checkpoint restart – Files can
be lost
Transfers can terminate
without notification or any
record – corrupt or partial files
can be accidentally used
File data can be unusable
after transfer – lack of
Character Set conversion
Limited
Flexibility
Limited
Security
Often usernames and
passwords are sent with file –
as plain text!
Privacy, authentication and
encryption often not be
available
Non-repudiation often lacking
Limited visibility
and traceability
Transfers cannot be monitored and managed
centrally or remotely
Logging capabilities may be limited and may only
record transfers between directly connected systems
Cannot track the entire journey of files – not just from
one machine to the next but from the start of its
journey to its final destination
Changes to file transfers often require updates to many
ftp scripts that are typically scattered across machines
and require platform-specific skills to alter
All resources usually have to be available concurrently
Often only one ftp transfer can run at a time
Typically transfers cannot be prioritized

What is WebSphere MQ Managed File Transfer?
……
WebSphere MQ Managed File Transfer
A B C X Y Z
 Auditable Full logging and auditing of file transfers + archive audit data to a database
 Reliable Checkpoint restart. Exploits solid reliability of WebSphere MQ
 Secure Protects file data in transit using SSL. Provides end-to-end encryption using AMS
 Automated Providing scheduling and file watching capabilities for event-driven transfers
 Centralized Provides centralized monitoring and deployment of file transfer activities
 Any file size Efficiently handles anything from bytes to terabytes
 Integrated Integrates with MB, WSRR, ITCAMs for Apps, DataPower + Connect:Direct
 Cost Effective
Reuses investment in WebSphere MQ. Wide range of support (inc. z/OS and IBM
i)

A consolidated transport for both files and messages
 Traditional approaches to file transfer
result in parallel infrastructures
– One for files – typically built on FTP
– One for application messaging – based on
WebSphere MQ, or similar
 High degree of duplication in creating
and maintaining the two
infrastructures
 Managed File Transfer reuses the MQ
network for managed file transfer and
yields:
– Operational savings and simplification
– Reduced administration effort
– Reduced skills requirements and maintenance
File Transfers Application
Messaging
Consolidated Transport
for Messages & Files

Agenda
You are here
 Usage scenarios for MQ Managed File Transfer

With Sterling Commerce, IBM offers comprehensive MFT Capabilities
Addressing multiple use cases and scenarios for both internal and multi-enterprise
file transfer
 WebSphere MQ Managed File Transfer provides file
transfer optimized for data delivery across WebSphere
MQ networks
 Sterling Connect Direct provides peer-to-peer file
transfer optimized for data delivery within and between
enterprises across Connect:Direct protocol
 Sterling File Gateway provides trading partner on-boarding,
broad protocol support, management and
visibility
 For comprehensive file transfer needs IBM provides
integration between WebSphere MQ Managed File
Transfer, Sterling Connect:Direct, and Sterling File
Gateway

Components of a typical WMQ MFT network
 Agents
Applications exchanging file data
– The endpoints for managed file transfer
operations
 Commands
– Send instructions to agents
 Log database or file
– A historical record of file transfers
 Coordination queue
manager
– Gathers together file transfer events
Agent Agent Agent
WebSphere MQ
“Coordination”
Queue Manager
Commands Log database
or file

Agent Agent Agent
WebSphere MQ
“Coordination”
Queue Manager
Log database
or file
Agents
 Act as the end points for file transfers
 Long running MQ applications that
transfer files by splitting them into
MQ messages
– Efficient transfer protocol avoids
excessive use of MQ log space or
messages building up on queues
 Multi-threaded file transfers
– Can both send and receive
multiple files at the same time
 Generate a log of file transfer
activities which is sent to the
“coordination queue manager”
– This can be used for audit
purposes
 Associated with one particular queue
manager (either v6 or v7)
– Agent state on queues
Commands

Agent Agent Agent
WebSphere MQ
“Coordination”
Queue Manager
Log database
or file
Commands
 Send instructions to agents and
display information about agent
configuration
– Via MQ messages
 Many implementations of
commands:
– MQ Explorer plug-in
– Command line programs
– Open scripting language
– JCL
– Documented interface to
program to
Commands

Log Database & File
 Keeps a historical account of transfers
that have taken place
– Who, where, when… etc.
 Implemented by the ‘logger’ component
which connects to the coordination
queue manager
– Stand alone application
- Can log to database or file
– Or JEE application
- Can log to database only
 Queryable via Web Gateway
– Also a documented interface
Agent Agent Agent
WebSphere MQ
“Coordination”
Queue Manager
Commands Log database
or file

Agent Agent Agent
WebSphere MQ
“Coordination”
Queue Manager
Log database
or file
Coordination Queue Manager
 Gathers together information
about events in the file transfer
network
 Not a single point of failure
– Can be made highly available
– Messages stored + forwarded
 MQ v7 publish / subscribe
– Allows multiple log databases,
command installs
– Documented interface
Commands

Granular Access Control
Access control to agent capabilities can be broken down into steps:
Determine a user’s identity
– (MQMD user ID of request message)
Work out what action is being taken
– (Parse payload of request message)
Map what they are trying to do to one (or more) FTE authorities
– (Simple ‘look-up’ table in the code)
Determine the agent’s identity
– (MQMD user ID of messages sent by the agent)
Check to see if the identities have the appropriate authorities
– (Map FTE authority to MQ authority and see if the user is authorized)
Permit or deny the action
– (Either carry on as normal, or fail the request)

Example authority checks before transfer occurs
Source agent Destination agent
Agent1
Request message
MQMD user: bob




Direction file data will move in
User: accounts User: payroll
Checks that occur before the transfer starts:
– Does ‘bob’ have ‘transfer source’ authority?
• (i.e. can bob move files off agent1?)
– Does ‘accounts’ have ‘agent source’ authority?
Agent2
• (i.e. is ‘agent2’ going to allow ‘agent1’ to transfer files to it?)
– Does ‘bob’ have ‘transfer destination’ authority?
• (i.e. can bob move files onto agent2?)
– Does ‘payroll’ have ‘agent destination’ authority?
• (i.e. is ‘agent1’ going to allow ‘agent2’ to receive files from it?)
Checks 1+4 happen at the source agent, and 2+3 at destination agent

Mapping FTE Authorities to MQ Authorities
We have talked about FTE authorities (like ‘transfer source’ or ‘schedule’)
– But how does an administrator configure these?
FTE authorities are mapped to MQ authorities on specific MQ objects
– E.g. the FTE ‘administration’ authority maps to the MQ ‘browse’ authority on queue
‘SYSTEM.FTE.AUTHADM1.agentname’.
The same model used for Distributed platforms (via the OAM) and for z/OS (via SAF)
Queue names:
SYSTEM.FTE.AUTHADM1.agent_name
SYSTEM.FTE.AUTHAGT1.agent_name
SYSTEM.FTE.AUTHMON1.agent_name
SYSTEM.FTE.AUTHOPS1.agent_name
SYSTEM.FTE.AUTHSCH1.agent_name
SYSTEM.FTE.AUTHTRN1.agent_name

Agenda
 Usage scenarios for MQ Managed File Transfer You are here

Example usage of monitoring + program execution
Existing
Application
1. Application writes
file to file system
3. MFT transports file
to destination
WMQ
MFT
Agent
5. MFT can also start another
application to process the file
WMQ
MFT
Agent
Existing
Application
*tap*
2. Agent monitors file
system, spots arrival
of file and based on
rules, transfers the file
4. At destination MQ MFT
writes file to file system

XML Scripting using Apache Ant
Step 1
Invoke a File Transfer
Step 2
If Step 1 completes Ok then
invoke program to process file
1
2
3
Step 3
If Step 1 fails then
send an email to the
Administrator

Protocol Bridging Agents
 Support for transferring files located on FTP and SFTP servers
– The source or destination for a transfer can be an FTP or an SFTP server
 Enables incremental modernization of FTP-based home-grown solutions
– Provides auditability of transfers across FTP/SFTP to central audit log
– Ensures reliability of transfers across FTP/SFTP with checkpoint restart
 Fully integrated into graphical, command line and XML scripting interfaces
– Just looks like another MFT agent…
Files exchanged between MFT and FTP/SFTP
Agent
WebSphere MQ
Audit
information
Agent Agent
Protocol
Bridge
Agent
FTP/
SFTP
FTP/
SFTP
Server
FTP/
SFTP
Client
FTP/
SFTP
Client
FTP/
SFTP
Client

WebSphere Message Broker Nodes
WMQ
MFT
Agent
WMQ
MFT
Agent
WMQ
MFT
Agent
 FTEInput node
Message Broker
Execution Group
Message Flow
Part of
WMB
7.0.0.1
Part of
WMB
7.0.0.1
FTEInput FTEOutput
WMQ
MFT
Agent
– Build flows that accepts file transfers from the WMQ MFT network
 FTEOutput node
– Build flows that are designed to send a file across a WMQ MFT network
 When WMQ MFT nodes are used in a flow an MFT agent is automatically started in the
Message Broker Execution Group

C:D
Node
Trading Partner
Integration with IBM Sterling Connect:Direct
Agent
WebSphere MQ
MFT
Audit
Agent Agent
C:D
Bridge
Agent
C:D
Node
C:D
Node
C:D
Node
C:D
Audit
Reference
Inside the MFT audit trail…
The audit information for each MFT transfer
references related C:D audit information
 The Connect:Direct Bridge capability
supports managed file transfers that
span MFT and C:D with a joined up
audit trail

Interoperation with DataPower B2B Appliance XB60
 Documented and tested configurations for integrating with DataPower Appliances
– WebSphere DataPower XB60 B2B Appliance – for B2B connectivity
– WebSphere DataPower IX50 Integration Appliance – for ESB connectivity
 Enables sending files to trading partners over a range of protocol transports
– via DataPower Appliances acting as B2B gateways
Multi-protocol transfers to B2B trading partners
Agent Agent HTTPS
XB60
Agent Agent
Internal Network DMZ
Internet Trading Partner
Company A
Agent
WebSphere MQ
SFTP
AS2
…etc
Company B
Company C
… etc

Securing file data with SSL and WMQ AMS
 WMQ MFT supports transport
level encryption using SSL
 Data is encrypted before it is
sent over a channel and
decrypted when it is received
WebSphere
MQ
Queue
Manager
WebSphere
MQ
Queue
Manager
svrconn
channel
sndr/rcvr
channels
Agent Agent
WebSphere
MQ
Queue
Manager
WebSphere
MQ
Queue
Manager
svrconn
channel
sndr/rcvr
channels
 When combined with WMQ
Advanced Message Security
– Allows file data to be encrypted
at the source system and only
decrypted when it reaches the
destination system
– Data is secure even when at
rest on a queue
Agent Agent

Staged migration to messaging
 Pain-point:
– Hard to migrate to an event driven architecture as lots of applications communicate
by transferring files
 Managed File Transfer Helps:
– Deliver files as message payloads and vice versa
– Monitor queues and transfer message payloads to files

Options for converting data between files and messages
One file to one message
WMQ
MFT
One file to a group of messages
One message to one file
WMQ
MFT
WMQ
MFT
A group of messages (or all messages on the queue) to one file
WMQ
MFT
 One file becomes one
message
 The file can be split based on:
– Size
– Binary delimiter
– Regular expression
 One message becomes one
file
 Optionally, a delimiter can be
inserted between each
message used to compose the
file

Monitoring queues for the arrival of messages
 The WMQ MFT agent can
monitor queues for the arrival
of messages, then perform an
action, such as transferring
the payload fro the messages
as a file (as per the previous
slide)
 Conditions that can be
monitored for:
– Queue not empty
– Complete group of messages
Remember we said MFT
can monitor for files arriving…
Existing
Application
WMQ
MFT
Agent
Well, it can also monitor for
messages arriving on a queue…
Existing
Application
WMQ
MFT
Agent

Web-based managed file transfers
 Pain-points:
– Difficult to mix human imitated file transfers with existing
infrastructure for machine-to-machine managed file transfer
– Managed file transfers to zero-install, small-footprint devices
 MQ Managed File Transfer Helps:
– A RESTful API for exchange files with an WMQ MFT network
– Example web 2.0 applications to use as a starting point
Agent
WebSphere MQ
Agent Agent
Agent Agent
HTTPS
HTTP
HTTPS
Web
Gateway

Enabling business users to upload files from a remote location
 In this example usage scenario the Web Gateway allows a business user to
upload a file (via the company web portal) to a back-end system where it can
be processed
Business user
on-site with
customer
Logs on to
company web portal
Report
Internet
Head Office
Web
Gateway
Back-end
System
WebSphere
MQ network
Report is
processed
and loaded
into database
File transported using HTTP File transported using WMQ MFT
1. The business user
logs onto the company
web portal using a web
browser and is
prompted to select a
file to upload
3. The Web
Gateway transfers
the file, using WMQ
MFT, to a back-end
system
4. At the
back-end system
WMQ MFT starts
a program to
process the data
from the file
2. The portal
uses the RESTful
API provided by
the Web Gateway
to upload the file
using HTTP

Enabling researchers to pick up the results of a batch process
 In this example usage scenario the Web Gateway is used to enable a researcher to pick
up files that have been produced (hours earlier) by batch processing at a back-end
system
WebSphere
MQ network
Results
Web
Gateway
File Space
a per user “in-tray” for files
Data Centre
Back-end
System
Processes
research data
during a 3am to 4am
batch processing window
WAN
Research
Lab
Logs into research
portal to collect results
File transported using WMQ MFT File transported using HTTP
1. A batch process
running at the data
centre produces a set of
results which it sends,
using WMQ MFT, to the
web gateway
3. The user logs
in to the research
portal using her web
browser and is shown
a list of files waiting
for her attention
4. The user
selects a file to
download and the
Web Gateway
transfers the file
to her computer
2. The Web
Gateway system
places the data into
a file space where it
awaits collection
by the user

Components used for uploading to a back-end system
Logical path for file data
Web Gateway System
JEE
Application
Server
Web
Gateway
Application
WMQ
Queue
Manager
MFT
Agent
MFT
Agent
Processed
WebSphere MQ Network
Coordination
Queue
Manager
Database
Back-end
System
Database Logger
RESTful
Web
Interface
HTTP(S)
Filesystem
Audit information
User
Web Browser
File
The database
logger could,
alternatively,
be run in the
JEE container

New features in MQ MFT V8
 Inlining file data with transfer handshake
– Improved performance for small file transfers
 More options on resource monitors
– Include meta-data in transfers
– Specify file list in trigger file
– Other related features – see InfoCenter for
complete set of new options
 Support for z/OS and IBM i
– MQ MFT 7.5 didn't support these platforms

Thank you! Questions?

Legal Disclaimer
• © IBM Corporation 2014. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you
will result in any specific sales, revenue growth or other results.

WebSphere MQ Managed File Transfer V8 - Capitalware MQTC Conference

More Related Content

What's hot (20)

Similar to WebSphere MQ Managed File Transfer V8 - Capitalware MQTC Conference (20)

More from matthew1001 (12)

Recently uploaded (20)

WebSphere MQ Managed File Transfer V8 - Capitalware MQTC Conference