![• Timer watches the “/download”
directory for incoming payment files.
• When a file arrives main.script.001
generates a new script under
/job/main.script.001.tmp
• main.script.001 then copies the
incoming file from the “/download”
directory to the /import directory
• main.script.001 runs the newly
generated script, which will then
process the incoming file
• main.script.001.tmp uploads the
incoming file (and its transactions) into
the database
Before…Use Case - Load Transaction Data
9All contents Copyright © 2013, MuleSoft Inc.
main.script.001
/job
main.script.001.tmp
DB
/download
[Original File]
/import
[Duplicate File]
Timer
flow: payment.file.script](https://image.slidesharecdn.com/muleconferencev3-140618221441-phpapp01/85/High-Volume-Payments-using-Mule-9-320.jpg)
![• main.script.001 watches the “/download” directory for
incoming files
• main.script.001 copies any incoming files to the “/import” for
processing. It converts these file into a different format and
puts them into the same directory
• main.script.001 generates the 1st ack file and the 1st ack
email in the “/export” directory
• cntry.script.002 watches the “/export” directory for 1st ack
files
• cntry.script.002 copies the 1st ack files into the “/ack1”
directory
• cntry.script.002 sends the 1st ack files to down system via
NDM
• cntry.script.003 watches the “/export” directory for 1st ack
emails
• cntry.script.003 copies the 1st ack email to the “/email”
directory
• cntry.script.001 watches the “/import” for new files
• cntry.script.001 uploads any files (and its transactions) into
the database
Before…Use Case - Load Transaction Data / Ack
10All contents Copyright © 2013, MuleSoft Inc.
main.script.001
/download
[Original File]
/import
[Duplicate]
/export
/ack1/ndm /email
cntry.script.001
DB
cntry.script.002
cntry.script.003
flow: payment.file.ack](https://image.slidesharecdn.com/muleconferencev3-140618221441-phpapp01/85/High-Volume-Payments-using-Mule-10-320.jpg)
High Volume Payments using Mule
- 1. Why Mule? Use Case / User Stories
- 2. 2 All contents Copyright © 2013, MuleSoft Inc. File Overview Java Problem Statement Java Challenges Why Mule? VM The Solution Java Before & After Java Lesions Learned Q&A flow: Agenda flow: conf_Solution_Architecture
- 3. Overview 3 All contents Copyright © 2013, MuleSoft Inc. http://www.linkedin.com/in/adhishpendharkar
- 4. Problem Statement - Numbers / Numbers….. 4 All contents Copyright © 2013, MuleSoft Inc. i Be rational π Get Real.. Capacity in Transactions per Day Throughput in Transactions per Second (TPS) Location 2011 Peak Volume Growth 2012 Capacity 2012 Capacity + Headroom Aggressive Cut-off & STP + Headroom in Aggressive Cut-off & STP Japan 685,000 2x 1,000,000 1,500,000 556 834 China 9,800 4x 34,000 51,000 19 29 Taiwan 531 51x 26,821 40,232 15 23 Hong Kong 11,714 12x 140,568 210,852 79 118 Korea 3,846 16x 58,630 87,945 33 49 India 167,000 3x 500,000 750,000 278 417 Singapore 10,373 17x 167,000 250,500 93 140 Philippines 1,919 27x 50,100 75,150 28 42 Indonesia 7,922 17x 127,540 191,310 71 107 Malaysia 1,227 41x 50,100 75,150 28 42 Thailand 2,945 18x 50,100 75,150 28 42 Vietnam 323 156x 50,100 75,150 28 42 Australia 142,248 7x 995,736 1,493,604 554 830 New Zealand 6,416 20x 128,320 192,480 72 107
- 5. Problem Statement - Non Functional 5All contents Copyright © 2013, MuleSoft Inc. • No single point of failure. • RTO within a data center of 30 minutes. • RTO to Disaster Recovery site within 2 hours • Resilience is designed in, built for, and verified through failover testing. • Compute capacity to support current volumes with 100% headroom at the time of go-live. • Application and system architecture should allow future scalability of compute capacity to support future business forecasted peak daily volumes with 100% headroom without the need for application change • Storage capacity to support current volume with 50% headroom at launch • Application and system architecture should allow future scalability of storage capacity to support future business forecasted peak daily volume with 50% headroom without the need for application change.
- 6. • Platform / OS / Infrastructure Independent. • Average CPU utilization of the application should not exceed 2%. • Memory usage of the application should not exceed 2% • Database connections used by the application should not exceed 2% • There should be no explicit garbage collection unless approved by architect. • No orphan or hang sessions or processes • Response times of the application in terms of application loading, screen open & output on screen (2 seconds as threshold) • Number of out-of process creations should not exceed 5% Problem Statement - Functional 6All contents Copyright © 2013, MuleSoft Inc.
- 7. • 2 Physical Solaris Machines humming for number of years. • Monolithic Java Applications written 10+ years back. • 5000+ AutoSys Jobs and same number of scripts • 1,214,699 Lines of code • Code Complexity of the range of 7.8 ~ 36.2 • Hardware Infrastructure availability • H/A, Cluster, Failover capabilities • High Speed Storage on Virtualized Platform. • Regulatory In country requirements • Security / Audit and AML requirements • Cost of Developers / SME’s and time to market. • Running costs / support - Commodity Hardware - User Support / Communities - Hello Worlds!!! and lots of examples on World’s Wonderful Webites (www) Challenges 7All contents Copyright © 2013, MuleSoft Inc.
- 8. Why Mule? 8 Hold on your Horses, lets visit this topic in short gallop.
- 9. • Timer watches the “/download” directory for incoming payment files. • When a file arrives main.script.001 generates a new script under /job/main.script.001.tmp • main.script.001 then copies the incoming file from the “/download” directory to the /import directory • main.script.001 runs the newly generated script, which will then process the incoming file • main.script.001.tmp uploads the incoming file (and its transactions) into the database Before…Use Case - Load Transaction Data 9All contents Copyright © 2013, MuleSoft Inc. main.script.001 /job main.script.001.tmp DB /download [Original File] /import [Duplicate File] Timer flow: payment.file.script
- 10. • main.script.001 watches the “/download” directory for incoming files • main.script.001 copies any incoming files to the “/import” for processing. It converts these file into a different format and puts them into the same directory • main.script.001 generates the 1st ack file and the 1st ack email in the “/export” directory • cntry.script.002 watches the “/export” directory for 1st ack files • cntry.script.002 copies the 1st ack files into the “/ack1” directory • cntry.script.002 sends the 1st ack files to down system via NDM • cntry.script.003 watches the “/export” directory for 1st ack emails • cntry.script.003 copies the 1st ack email to the “/email” directory • cntry.script.001 watches the “/import” for new files • cntry.script.001 uploads any files (and its transactions) into the database Before…Use Case - Load Transaction Data / Ack 10All contents Copyright © 2013, MuleSoft Inc. main.script.001 /download [Original File] /import [Duplicate] /export /ack1/ndm /email cntry.script.001 DB cntry.script.002 cntry.script.003 flow: payment.file.ack
- 11. After.. Vendor 1 11 All contents Copyright © 2013, MuleSoft Inc.
- 12. After.. Vendor 2 12 All contents Copyright © 2013, MuleSoft Inc. Ring Buffer Service Bus Service Bus 2 Service Brokers Bus In Memory Data Architecture Cascade Ring Buffer DB Persistence
- 13. After.. In House Team (using Mule POC’s) 13 All contents Copyright © 2013, MuleSoft Inc. Validate Persist & Parse Validate 1st Ack Persist & Parse CSV File Flow XML File Flow Send Ack Send Ack (FileBase) Send Ack Send Ack (Email) Product Classification Validation Persist Txn Extensions Send Ack 2nd Ack Txn Ack
- 14. After.. Mule 14 All contents Copyright © 2013, MuleSoft Inc. File File Watcher Java File Validation Java File Persistence File Parse flow: csv_parser_flow Java File File Watcher Java File Parse FirstAck File Parse flow: iso20022_ack_flow Java Java Java File Persistence VM GIRO Txn Java Message Classification Java Message Validation Java Message Persist All Sec. Ack Msg flow: giro_txn_vm_flow Ctrl C Ctrl V
- 15. After.. How it looks on Infrastructure 15 All contents Copyright © 2013, MuleSoft Inc. • Active-Active-Active: Application can run across data center within the country or outside the country. • Active Data Replication across geo / metro and local capabilities using combination of SRDF and In Memory Site A Virtual Server Infrastructure Scale! VM VM VMVM Virtualized high-availability infrastructure Scale OS OS OSOS HA Seamless Flip-Flop via VMotion App Storage SAN Database Active-Active Applications Primary database across sites Phy OS Phy OS Oracle RAC Site B Virtual Server Infrastructure Scale! VM VM VMVM Scale OS OS OSOS HA App Storage SAN Database Phy OS Phy OS Oracle RAC Shared Data volume across sites DataGuard / GoldenGate Replication App Storage SAN App Storage SAN Shared Data volume across sites Reporting database across sites In Memory Architecture In Memory Architecture Cross Site Replication Physical HA Physical Server Infrastructure In Memory Data Caching Different Country / Data Center Physical HA
- 16. Lessons Learned 16 All contents Copyright © 2013, MuleSoft Inc. “Eight months ago I started working at MuleSoft as an intern on the SaaS ISV partner team. When I evaluate my experience at MuleSoft, a cliche yet valid saying comes to mind: Everything happens for a reason.” “I would recommend MuleSoft to any intelligent, dedicated individual. We work really hard here, but all that time you are surrounded by awesome people that make working a fun and rewarding experience. If you want to work with people who want you to succeed, while challenging yourself everyday — come work at MuleSoft” - Source http://themuleternship.wordpress.com/ by Moira Chambers.
- 17. Q&A 17All contents Copyright © 2013, MuleSoft Inc.
Editor's Notes
- #3: Overview: Me / Myself / Adhish (its all about me) Problem Statement: Numbers (business bat) / Non Functional / Functional Challenges: Weight Loss / Developer Debt (Sonar / SQUEL) Before & After : Vendor Solutions Vs. Mule/Inhouse
- #4: That’s me, yehey!!!
- #5: Numbers: This project started in 2011 as a 5 year transformation.
- #6: Current Volumes: This project was started in 2012 (planning in 2011) with 5 years to go live. Recover Time Objective: was twisted by business to Recover Time Operate. Future business: 2015 and beyond, as this was a 5 year plan, it started back in 2011. Storage: Played a critical aspect of the solution, we had to cut down on logging, implement other mechanism i.e. Splunk etc.
- #7: Average CPU: was based line on the older application which was in production. Memory: same as the older application Use of GC: … DB Connections / MQ / Qpid etc.
- #10: Talk about the Cash Business, file based, low value, large amount of data. Compound the problem with Country / Client / Time of transmission et`c.
- #11: Talk about the complex of multiple files / clean up / data files / security / regulation(s) etc.
- #17: Mule Shipped a trainer from Antwerp (Tom) to help us get the team in Hong Kong up and running, it was a 5 days intense course with hands on lab.