Hw09 Large Scale Transaction Analysis
Download as PPT, PDF9 likes2,162 views
1. Visa operates the world's largest payment network, VisaNet, which processes over 130 million authorizations per day across 28 million merchant locations. 2. Hadoop offers the potential to analyze Visa's large volumes of transaction data faster than currently possible. Visa is researching how to configure Hadoop solutions and integrate them securely within its existing financial systems. 3. Key questions include how to securely load data into Hadoop clusters, integrate security features like hardware security modules, and ensure solutions meet Visa's requirements of being fast, secure, flexible, reliable and scalable for enterprise use.
Hw09 Large Scale Transaction Analysis
- 1. Hadoop@Visa Joe Cunningham Technology Strategy and Innovation October 2, 2009
- 2. Agenda 1 Visa 2 Value-Added Information Products 3 Hadoop@Visa – research results VisaNet – an overview
- 3. Keeping the Brand Promise Zero Liability go Security Flexibility
- 4. Keeping the Brand Promise 3 Global Acceptance Reliability Zero Liability go Security Flexibility Guaranteed Payment
- 5. VisaNet Today VisaNet is the world’s largest, most advanced payments network. 28 million acceptance locations 130 million authorizations per day 1,600 endpoints Processes transactions in 1 second 16,300 financial institutions Less than 2 seconds unavailability per year Processes in 175 currencies 1.4 million ATMs fast secure flexible reliable scalable
- 6. 1 VisaNet – an overview 2 3 Value-Added Information Products Hadoop@Visa – research results Agenda
- 7. Visa Processing Architecture Security/Access Services Message File Web VisaNet Services Integration Common Data Infrastructure Authorization Clearing & Settlement Information Disputes Risk
- 8. Value-Added Information Products Information Services Client Portfolio Analysis Visa Incentive Network Accountholder Transaction Alerts Account Updater Tailored Rewards Risk Management Services Authentication Advanced Auth Verified by Visa Address Verification Account Monitoring Fraud Detection Compromised Accounts Consumer Alerts Encryption Dynamic Key Exchange Dynamic Cryptograms
- 9. 1 VisaNet – an overview 2 Value-Added Information Products 3 Hadoop@Visa – research results Agenda
- 10. Research Lab Setup Hadoop Systems VM System Relational Database Custom Analytic Stacks Encryption Processing Management Stack Hadoop #2 ~300Tb / 28 nodes Hadoop #1 ~40Tb / 42 node
- 11. Risk Product Use Case Create critical data model elements, such as keys and transaction statistics, which feed our real-time risk-scoring systems Hadoop Cluster Research Sample (Synthetic Transactions) 500 million distinct accounts 100 million transactions per day 200 bytes per transaction 2 years: ~ 73 billion transactions, ~ 36 TB One Month 13 minutes! Input: Transactions Merchant Category Country / Zip Output: Key & Statistics MCCZIP Key Statistics related to account, transaction type, approval, fraud, IP address, tx. counts
- 12. Financial Enterprise Fit Some key questions we are researching… What will the Hadoop Solution Stack(s) look like? File System Transaction Sample System Relational Back-End Analytics Processing Internal vs. External cloud How do I get data into a cloud in a secure way? How does HSM and security integration work in Hadoop? What are the missing pieces?
- 13. Why Hadoop@Visa? For our information products, we see promise in combining Visa’s data analytics capability with the power of Hadoop Analyze volumes of data with response times that are not possible today… … and apply analytic models to individual client, not just client segment But, need to answer questions relating to stack configuration and support… … and, eventually must fit with our financial enterprise paradigm: fast secure flexible reliable scalable
- 14. Keeping the Brand Promise Zero Liability go Security Flexibility
- 15. Keeping the Brand Promise 3 Global Acceptance Reliability Zero Liability go Security Flexibility Guaranteed Payment
Editor's Notes
- #6: Fast – On average, VisaNet fully processes transactions in approximately 1 second. Secure – VisaNet supports multiple defense layers to prevent network intrusions, protect against account fraud, and to make stolen card data unusable for criminalsincluding data encryption, sophisticated intrusion detection systems and neural network technology capable of analyzing each transaction in real-time. Over the last two years, Visa has spent more than $100 million on data security enhancements. Reliable – VisaNet runs multiple, parallel processing systems in highly redundant data centers. This ensures near 100 percent availability. In the past 15 years, VisaNet experienced less than two seconds average unavailability per year. State-of-the-art monitoring tools are in place to detect network availability and client-specific processing anomalies, such as high rates of decline. Scalable – VisaNet authorizes an average of 130 million transactions a day, totaling more than 92 billion authorization. On Dec. 21, 2008, Visa’s busiest day last year, VisaNet authorized more than 200 million transactions. Based on recent stress tests, VisaNet is capable of authorizing over a half billion transactions per day.