Dealing with Common Data Requirements in Your Enterprise
1 like596 views
The document outlines common data challenges organizations face, including data integration, transformation, and analytics in modern data ecosystems. It emphasizes the significance of exposing data as services, managing secure access, and leveraging analytics for informed decision-making, including batch, real-time, and predictive analytics. Additionally, it discusses the importance of data visualization for improved data comprehension and interaction.
Dealing with Common Data Requirements in Your Enterprise
- 1. Dealing with Common Data Requirements in your Enterprise Nipun Suwandaratna Senior Solutions Engineer - WSO2 WSO2 Solution Architecture Best Practices Webinar Series - 2016
- 2. Agenda ● Organizational Data ● Common Data Challenges of Modern Organizations ● Integrating with Different Messaging Infrastructures ● Data Services ● Data Analytics & Visualization ● High Availability ● Q&A
- 5. Modern Data Ecosystem External Systems External Users
- 6. Organizational Data ● Master data Eg: Customer data, employee records, Supplier details, Product related data etc. ● Transactional data The data that master data participates in… transactions, discounts on bills etc. (changes constantly) ● Meta-data Data about data
- 7. Common Data Challenges Organizations Face ● Work with multiple Data Transports and Data Formats ● Data Transformation and Validation ● Exposing data as services ● Secure and managed data access ● Federated data stores ● Data/Entity Aggregation ● Data Analytics ● Visualization of Data
- 8. Data Transports & Formats Formats of data, their storage and transport mechanisms vary among different systems ● Transports: HTTP, HTTPS, FTPS, SFTP, TCP, UDP, WebSocket, POP, IMAP, SMTP, JMS, AMQP, MQTT ● Formats & protocols: JSON, XML, SOAP, WS-*, HTML, EDI, HL7,Text, JPEG, MP4, binary formats
- 9. Integration
- 10. Integrating with Messaging Infrastructures
- 11. Message Transformation ● Protocol and Format conversion and Message Translation ○ eg: SOAP to REST and XML to JSON and translate the output from one system to match the input format required by the other system ● Enrich Content ○ eg: Add or remove data fields; may require accessing a separate data source ● Wrap Content ○ eg: Include additional message header fields or encryption source to query required data ● Data Validation ○ eg: Validate input data against a schema
- 13. Message Transformation Example Protocol / Content-Type Conversion
- 14. Data Services
- 15. Exposing Data-As-Services Why ? ● Decouple data from the infrastructure and the data sources and expose them through standard web services interfaces. ● Ability to incorporate multiple data sources/entities into a single data model (Data Federation)
- 16. Secure & Managed Data Access ● Transport and Application level security ● Authentication, authorization, confidentiality, integrity and encryption - with HTTP(S) Basic Auth, WS-Security, WS-Trust, WS-SecureConversation, WS-Policy, WS-Policy Attachment and WS-SecurityPolicy ● Authorization deals with defining who can access what ● Role based access control ● Fine-grained authorization with XACML ● Throttling access to data
- 17. Federated Data Stores ● Expose data from multiple data sources through a single service ● Facilitates entity aggregation
- 19. WSO2 Data Services Features Ref: http://wso2.com/products/data-services-server/
- 20. Analytics
- 21. Data Analytics ● Batch Analytics Analyze a set of data collected over a period of time. Suitable for high volumes of data. ● Real-Time Analytics Continuous processing of input data in real time. Suitable for critical systems where immediate actions is required e.g: Flight radar systems ● Interactive Analytics Obtaining fast results on indexed data by executing ad-hoc queries ● Predictive Analytics Predict future events by analyzing historical and current data
- 22. Big Data What is Big data ? “Big data is a term for data sets that are so large or complex that traditional data processing applications are inadequate to deal with them” - Ref: Wikipedia
- 23. Big Data Analysis Why ? ● Make informed Business decisions - make decisions based on patterns emerging from analyzing historic data ● Improve customer experience - discover customer preferences, purchasing patterns and present the most relevant data ● Process Improvements - identify areas of the business process that needs improvement
- 24. Big Data Analysis Example Better customer experience in airline seat reservation/allocation img ref: http://staticcontent.transat.com/airtransat/infovoyageurs/content/EN/seating-plan-a310-300(1).png
- 25. Real Time Analytics ● Identify most meaningful events within an event cloud ● Analyze the impact ● Acts on them in real time
- 26. Real Time Analytics Example City Transport Control System - Analyzing traffic, monitor movement of busses, generate alerts based on traffic, speed & route
- 27. Predictive Analytics & Machine Learning Approaches: ● Machine Learning Machine learning is the science of getting computers to act without being explicitly programmed - http://online.stanford.edu/ ● Other approaches such as statistical modeling
- 28. Predictive Analytics Example e-Commerce sites use predictive analytics to suggest the most relevant merchandize, increasing sales opportunity
- 31. Data Visualization Contd. What is Data Visualization ? ● View data in a constructive and comprehensible format ● Facilitates interaction with data - drill into the data for visual analysis ● Detect patterns (e.g: sales patterns) that may go un-noticed unless data is properly visualized
- 33. High Availability of Data
- 34. CONTACT US !