Big data technologies with Case Study Finance and Healthcare
1 like2,427 views
The document discusses the concept of big data characterized by the 5 V's: volume, variety, velocity, variability, and value, highlighting its significance across various platforms like YouTube, Facebook, and Twitter. It explains the shift from traditional relational database management systems (DBMS) to NoSQL databases that handle unstructured data more efficiently, along with technologies such as Hadoop and data science tools like R and Pig for data analysis. Lastly, it emphasizes the importance of big data in sectors like retail and finance, where real-time data analysis is crucial for making informed decisions.
Big data technologies with Case Study Finance and Healthcare
- 1. The Term Bigdata stems from Characterisized by 5V: Volume: Large Volume of data Velocity: amount of data per seconds Variability: level of unintentional modification affecting data Quality throughout lifecycle of data. Value: Value derived from data. Variety: large range of data which is received from video , audio, text, image.
- 2. Sources Example by 5V. Volume: Youtube, large volume of video feeds received and maintained at many video sites like youtube, vimeo etc… Variety: Large variety of data text, audio, video, images, received in sites like facebook, twitter, other social media platforms. Velocity: Speed at which data is received in sites like twitter, facebook (1 billion people all feeding there data on one site)
- 3. Batch Processing Vs Real Time processing Batch Jobs run at particular time of day like Nightly jobs or morning jobs which depends on slack time When server has less load. But people now want to see the Status like in transportation when bus is arriving on particular stand in real time. Or in Retail as soon they update there status the require real time advertisements. This is shaping move towards Big data.
- 4. Problems differentiated by 5V. Velocity: With large volume of data received and quick turn around latency required to reflect the data fed at facebook then Can it be managed by regular DBMS? DBMS- maintains ACID properties & have lots of constraints like primary, foreign keys, check constraints etc.. with quick turnaround or short latency required these constraints add up processing time and volume required for storage. So all of these sites have there own File based storage DBMS like systems with does not have these constraints. All data is maintained in files, id assigned to files are indexed and regularly moved (these are publically know open sourced databases like Cassandra developed by facebook, BigTable by Google, etc…) Most of this databases are popularly Categorized as NoSQL databases.
- 5. Technology Company Open Sourced On Cassandra DataStax Apache Cassandra used by Facebook , Linkedin , Twitter BigTable Google Google BigTable Apache HBase Apache HBase ( used by many companies most popular) MongoDB MongoDB Inc. Apache (written on C++,Erlang,C) Couchbase CouchBase Inc Apache (written on Erlang)
- 6. Category No SQL database Column Oriented Accumulo, Cassandra, Hbase. Document Clusterpoint,Couchdb, Couchbase, MarkLogic, MongoDB Key-Value Dynamo, FoundationDB, MemcacheDB, Redis, Riak, FairCom c- treeACE Graph Allegro, Neo4J, OrientDB, Virtuoso, Stardog - Column Oriented DB store database store Values in Column By Column rather in other RDBMS row by row. - It leads to better Compression Of data and hence less space required to store DB. - There are Still higher Compression can be achieved when used Probabilistic Databases. - Similarly Document oriented Store and arrange data in form of documents. - Key-Value store Data in form of collection of Key-value pairs. Allowing add, insert, delete to key-value pairs. - Graph Databases: Every Element is direct pointer to its adjacent hence no- lookup required.
- 7. Go through the link below: http://sandyclassic.wordpress.com/2013/07/02/data- warehousing-business-intelligence-and-cloud- computing
- 8. As we know now Bigdata is solving problems of 5V like the huge (V)olume of storage required for video sites like youtube. Etc. It’s changing how We perceive and Visualize or Analyze data like HBase used for data storage, Mahout of used to run analytics and find patterns. These databases have variety of data which require different kind of processing cannot be achieved by traditional RDBMS based products. Example link below: http://sandyclassic.wordpress.com/2013/06/18/gini- coefficient-of-economics-and-roc-curve-machine- learning/
- 9. Map-Reduce Algorithm was starting point of All we see in BigData created by Google researcher. Mapper divides work into multiple parallel task, sorts within queue and filters into queue of say 1 queue for each name. Reducer Component Aggregates data or summarizes from multiple units.
- 11. So Since data is mostly unstructured the best way to analyze unstructured data is using Analytics here Comes New Career Called Data Scientist. Skill Set Required for Data Scientist: Mathematics (mostly statistics), Computer Science, Domain like Sociology (like Social Media Analysis),
- 14. One application of Bigdata has been to gather feedback about product from social media. Here is Sample project Report below How and what tools can be used to Analyze social media. http://www.slideshare.net/SandeepSharma65/social- media-analysis-project
- 15. Hadoop allows to distribute load among many clusters. There can be Database clusters, OS clusters, Application Web server level clustering But here we are dealing with OS like Distributed File System(DFS). Hadoop DFS (HDFS) File system developed by yahoo Competes with BigTable of Google providing quick storage and retrieval of data in form of files used by many social media platforms.
- 16. ‘R’ was open source Statistical Analysis language having Statistical Constructs available used for Analysis of data. Java data mining API, .Net data mining API , python libraries are used to mine and understand trends in Data. PIG is another Apache Hadoop based system used provide High level language for analyzing large data sets.
- 17. Data Science http://thedatascience.wordpress.com/ Big Data :http://thebigdatatrends.wordpress.com Data Science Blog2: http://thedatascientistview.blogspot.ie/
- 18. Retail generates huge amount of data for product positioned on different shelf at store, replenishment level, reorder level, merchandising, assortment planning all this data most of it usually structured Since lots of system is Automated but there are lots of forms, customer feedback, planning data analysis of mails other chat platforms. Large Warehouses of Retail store needs plan positioning and containers in Aisle. Analyze trends from social media to find customer preferences for products and offers. Retail Innovation read: http://sandyclassic.wordpress.com/2013/10/26/retail- sector-innovations/
- 19. Retail uses lots of Sensors for tracking items with warehouse and inside Store. The Huge real time data (video , text and other forms) generated every milli- second from Sensors embedded across every store and warehouse Cannot be analyzed by any other medium better than in Hadoop or Bigdata based System.
- 20. Finance being Game of numbers huge data from Book of accounts, P&L, Balance sheets of etc accumulates of different business over a period of time But most books are Structured and hence the data. But Hadoop offers huge scalable clusters to quickly analyze structured data as well. Lots of social media data about interest for share or any instrument does get reflected in numbers. Spreadsheets are popular medium of analysis and other textual forms can be better analyzed if available over Hadoop like clusters for a kind of semi-structured data analysis.