Redefine Big Data
2 likes430 views
The documents discuss the evolution of DevOps and big data analytics. They describe how development and operations teams used to work in silos but now must work more collaboratively. There is a movement toward breaking down barriers between teams to increase speed and release rates. With new tools and infrastructure, teams can better align and integrate work. Big data provides an opportunity for DevOps to further mature by treating data as a shared resource and asset to be managed holistically.
Redefine Big Data
- 1. 1© Copyright 2015 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology
- 2. 2© Copyright 2015 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology The Evolution of DevOps
- 3. 3© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology The Olden Days of working in vacuums In space, no one can hear you scream… It doesn’t appear to be working correctly Application is complete – here you go
- 4. 4© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology Commonly Accepted Analytics Platform Or why dev and ops just can’t get along in this new space =
- 5. 5© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology Operations fixes things that are broken Lots of dedicated servers is very broken – fixed it Compute Hardware Storage Virtualization Administration UI Virtual Servers and Services ApplicationAdministrators Application Storage Managers Application Self-Service UI Users
- 6. 6© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology DevOps Movement • Use of agile and other development processes and methodologies • Demand for an increased rate of production releases from application and business unit stakeholders • Wide availability of virtualized and cloud infrastructure from internal and external providers • Increased usage of data center automation and configuration management tools Better alignment of Development and Operations
- 7. 7© Copyright 2015 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology Big data is where the rubber meets the road • Hadoop and emerging analytics is the opportunity DevOps has been waiting for to mature • As stewards of Hadoop/HDFS and emerging tools we have a responsibility to think ahead • Its not OK to revert to outdated operational models just because it’s easier for us or because its always been done that way Monumental shifts are forcing Dev and Ops to come to terms
- 8. 8© Copyright 2015 EMC Corporation. All rights reserved. DELIVER A SEAMLESS EXPERIENCE FOR EVERYONE: DATA IS A TEAM SPORT BUSINESS USERS DATA PROGRAMMERS INFRASTRUCTURE DEVELOPERS BUSINESS ANALYSTS APPLICATION DEVELOPERS DATA SCIENTISTS
- 9. 9© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology But, but, but, but • Has to be compute and storage together in the same server • Virtual Hadoop is fine for testing but not for production • Cloud-based Hadoop is so much easier and faster • Hadoop is shared nothing – you can’t use shared storage and resources Hadoop HAS to be __________ (insert anachronism here)
- 10. 10© Copyright 2015 EMC Corporation. All rights reserved. INCREASE DATA GRAVITY! INCREASING DATA GRAVITY ENABLES RAPID, VALUABLE ANALYTICS ANALYZE EVERYTHING All of the data More sophisticated analyses New combinations and correlations KEEP DATA TOGETHER Structured, unstructured, dark Generated by the enterprise, imported from outside Historic & real-time SPEED ANALYTICS APPS DATA KEEP APPS CLOSE TO DATA Deliver data consistently & in a standardized way Get at the data quickly Build views and applications each user really needs EMC Solutions are Powered by Intel® Xeon® Processor Technology
- 11. 11© Copyright 2014 EMC Corporation. All rights reserved.© Copyright 2014 EMC Corporation. All rights reserved. EMC Solutions are Powered by Intel® Xeon® Processor Technology IT NEEDS TO BE MANAGED LIKE ONE Data is a fixed asset • Organizations need a data-centric culture • Everyone needs to increase data awareness • Data needs context and purpose • Data should be controlled, provisioned, and purposed just like any other fixed asset
Editor's Notes
- #4: Evolution of DevOps – or – Why your IT guys want to punch you in the face The Old Way: Dev builds stuff without any operational knowledge - Throws stuff over wall to IT - Complains when it doesn’t work the way it did on their test box
- #5: Why traditional Hadoop (and indeed many analytics tools) <> DevOps – or – What is this 1993? Clusters of monolithic, single-purpose servers are the “Insert something that used to be cool but isn’t anymore – like parachute pants or camcorders" of Operations – so very, very not happening anymore The monolithic data-center construct died a long time ago, but new analytics apps are stuck in that past – time to move on
- #6: The middle Ground: things like VM Ware come into being and suddenly it is possible for Dev to have shiny toys too! They still don’t really understand ops, but at least their test rigs are better approximations of reality
- #7: DevOps Movement: Aligning the Dev and Ops mentality with technology to merge the ways of thinking: Dev wants to move fast Ops wants to be able to fix broken stuff How did Ops get to where they are? Shared things – storage, compute, now networking and even platforms Virtualize ALL THE THINGS
- #9: We also recognize that big data is a team sport – we needed to support various user types including IT Operator, Data Scientist, App Developer as well as LoB user and the sponsoring executives on the business and IT In the first instance we’ve decided to make the life of the IT Operator easy (months down to a week for standing up these environments) Next we will focus on providing a true self service big data environment
- #10: Hadoop HAS TO BE SHARED NOTHING – why? Because it’s cheaper? LIES! It sure as hell isn’t easier for your ops guys! Why is it better? CPU and Disk have to be together – NOPE Performance of VM is bad – NOPE And don’t even start on Cloud – it is just another word for Virtual Hadoop that runs NOT in your datacenter…