Hadoop Twelve Predictions for 2012
18 likes9,757 views
In 2011, Apache Hadoop gained significant mainstream traction, with increased enterprise adoption and interest from major corporations and investors. Analysts predict that by 2015, organizations leveraging diverse information types will outperform peers financially, while Hadoop's evolution will focus on enterprise-class features and security. Predictions for 2012 include the growth of data science roles, a decline in Hadoop alternatives, and the maturation of the open-source platform, driving further adoption and commercial opportunities.
Hadoop Twelve Predictions for 2012
- 1. Presenter: Mike Olson, CEO, Cloudera Thursday, December 22, 2011 11:00 am PT, 2:00 pm ET Hadoop: Twelve Predictions for 2012
- 2. Looking Back at Apache Hadoop in 2011 • 2011 was the year that Apache Hadoop became part of mainstream dialogue, and industry interest in Hadoop exploded. – VC firms and investors continued to make significant commitments to Hadoop, like Accel Partners’ $100M Big Data Fund. – Big corporate players such as EMC, IBM, Microsoft and Oracle announced their entry into the Hadoop space. A handful of Hadoop-focused startups also launched this year. – Hadoop-related inquiries into major analyst firms moved from “What exactly is Hadoop?” to “Which vendors offer robust Hadoop solutions?” • Hadoop adoption increased within enterprises and new industry use cases emerged. • With the uptick of interest in Hadoop, there was a notable increase in demand for skilled Hadoop workers and data scientists. 2
- 3. 3 Key Analysts’ Predictions for Big Data and Hadoop in 2012 Looking Ahead
- 4. Industry Analysts Predict… “We predict that through 2015, organizations integrating high-value, diverse, new information types and sources into a coherent information management infrastructure will outperform their industry peers financially by more than 20%.” — Merv Adrian, Research VP, Information Management, Gartner 4
- 5. Industry Analysts Predict… “The core Hadoop platform will mature rapidly. “While I question some of the Hadoop industry's marketing, I have an optimistic view toward the community's plans for maturing core Hadoop. There's low-hanging technical fruit. There's understanding of the need. There's money. There are great engineers. There are partners with market and technical understanding. The Hadoop engineering stars are favorably aligned.” — Curt Monash, Analyst, Monash Research 5
- 6. Industry Analysts Predict… “1. Big Data was the term du jour for 2011. In 2012, we’ll move away from this term, as ANY DATA will be of paramount importance for enterprises. At the same time, IT resources are scarce, and quite frankly, most IT organizations are trying to simplify their IT environments and look at it more strategically and holistically. Intelligent management will be key, as enterprises look to analyze any data, at any given time (and as near real- time as possible), and management tools will need to determine, in the most efficient manner, where the HW/SW resources must come from, whether it’s for structured, semi- structured, or unstructured…or any combination thereof. This demand will push the ecosystem to continue to grow and deliver the necessary solutions to the market. “ 2. Data protection, security are key areas that have not been adequately addressed…and will become important to do so as enterprises think about leveraging cloud for their analytics. “3. ‘Big Data as a service’ the new term du jour. Enterprises will look to leverage both their own resources for sensitive datasets and cloud for others…service providers will become players in delivering these services.” — Vanessa Alvarez, Analyst, Infrastructure & Operations, Forrester 6
- 7. Industry Analysts Predict… “1. We will see a major shift of current experimental Hadoop users (the bulk of current users) to mainstream production applications that will ultimately drive the need for more enterprise-class, advanced data management features. While these applications will be limited in scope and moderate in cluster size, the business requirements will drive the need for more options related to HA, advanced data protection and remote-site failover. “2. We will see more IT organizations initially considering Hadoop as an alternative lower cost online storage for Big Data – essentially a cheaper data store (versus in relational databases with associated license and SAN attached storage) in preparation for future data processing and analytics tasks.” — Julie Lockner, Senior Analyst and Vice President, Data Management Solutions, ESG 7
- 8. Industry Analysts Predict… “In 2011, the federal markets with the most dramatic growth in Hadoop were associated with counter terrorism missions and law enforcement. Fraud detection was also an important market, as was the use of Hadoop-centric solutions for search needs. “We expect 2012 will result in growth in federal heath missions and in military logistics. Research and interest by agencies indicates health related solutions will leverage Hadoop for predicting treatment outcomes based on patient data and historical medical use. We expect military logistics use cases to provide DoD with new efficiencies in optimizing supply chains and predicting needs before they arise.” — Bob Gourley, Founder and CTO, Crucial Point LLC 8
- 9. Industry Analysts Predict… “We believe that during 2012, enterprise distributions of Hadoop will mature enough that enterprises will accelerate production deployments and begin to yield tangible organizational value.” — Ben Woo, VP, Storage and Big Data, IDC 9
- 10. 10 Mike Olson’s Twelve Predictions for Hadoop in 2012
- 11. #12 The Open Source Platform Expands • New contributions will go mostly to existing and new packages that complement Hadoop. In 2011, nearly 70% of all new work was outside the core project. • New and incubator projects like Flume, Bigtop and Crunch will graduate to critical pieces of the Hadoop platform. This is the continued “Linuxification” of Hadoop. 11 2006 2007 2008 2009 2010 2011 Core Hadoop as % of New Contribs Nothing but Hadoop … plus 11… plus 8…plus 6Hadoop plus 3 100% 100% 37% 58% 37% 31%
- 12. #11 Vendor Investment Accelerates • Every significant server, storage and data infrastructure vendor will have a clearly-articulated Hadoop strategy by end of year. • Product development dollars will go to: – Contributions to the open source platform – Integration of the open source platform with existing IT infrastructure • Best-of-breed, rather than single-vendor, deployments will dominate in 2012 (and for at least three years following). 12 The Cloudera Connect partner program has added at least one new member every day since its launch in 3Q11.
- 13. #10 The Platform Grows Up • Key issues that worry enterprise users will be addressed. – High availability guaranteed by improvements to storage infrastructure • No single point of failure! – Dramatic performance improvements for storage and analytical workloads • Integration with existing data center infrastructure will become complete. • Computer science PhD will no longer be required to operate Hadoop in production. 13
- 14. #9 Hadoop Stops Being Cool • Fascination with the technology ebbs and platform novelty subsides. • New focus: What problems can Hadoop solve for me, and how do I put together a complete suite of products – stack, analytics, visualization – that I can deploy and use easily? 14 (This is a lie. Hadoop will remain cool.)
- 15. #8 It’s All About Apps • Existing BI and analytics products will connect to Hadoop, unlocking it to a large audience of business and technical professionals. • New offerings from established vendors and emerging companies will be designed from the ground up to take advantage of rich data, powerful algorithms and new techniques for data visualization. This represents a major expansion of BI and analytics into new data and problem domains. 15
- 16. #7 Chasm? What Chasm? • Hadoop adoption by mainstream enterprises will accelerate. • Smaller companies are seeing the value of Big Data, too, and will need the tools to analyze it. • Deployments expand beyond single use cases inside departments to the broader enterprise. • It's not how much data you have (PB, TB) but what kind of data you have and what you do with it. 16 Financial ServicesTelecommunications Life SciencesRetailGovernment Platform maturity and app proliferation will drive broad adoption.
- 17. #6 Here Comes Everybody • Small and emerging companies, not widely viewed as having Big Data problems, will discover ways to capture and analyze information in new ways, just like their larger counterparts. • Cloud deployment of Hadoop will accelerate as a result of this trend – small companies and startups have little sunk infrastructure cost and are cash- conscious, so will deploy on hosted platforms faster than will their peers at large enterprises. 17 • With the creation of the Accel Big Data Fund and other VC firms’ interest in Big Data and Hadoop, we will see the launch of numerous Hadoop-focused startups in 2012.
- 18. #5 Opportunity Expands: Security, Storage, Efficiency • As use of Hadoop in core workloads – storage, data processing and analytics – matures inside enterprises, they will begin to look at it as a more general-purpose platform able to attack other business problems. • Improvements in platform security, access control and audit logging means that some compliance workloads will begin to migrate to Hadoop in the second half of the year. • Improvements in the open source storage layer mean that storage gets cheaper still, making Hadoop attractive as an online archive for data previously written to tape and stored in caves. • Better measuring and management of resource consumption will lead to improvements in computational efficiency. • The variety of problems that Hadoop addresses will expand. 18
- 19. #4 Skilled People Get Easier to Find • Professional training courses aimed at developers, operators and users will be widely available and very popular in 2012 and beyond. – Finding people who can deploy, configure and use Hadoop has been difficult in 2011. • Systems integrators and consulting firms will establish Hadoop- focused practices around the world. 19
- 20. #3 Data Science Gets Hotter • Widespread use of Hadoop in new industries, aimed at previously- intractable problems, will create a new role in the organization: “Data Scientist.” – Mathematical and statistical expertise – Programming skill – A passion for data and an understanding of the business • Industry and academia will collaborate to produce this new class of expert. 20
- 21. #2 The Demise of the “Hadoop-Killer” • Second and third quarters of 2011 saw several vendors announce “Hadoop killers” – legacy products, deployed in niche markets, that they wanted to use to capitalize on the opportunity that Big Data presents. • In the fourth quarter of 2011, Microsoft announced it was discontinuing its investment in Dryad. Dryad was, simply, the most interesting alternative to Hadoop in the market. Instead, the company is investing in Hadoop. • The vendors offering alternatives to Hadoop will follow Microsoft’s lead. These products have no real traction in the Big Data market and will quietly disappear by the end of 2012. 21
- 22. #1 Forks Begin to Fade • The open source Hadoop platform will be robust, scalable and reliable enough to take on any enterprise Big Data workload. • As a result, proprietary forks – MapReduce grafted onto legacy-vendor SQL databases, appliances that replace the storage layer with proprietary file systems and others – will see their differentiation erode or disappear. • Buyers will elect to deploy the open source platform once it demonstrates feature parity or superiority with these forked products. 22 Commercial opportunities will constrict for vendors of forked products. It will take several years for these offerings to disappear altogether.
- 23. #0 Bonus Prediction! M&A Consolidation Begins • Large vendors lacking product and expertise in Hadoop will survey the market for teams and technologies that can close those gaps. • Specialist vendors, especially those that have struggled to grow or with positioning and differentiation challenges, will hire investment bankers to seek acquirers. • These early, small acquisitions, combined with an expanding market and accelerated demand for big data solutions, will drive valuations for new and existing independent companies. Buyer meets seller. Nature follows its course. 23
- 24. 24 We appreciate your time and interest. For additional information: cloudera.com +1 (888) 789-1488 sales@cloudera.com twitter.com/ cloudera facebook.com/ cloudera
Editor's Notes
- #3: Hadoop-related inquiries into major analyst firms moved from “What exactly is Hadoop?” to “Which vendors offer robust Hadoop solutions?”: (This noted in a blog post by Forrester analyst James Kobielus)