Keynote - Cloudera - Mike Olson - Hadoop World 2010

Editor's Notes

• #3: ** 180 public, self-declared powered by Apache Hadoop and related projects (apache powered by wiki)** that’s a small fraction of users** There are 900+ people hereAttendees right now: 915Responses: 151Avg usage in months: > 12mo (for non 0 responses), 8.76mo overallAverage storage size: 114.5TB (for non 0 responses) 78TB overallTotal data under management: 11.79PB for responses, 104.7PB extrapolatedAverage # nodes: 66.25 (for non 0 responses) 46 overallTotal notes under management: 6957 for responses, 60625 extrapolatedLongest time spent on Hadoop: 36 monthsMost data: 2PBLargest cluster: 1300 nodesPercentage over 10 nodes: 55%Percentage over 10TB: 38.5%** 40% are here to start learning about Hadoop
• #4: Last year, we told you that Hadoop was going to be a big deal. It is.You’re here, so you know that. You’re the reason why.Shout outs: Doug, Tom (new book!); Yahoo! for its early and continued contribution (security). Global community.Three things driving success.
• #5: The big story is data. There’s a lot of it. The increase is fastest in varieties of data poorly suited to RDBMS’ tabular storage. 2.5 zettaytes (x1000 exabytes) of new information in 2012 Digital universe grew by 62% last year to 800 zettabytes and will grow to 1.2 yottabytes this year
• #6: In case you want to know what comes next.Keynotes will sound increasingly ridiculous.
• #7: Corollary:With volume, variety and velocity, data is messy.In the 80s and 90s, RDBMS was the answer.Now, we see row stores and column stores (Vertica), emerging class of distributed hash-style systems (Membase, Hbase, Cassandra), Hadoop. Major shift in the data center!
• #8: Hadoop was built for data. Cheap storage lets you deal with volume, but also: variety and velocity.This technology was invented by the biggest names on the Web – Google, Facebook, Yahoo! – to solve their business problems. But those are also your business problems. Those companies weren’t different than you; they were just a little bit in the future. Hadoop is the high-value analytics engine for today’s businesses.
• #9: It is famous for getting big – no “up to”.
• #10: Variety and velocity sometimes come before volume.RECALL: First reason for Hadoop’s success: data.
• #11: Second reason: Open source.Don’t hand Oracle a loaded gun.Community is thrashing proprietary. Bill Joy: All the smartest people work somewhere else.Linux, Postgres, MySQL, JBoss, Xensource: Commoditiized existing markets.Not so, Hadoop! No existing commercial product does large-scale, powerful analytics over a mixture of complex and structured data. First truly innovative OSS platform I know.
• #12: Third reason: No longer just HDFS and MapReduce.Talking to Kellan Elliott-McRae, VP Eng Etsy. “Toolchain is mature.” Talks like an engineer, but he’s right.Last year: Hadoop. This year: A mature, flexible, powerful data management platform ready for enterprise-grade use.Cloudera has been in the market for two years, solving real problems. We’ve badly needed this platform to emerge from the pieces.We believe deeply in the open source ethos. Not political: Gives us a huge unfair advantage in the market.We spent two years listening to you.
• #13: Today, we are announcing the newest release of CDHv3.Integrated the critical pieces of the platform from the global open source community.Incorported Yahoo!’s excellent work on security – see Todd Lipcon’s talk today.Deeply committed to OSS. This is and will always be a 100% pure Apache-licensed platform. No one can turn off your access to your data, and no one can take away your critical business analytics.
• #14: Hadoop: Innovative, powerful, platform, but just one piece of the puzzle in your data center.Needs to play nicely with all the other systems and tools you run.Established vendors, with Cloudera, have announced integration with Hadoop. You’ll see this list grow quickly over the next year. “Right tool for the job.”
• #15: This platform is ripe for creation of new vertical apps and analytical tools. Expect to see an explosion in the number and variety of solutions built on it.
• #16: So what are people doing?You’re surrounded by more than 900 other people today. They’ll tell you how they’re using Hadoop to solve real business problems. Not science projects; not a toy. The platform is driving costs down and letting companies extract profit from data in ways that were impossible just a few years ago.You’ll hear about real use cases from smart people.
• #17: You'll hear General Electric talk about how they're able to capture and monitor data on their mission-critical IT systems. They're able to track load and usage, monitor what users are doing and watch for impending failures. GE runs a large internal cloud; with the Hadoop platform, they're squeezing costs out of that cloud, and driving value straight to the bottom line.
• #18: You'll hear Bank of America talk about its very fast-growing Hadoop cluster. BofA uses the platform to better understand its customers, and to more precisely score risk in its individual user accounts and in its portfolio as a whole. These days, the player in the market who is the most right about risk wins, and wins big. Banks can reduce their reserve capital requirements, freeing up money to invest elsewhere, and can avoid loss and drive gains by choosing market positions with the best information available.
• #19: You'll hear eBay describe how they're using this platform to get much closer to customers. They're able to understand much better the preferences and behavior of individual users. That means they can deliver precisely-targeted products in auctions, tailored exactly to the individual. That precise targeting turns into more purchases, and more purchases drives eBay's top line.
• #20: Forty-two sessions across a very long day.Unlike last year, we’ve scheduled lots of breaks for networking and hallway conversations – always the real reason to attend a conference!Hilary Mason from bit.ly, Flip Kromer of Infochimps, Drew Conway, Terry Jones and Clay Johnson are the young Turks transforming the industry, defining a whole new profession of "data scientist" and creating new frameworks and tools for understanding data and creating insight and profit out of information. Hit the coffee breaks and work the hallways hard.
• #21: We are, all together, in the middle of a dramatic change in how the world works. Data is central to government, business, security and leisure. That's more true today than it was when we saw you here last year. It will be still more true next year. We have all, together, created this amazing, powerful, innovative new open source software platform.Over the next twelve months, you’ll see the market realize this even more broadly. You’ll see the platform continue to evolve and improve. You’ll see more companies take advantage of the power of Hadoop to unlock information from data and drive value and profit.
• #22: Look how far Hadoop has come!Data, open source and the emergence of the platform have made the software mainstream. It’s enterprise-ready, and works with enterprise tools. Look at the progress since Doug created the project back in 2006. Imagine the transformation we’ll see over the next four years.And I love how, even with all that progress, the little yellow elephant is still pushing!Go out and enjoy the day. Hear some great talks and make new friends. Get a book signed by Tom. We’ll see you back here this afternoon – and don’t forget the reception afterward, one floor down.
• #23: Tim is the founder and CEO of O’Reilly Media. Its goal: to be a catalyst for technology change by capturing and transmitting the knowledge of "alpha geeks" and other innovators.Privileged to know him for about fourteen years.Learned a great deal about open source, community and technology from conversations, his speeches and his writings.Created a number of very important industry conferences – OSCON, Etech and others. New in February – Strata, aimed squarely at data scientists.Thank Tim for coming.
• #26: Let's also try to drive home the message that we expect hundreds of vertical apps to be built on top of CDH over the next year or two. So it's not just about the users, it's about the developers (, developers, developers!!!).
• #28: Opportunities for individuals to find and solve interesting, professionally rewarding problems.help make critical changes in the way customers are supported and interacted withContribute to the bottom linemajor new business opportunities etc.
• #29: New kinds of data introduced daily – mashups, sensors, log files, videos etc.Data created at machine speedI think it's worth highlighting that this platform acknowledges the three biggest trends in enterprise data management today: growth in data volumes, growth in unstructured data, and clouds/commodity hardware. Analytics and apps are compelling – see the stories at this conference
• #35: Creation of new leadersSeparate competitors