HBaseCon 2012 | Overcoming Data Deluge with HBase to Help Save the Environment - OPower

Editor's Notes

• #2: Name Email Address Title
• #3: WARNING THESE ARE MY WORDS, not FDS, Cloudera or OPower Factest 2005: - Maybe I was crazy to use it - Tens of databases 10 of query langagues, VMS moving towards commdity servers. Running into issues with scaling on environments like MySQL - They were used to code that crashed. In fact, I would say while I was on call, a service from one of the sites was down, at least once a week. Luckily they had redundancy in multiple sites, and multiple servers within those sites. The redundancy was added at a higher level, so generally, at least all of the times I remember, it was able to increase the availability and downtime wasn't actually an issue. - What was an issue was scale. - INteresting enough Hbase, even at that time, was a pretty highly available database. So what did they use it for - Time and Sales. This is the collection of all of the Quotes and Trades, for different securities. So to translate you put out quotes to buy or sell stocks at a certain price. If they overlapp, the echange registers a trade, and you just bought or sold a security. Not just stocks, but options and extremely high frequency data. - There was some value add on top of that, for calculating more complicated statistics on the fly through a home grown Web SASS thing - Cloudera: - Started off in kitchen focusing on building the packages that y’all know and love. When I entered it was all manual, when I left it was all automated. One could think of this as sortof like dev-opsie, meets, qa, meets release engineering, meets generic development - Moved into our first management tools team as a developer. Where we developed the cloudera manager. It was originally part of HUE and it became more springy. - Then I left Cloudera to be a founder in Drawn to Scale. We built a prototype and started pitching it for about 6 to 7 months. - While that was going on, I because the Lead Data Architect at OPower. And then more recently, after funding, I have returned to drawntoscale as a coder in the trenches, and have changed myself to a advisor to opower. The reason why I bring this up, is I have been working with HBase in production for about 5 years.
• #4: Opower helps people use energy more efficiently and ultimately save money on their energy bills.it vastly improves the overall customer experience by making energy use personally relevant. - Behavioral Science (Great marketing, understanding people, great hci) - Data Science (Analytics, Data Infastructure Teams) - Lobbying (Yep we do lobbying)
• #5: - OHow many of you get a bill - OPower White labeled websites. So this is the interface you probably use through your energy website to view how much power you use. Bill forecasting, etc. - Smart Thermostats - Gas and Electric - Social
• #6: - Analytics is used to understand who we should be targeting - Answering questions that our customers what answered. We can help them improve customer service, improve there marketing, etc. - Justifying our own existence. (Compliance)
• #7: - This is an old slide which doesn’t really include all the places we get data - Story about detecting broken thermostats
• #8: - But it had it’s up - Spring and MVC provided a very clear and systematic way for developers developer systems. - It was very easy to manage from an operations perspective.
• #9: - WE did this at FDS as well. Of course not with R, but specialized langauages. - IN fact our customers did as well, and they had a whole team of people to help customers do it.
• #10: So here is the data sizes we have, along with the costs with traditional hadoop systems. - We were a cisco shop but we ended up going with dell, mostly because of the 3.5 inch disks. It looks like cisco is wising up to this whole hadoop thing. - These numbers are for dell. So I think this is priced out assuming a 710, then a 810 and then a 910 for the RDBMS, and 510's for hadoop.
• #11: - A lot of this data just doesn’t work well with traditional databases. - An unnamed utility takes 3 days to mysqldump the ami data out. subsampling interpolation
• #14: - I should warn you, i drawn almost all of my drawings in xfig so if this isn’t clear I’m sorry. - Basically the utility data has to come in from a variety of different protocols, as we integrate into the utility pipeline. It then flows into hbase, it’s validated from hbase, and then imported into our existent workflow. - Some of that data, you could imagine for instance information about user is still stored in MySQL. - All of the data is in a HIVE data lake
• #15: All of our timeseries data in regards to high frequency data is being ported to being stored in HBase. Also soon things like bill forecasting, and a bunch of cool other stuff I probably should mention is being moved here. This includes data from the utilities, and data that users are enterring themselves. In additition thermostat data is moving here.
• #16: - We still need to improve effeciency - We are doubling the size of the cluster this year - We have a ton of room to grow.
• #17: - Having all of your data is a huge thing - Having a place to do m/r based R is great - No more running out of memory or being bounded to a single machine - Having a cheap scratch space
• #18: At cloudera i thought all we needed was cfengine, snmp and syslog. Frankly that would have made ops happy. But more and more I think we made the right decision and that these tools really aren’t the right answer. JuJu looks interesting. - cloudera of course built there own tool. - access and auth