Open Source Software, Distributed Systems, Database as a Cloud Service
1 like5,404 views
- Treasure Data is a database as a cloud service company that collects and stores customer data beyond the cloud [1]. - It uses open source software like Fluentd and MessagePack to easily integrate and collect data from customers [2]. It also uses open source distributed systems software like Hadoop and Presto to store, process and query large amounts of customer data [3]. - As a database service, it needs to share computer resources securely for many customers. It contributes to open source to build and maintain the distributed systems software that powers its cloud database service [4].
Open Source Software, Distributed Systems, Database as a Cloud Service
- 1. Open Source Software, Distributed Systems, Database as a Cloud Service 第106回オープンソースサロン・総会記念講演 Jul 29, 2016 Satoshi Tagomori (@tagomoris)
- 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
- 3. Topics • What is Treasure Data? • Who is tagomoris? • Treasure Data: Database as a Service • DB as a Service and Distributed Systems • Distributed Systems and Open Source Software • Open Source Software and Developers
- 7. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server 3 Complex ETL 4 End User System 2 Time consuming integration 1 Disparate data silos Without Treasure Data Advanced Analytics Reporting BI
- 8. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK With Treasure Data 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs Multi-Tenant Cloud Service 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations Schema-flexible, Access via SQL, Unlimited Users, Queries
- 11. Treasure Data, Inc. • Since Nov 2011 • Headquarters: Mountain View, CA, US • Japan Branch: Marunouchi, Chiyoda, Tokyo • Korea Branch: Gangnam, Seoul • Some remote workers - US, UK, Costa Rica
- 12. Developers in TD • Daily development in each offices • Communication over Internet • Slack, JIRA, Confluence & Zoom • Frontend Team: mainly in US • Console, Web services, etc • Backend Team: mainly in JP • Database, Distributed processing systems, etc
- 13. Satoshi "Moris" Tagomori (@tagomoris) Born in Matsue, Shimane Living in Tokyo from 1999
- 14. Started to work as an OSS developer 1. Asahi Net Internal system developer 2. NTT DATA Intellilink System consultant 3. livedoor - NHNJ - LINE Infrastructure engineer Data analytics platform engineer 4. Treasure Data Backend engineer OSS developer
- 15. @tagomoris as an Open Source Software Developer • Author • Norikra, Woothee, xbuild, Shib, Yabitz, Focuslight • Many fluent-plugin-* • And many libraries, tools, etc • Committer, Maintainer • Fluentd, MessagePack-Ruby, etc • Contributor • Docker (logging driver), etc
- 16. @tagomoris as an Open Source Software Developer • Talks • Many programming conferences (local, global) • Many small meetups • Articles • WEB+DB Magazine, Software Design • Many blog posts • Invented Event: ISUCON
- 17. OSS Developers in TD • MessagePack, Fluentd, Embulk & Digdag founder • Ruby committer • Ruby & JRuby committer • Fluentd & D-language committer • Hadoop/Spark contributor, pyenv author, ...
- 18. Why Are OSS Developers So Major in TD?
- 19. Treasure Data: Database as a Cloud Service
- 20. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs Multi-Tenant Cloud Service 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations
- 21. Database as a Cloud Service • Collect data • from remote site - customer side • Store/Process data • beyond cloud • Integrate data • to remote site - customer side
- 22. Two OSS Pattern in TD • OSS to collect/integrate data from/to remote site • OSS to store/process data
- 23. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs Multi-Tenant Cloud Service 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations
- 24. Make Input/Output Easy • Agent installed in our customers systems • OSS + Plugin to connect various systems • No barrier to use TD 1.Make a great OSS product to do it 2.Make it major 3.Potential customer already uses it :) • very easy to switch to use Treasure Data!
- 25. Multi-Tenant Cloud Service API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations
- 26. Database as a Service and Distributed Systems
- 27. Many Customers in a System • Share computer resource • Provide much more computer resource • Reduce total cost :-)
- 28. Big Data in a System • Manage big data from many customers • Manage computing power for many customers • Create a distributed system! • for fast query processor • for resource scheduler • for high availability
- 29. Distributed Systems and Open Source Software
- 31. Distributed System Software • Major software are all OSS • Hadoop, Presto, Kafka, Storm, ... • Concept and Implementation • MapReduce concept was from Google • Yahoo! engineers implemented it as Hadoop • Many others made Hadoop better • Data is always growing -> Software MUST be growing too
- 32. Deploying Distributed System • Many things make it hard to fix issues • Big data, many computers, complex queries, ... • We MUST fix our issues as soon as possible • for our customers • for our operation costs DO IT YOURSELF! → OSS
- 33. Updating Distributed System • It's very hard to update distributed systems • many servers, no data lost, no downtime, ... • Use OSS as-is without dirty fix • to keep it easy to upgrade "software" • Contribute your patch to community • to use patched mainstream software as-is
- 35. DIY Policy Makes "Tech" Company • Do it yourself "At Your Own Risk": OSS • Taking risk: more OSS • OSS: more controllable than proprietary software • We can read/contribute source code :) • Technology problem: Can we take a risk? Or not?
- 36. Tech Company and Developers • Taking risk for business success: more focus on technology • Quality of OSS depends on each developers • Who is the committer of that product? • Who can review quality of that product? • Tech company needs great developers seriously!
- 37. OSS and Developers • "OSS Committer", not "OSS Committing Company" • the initiative by developer, not company • Commit log shows everything about common things • Who did contribute to that software? • Who did develop that feature? • Who did fix that problem? • People can know who is a good software engineer • it makes good developers happy!
- 38. Developers love OSS Company • OSS Company: a kind of Tech Companies • easy to find it: see committers/contributors • Developers love: • challenging "technical" tasks/issues to be solved • great coworkers, like committers of great software • nice salary brought by taking risk :P
- 39. Enjoy Engineering! MOST IMPORTANT THING: Thanks!