![15
Document-Structured Data Challenges
About HBX
HBX Data
Management
Initiative
Architecture &
Implementation
Challenges
Data Driven Culture
Impacts
What’s Next?
{"_id": ObjectId("562666868c58dab88be84345"),
"course_instance_id": "561829c02804c51563c9c772",
"tei_id": "55c21b88241b5a14b75fab8d",
"user_id": "3312",
"state": {"answer": "The case really drove home..."}}
{"_id": ObjectId("56414b498c58dab88bf10873"),
"course_instance_id": "561829c02804c51563c9c772",
"tei_id": "5639ef402804c509af1d2721",
"user_id": "3312",
"state": {"summary":
[{"content": "Incorrect: Being quick to market...",
"correct": False,
"id": "5bc32d05-1173-452c-801b-34c2368ea4b6"},
{"content": "Correct: In the early stages...",
"correct": True,
"id": "88893c55-3e30-4f50-8495-a6fe1f1cef94"},
{"content": "Incorrect: Customization becomes...",
"correct": False,
"id": "4afbf29f-d23d-43a3-8266-e41df3defa69"}]}}
User state documents for reflection and multiple choice](https://image.slidesharecdn.com/itsummit-datamgmt-2016-160708173119/85/It-summit-data-mgmt-2016-06-02-final-15-320.jpg)
It summit data mgmt-2016.06.02-final
- 1. 2 JUNE 2016 BUILDING A CLOUD BASED DATA WAREHOUSE GILDAS BAH, BRENT BENSON, & RYAN FRAZIER
- 2. 2 Presenters About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Ryan Frazier – Director, Systems Engineering and Operations Brent Benson – Enterprise Architect Gildas Bah – Data Analyst Engineer
- 3. 3
- 4. 4 Harvard Business School’s newest division, tasked with reimagining business education for the digital age Launched in June 2014 Located in Allston, five minutes from HBS campus Moving from start-up to enterprise mode The teaching model sets HBX apart from many online learning options and is reflective of the HBS in- person classroom approach What is HBX? About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next?
- 5. HBX Platforms 5 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? HBX Online Platform HBX Live Mainly asynchronous online business education Engagement through student interaction in cohorts of ~400 Case-based learning with highly interactive teaching elements and peer help WGBH studio-based virtual classroom Synchronous audio/video with chat, polls, boards Up to 60 global students on studio wall, hundreds or more observers
- 6. 66 Building a Data Management Practice
- 7. Why Build a Data-Driven Culture? 7 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Enhance Outcomes • Proactively support struggling students • Identify challenging content • Evaluate and improve interactive content, social engagement, and retention Improve Effectiveness • Scale data intensive activities like marketing, admissions, & grading • Use data to test ideas and improve quality of decisions Refine Pedagogy • Evaluate new pedagogical approaches • Optimize evaluation approaches • Support pedagogical research activities and innovation STUDENTS STAFF FACULTY Foster Innovation & Continuous Improvement • Identify and evaluate innovation opportunities • Drive continuous improvement
- 8. Data Management Program Objectives 8 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Integrate Data Sources into Comprehensive Data Warehouse Build Reports and Dashboards Enable Self Service Ensure Data Quality and Integrity
- 9. Tool and Vendor Selection 9 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Data Warehouse • Standard relational DB, Redshift • Chose Redshift because of scalability, performance • Aligns with AWS platform focus ETL • Informatica, Talend • Chose Informatica because of university relationship and myriad of plugable connectors Reporting/Analytics • Microstrategy, Qlik, Tableau • Chose Tableau because of feature set and industry adoption
- 10. Reporting Copy HBX Data Ecosystem 10 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Course Platform Ver. A MongoDB MySQL Reporting Copy Course Platform Ver. B MongoDB MySQL Historical Data MongoDB MySQL Admin System MySQL Salesforce Redshift Informatica Secure Agent Tableau Server Progress ODBC for MongoDB sync
- 11. HBX Data Management by the Numbers 11 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Source Systems • 35 databases • 887 tables • 5,844 fields • 109,751,902 rows Data Warehouse • 4 Redshift clusters • 8 databases • 404 tables • 5,674 fields • 400,794,679 rows Daily ETL Process • 300 jobs • 6,515,599 rows * Updated 6/1/2016
- 12. HBX Data Models 12 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? { "_id": ObjectId("556f25ab662a9b059ea8df8b"), "tei_id": "554a607b241b5a3f0e09eefe", "course_instance_id":"556dcf55b7431f414d87f06f", "user_id" : "8701", "comments" : [ { "id" : "ce59a25a-ce69-47-c534611f7ebf", "text" : "This is a great response…, "author_id" : "6411", "date_created" : “2015-09-10”, MySQL-Relational MongoDB-Semi-Structured Course offerings Student demographics Applications & registration Limited course content Course structure Course content Student course state Metric (timing) data
- 13. Challenges 13 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Immature data connection support Large object storage limitations in Redshift Difficulty flattening complex/polymorphic data structures
- 14. 14 {"_id': ObjectId("2804c514e4c20e6d"), "course_instance_id": "2804c51563c9c772", "tei_id": "241b5a14b75fac83", "user_id": "3312”, "date_created": datetime.datetime(2015, 10, 14, 10, 56, 59, 137000), "category": "timespan", "metric": {"interaction_time": 180, "is_interaction_time": True}} {"_id": ObjectId("2804c514f92f1f64"), "course_instance_id": "2804c51563c9c772", "tei_id": "241b5a14b75fab70", "user_id": "3312”, "date_created": datetime.datetime(2015, 10, 14, 17, 11, 56, 967000), "category": "view_user_response", "metric": {"viewed_user_id": "9212"}} Document-Structured Data Challenges About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next?
- 15. 15 Document-Structured Data Challenges About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? {"_id": ObjectId("562666868c58dab88be84345"), "course_instance_id": "561829c02804c51563c9c772", "tei_id": "55c21b88241b5a14b75fab8d", "user_id": "3312", "state": {"answer": "The case really drove home..."}} {"_id": ObjectId("56414b498c58dab88bf10873"), "course_instance_id": "561829c02804c51563c9c772", "tei_id": "5639ef402804c509af1d2721", "user_id": "3312", "state": {"summary": [{"content": "Incorrect: Being quick to market...", "correct": False, "id": "5bc32d05-1173-452c-801b-34c2368ea4b6"}, {"content": "Correct: In the early stages...", "correct": True, "id": "88893c55-3e30-4f50-8495-a6fe1f1cef94"}, {"content": "Incorrect: Customization becomes...", "correct": False, "id": "4afbf29f-d23d-43a3-8266-e41df3defa69"}]}} User state documents for reflection and multiple choice
- 16. 16 Document-Structured Data Challenges About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? • Documents with simple and consistent structure are easy to translate into relational form • Documents with simple, but polymorphic structure are handled by modern MongoDB drivers (metric example) • Documents with complicated and polymorphic structure (user state example) push the boundaries of current drivers and declarative tools • Current solution: copy like-typed documents into separate collections • Preferred solution: copy all documents into warehouse and do post-copy transforms for summary and detailed information in relational form
- 17. 17 Creating a Data-Driven Culture
- 18. Creating a Data Driven Culture 18 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? People Technical Partners Leadership Staff Technology Self-service Eliminate Complexity Experimentation Process Process Governance Data Governance Education
- 19. Enablers for Building Data Driven Culture at HBX 19 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Strong Partners • Use off-shore partner Mindtree to accelerate • Active engagement of vendors on technology challenges Education • Short Presentations to staff • Data Analysis Exercise at all-staff team meeting Program Governance • Active interest & involvement from Business Areas • Alignment to organizational priorities Experimentation • HBX willingness to try new things • Helps drive engagement with vendors
- 20. Organizational Impacts 20 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Enablement of real-time data-driven decision making • Dashboards for Registration Pipeline and Demographics • Application Forecasting Dashboard A move from spreadsheets to dashboards and configurable business processes • Development of grading automation data pipeline • Reporting for B2B Participants A move from individually handled data requests to dashboards and self-service reporting • Self-service marketing data extract
- 21. What’s Next? 21 About HBX HBX Data Management Initiative Architecture & Implementation Challenges Data Driven Culture Impacts What’s Next? Streaming Data? Native JSON Data Warehouse? Analytics? Additional Data Sources?
Editor's Notes
- #7: Add link to the actual dashboard
- #11: AWS for core infrastructure Adminsitrative system (migrating to Salesforce over next year) Multiple Prod Environments—each new release, ensure stability Reporting Copy + Archive for Historical Data ETL Reviewed several options Picked Informatica to align with HU, HBS data mgmt architecture Use Cloud version Redshift for EDW Aligns to cloud, AWS Tableau for Reporting
- #19: People: Technical staff to lead and execute Partners & Vendors who work collaboratively to support you Leadership to support investments and organizational priority Staff support to learn new skills and build data literacy Process: Process Governance: Prioritization, Policies, Business Involvement Data Governance: Data Quality, Data Lineage, Data Model Definitions Education: Data Literacy and awareness Technology: Emphasize self-service to enhance data literacy Try to make tools, data models simple for business teams to use and understand Experiment with new technology to be more efficient, solve actual problems, generate more value