Mstr meetup
0 likes1,054 views
The LinkedIn MicroStrategy Bay Area Meetup on December 4, 2013, discussed business intelligence (BI) challenges, administration hacks, and enhancements in dashboard design using a metadata-driven approach that allows for real-time changes. Key challenges included handling big data, ensuring data quality, and improving navigation and accessibility to reports. The session also highlighted the importance of effective monitoring and administration techniques to support infrastructure and operational efficiency.
Mstr meetup
- 1. SF MicroStrategy Bay Area Meetup December 4, 2013 LinkedIn Confidential ©2013 All Rights Reserved
- 2. Agenda • About LinkedIn • BI Challenges • Administration Hacks • Q&A
- 3. Our Mission Connect the world’s professionals to make them more productive and successful. Our Vision Create economic opportunity for every professional in the world. Members First!
- 4. The world’s largest professional network Over 65% of members are now international 19 Languages 259M+ Members Worldwide Growing at more than two members per second >91% Fortune 100 Companies use LinkedIn Talent Soln to hire >3M Active Company Pages >30M Students ©2013 LinkedIn Corporation. All Rights Reserved.
- 7. Internal User Groups Product Biz-Ops Data Analytics Infrastructure Hiring Solutions Data Sciences Executives Marketing Solutions Talent Solutions
- 8. Business Intelligence Team Bhavani Akunuri Kevin Simon Poorna Lankipalli Sandeep Deviah Sunil Shirguppi Qun Li Sabeer Thajudeen
- 9. Agenda • About LinkedIn • BI Challenges • Administration Hacks • Q&A
- 10. Challenges • BIG DATA - 4 V’s : Velocity, Volume , Variety & Veracity • Data Quality & SLA • Navigation to reports/dashboards • Multiple sources for answering critical questions
- 11. Metadata Driven Dashboards by: Kevin Simon, Sr Data Warehouse Engineer Microstrategy Meetup December 4, 2013 LinkedIn Confidential ©2013 All Rights Reserved
- 12. FOR THE BUSINESS USERS: How much would you be benefited if you had the ability to effect real time change to your critical dashboards, with little to no IT involvement? FOR THE ENGINEERS: How much would you be benefited if your users could make changes to their critical dashboards, without needing to engage you to make simple changes?
- 13. THE OLD WAY … The fixed layout of this dashboard makes it difficult to manage.
- 15. HOW DID WE FIX THE PROBLEM? With a Metadata driven design… Real-time changes can effectively be made to dashboards. IT resources can be freed to work on other projects. Metrics can be introduced into your production dashboards, prior to ETL development.
- 16. PROCESS FLOW AND TABLE DESIGN Daily ETL Aggregate Process Daily Custom Metrics Forecasting Model (R) Dashboard View Aggregate DATE_SK METRIC_SK COUNTRY_ID LOCALE_ID METRIC_NAME METRIC_VALUE 2013-11-01 100 500 1000 PAGE VIEWS 1000 2013-11-01 200 500 1000 SUBSCRIBERS 2000 2013-11-01 9000 550 1000 PV / UNIQUE MBRS 1500 Look up METRIC_SK METRIC_NAME CAT_ID CAT_ORDER METRIC_ORDER DISPL_FACTOR 100 PAGE VIEWS 100 10 10 200 SUBSCRIBERS (M) 100 10 20 1000000 9000 PV / UNIQUE MBR 200 20 10 1000
- 19. IN CONCLUSION… A Metadata approach to dashboard design has allowed for real time change to critical dashboards. More power and control given to users. IT Resources can focus on other things. Time to market greatly reduced. Same message across multiple dashboards.
- 20. Next Play
- 21. Challenges • BIG DATA - 4 V’s • Data Quality & SLA • Navigation to reports/dashboards • Multiple sources for answering critical questions
- 22. Data Quality & SLA by: Sabeer Thajudeen, Senior Software Engineer Microstrategy Meetup December 4, 2013 LinkedIn Confidential ©2013 All Rights Reserved
- 23. Dashboard Delivery Daily ETL Aggregate Process M1 Daily Custom Metrics M2 Forecasting Model (R) …… Mn MSTR Workflow Metric Alert WoW No Data M1 Y 20% Y M2 Y NA Y M3 N NA NA Trigger Cube Workflow Scheduler Data Validation (BTEQ) Command Manager - Trigger Event (Alert Report) Command Manager Validator Trigger Internal Trigger External ABORT Command Manager - Trigger Event (Dashboard)
- 24. Challenges • BIG DATA - 4 V’s • Data Quality & SLA • Navigation to reports/dashboards • Multiple sources for answering critical questions
- 25. Navigation to reports/dashboards by: Bhavani Akunuri, Senior Software Engineer Microstrategy Meetup December 4, 2013 LinkedIn Confidential ©2013 All Rights Reserved
- 26. Too Many Clicks
- 27. Resolution • Identified top used reports • Create short links • Landing page re-design Demo
- 28. Sample Code Last Updated Timestamp <web:ifBeanValue bean="frame" property="getDesignMode" value="0"> <web:then> <div class="mstrMenuLastUpdatedCustom"> <web:displayGuiComponent name="report_last_update" isContainer="true"/> </div> </web:then> <web:else> <web:ifBeanValue bean="frame.rb" property="getXMLStatus" value="6"> <web:then> <div class="mstrMenuLastUpdatedCustom"> <web:displayGuiComponent name="report_last_update" isContainer="true"/> </div> </web:then> <web:else> <web:displayGuiComponent name="report_design_area" isContainer="true"/> </web:else> </web:ifBeanValue> </web:else> </web:ifBeanValue>
- 29. Challenges • BIG DATA - 4 V’s • Data Quality & SLA • Navigation to reports/dashboards • Multiple sources for answering critical questions
- 30. Q: How are my key products performing? Q: What caused the variance on KPI? Q: Does it have a global effect? Demo
- 31. Sample Code Overlay Effect Pop Up Code <style> .overlay_effect-ingraphs, { display:none; position:fixed; opacity:0.7; height:50%; width:50%; z-index:100; } </style> <script type="text/javascript"> $(document).ready( function() { var main = $(this); $("#ingraphs").click(function() { loadPopupBox(".overlay_effect-ingraphs",".popupInGraphs"); }); function loadPopupBox(overlay,popup){ $(overlay).fadeIn("slow"); $(popup).fadeIn("slow"); } }); </script>
- 32. Sample Code Overlay Effect Pop Up Code <style> .overlay_effect-ingraphs, { display:none; position:fixed; opacity:0.7; height:50%; width:50%; z-index:100; } </style> <script type="text/javascript"> $(document).ready( function() { var main = $(this); $("#ingraphs").click(function() { loadPopupBox(".overlay_effect-ingraphs",".popupInGraphs"); }); function loadPopupBox(overlay,popup){ $(overlay).fadeIn("slow"); $(popup).fadeIn("slow"); } }); </script>
- 33. Next Play
- 34. Agenda • About LinkedIn • BI Challenges • Administration Hacks • Q&A
- 35. Administration Hacks by: Sabeer Thajudeen, Senior Software Engineer Microstrategy Meetup December 4, 2013 LinkedIn Confidential ©2013 All Rights Reserved
- 36. Swapping iServer names in User Bookmarks Context: ▪ Disaster (DR) recovery is critical . MSTR infrastructure has been set up for DR ▪ Each time we test DR, VIP and base URL remains the same. However, iServer names change Impact: ▪ Bookmarks (to access reports directly) include iServer names and will not work Resolution: ▪ Use MSTR SDK to seamlessly swap incorrect iServer names in bookmarks (External Security Module) Link
- 37. Implementation Detail ▪ Create a Custom ESM (External Security Module) that extends ‘Abstract External Security’ ▪ Override the function ‘handlesAuthenticationRequest’ – 1) Read iServer names in incoming URL using regex. Matcher and regex.Pattern libraries ▪ Pattern pat_type_1 = Pattern.compile("Server=.{4}-rpt..", 2); ▪ Pattern pat_type_2 = Pattern.compile("mstrWeb=-.{5}-rpt..", 2); for(Matcher mat_type_2 = pat_type_2.matcher(requestURL); mat_type_2.find();) { //read iServer name in incoming URL String iServerName = mat_type_2.group(); …. – 2) Replace if iServer name does not match expected server name Example: //Does iServerName in URL match with expected Server Name if(! iServerName .equalsIgnoreCase((new StringBuilder("mstrWeb=-")).append(Location).append("*").append(Server1).toString()) && ! iServerName .equalsIgnoreCase((new StringBuilder("mstrWeb=")).append(Location).append("*-").append(Server2).toString())) { …… //Correct Server Name requestURL = mat_type_2.replaceAll((newStringBuilder("mstrWeb=-")).append(Location).append("*").append(Server1).toString()); ….} } – 3) Resubmit corrected URL - response.sendRedirect(requestURL); ▪ Update MicroStrategy Config Settings – Application Settings to use Custom ESM (Web Customization Editor)
- 38. Monitoring MSTR via Web Context: ▪ 24/7 Operations team to monitor and support infrastructure ▪ Handles not just MSTR but Teradata, Hadoop & ETL pipleline ▪ Prefer a web based system for monitoring rather than a desktop client Solution ▪ Built a Web GUI to monitor critical aspects of the MicroStrategy infrastructure – Delivery Status of all our critical dashboards – Cube Refresh Status – Job monitor for currently active jobs – User Connection Monitor MSTR Monitor (Web Application) InSights Monitor JDBC Statistics Tables SDK MSTR Application (iServer)
- 39. Implementation Detail ▪ Dashboard Delivery Status and Cube Status – IS_DOCUMENT_STATS and IS_CUBE_REP_STATS ▪ Instantiate WebObjectsFactory: factory = WebObjectsFactory.getInstance(); ▪ Force connection to each iServer //Get cluster information WebClusterAdmin wca = factory.getClusterAdmin(); WebClusters wcs = wca.getClusters(); for(int i=0;i<wcs.size();i++){ for(int j=0;j<wcs.get(i).size();j++){ //Set load balance factor to 0 to force cionnection to other iServer wcs.get(i).get(j).setLoadBalanceFactor(0); jobDetails+= getJobDetails(session,factory,httpReqId); wcs.get(i).get(j).setLoadBalanceFactor(1); } } ▪ Inside getJobDetails: //Retrieve job source. Loop through each job to get job info //Use the same logic for the source WebMonitorTypeUserConnection to obtain session info strSessionID = session.getSessionID(); source = (JobSource) factory.getMonitorSource(EnumWebMonitorType.WebMonitorTypeJob); results = source.getJobs(); for (int i=0; i < results.getCount(); i++) { Job singleJob = results.get(i); job_id= singleJob.getJobID(); duration = singleJob.getDuration();…. }
- 40. Next Play
- 41. Credits ▪ Vincent Kwong – Program Manager ▪ Anirudh Gupta & Crowe Bonnie – Meet up Organizers ▪ LinkedIn AV team, Communications & Security ▪ Data Science & Biz-Ops ▪ ETL & Platform Teams ▪ NOC, DS Operations & DBA
- 42. Agenda • About LinkedIn • BI Challenges • Administration Hacks • Q&A