xAPI State of the State: xAPI for Reals
2 likes837 views
The document provides an overview of the state of xAPI (Experience API) in 2018. It discusses challenges with scaling xAPI productivity and the need for shared expectations among xAPI talent, automated data operations, and market-fit policies. It then shares the results of 2018 research on xAPI-related skills for learning professionals and xAPI profiles. It also discusses the xAPI specification, its use by the US Department of Defense, and ongoing IEEE standardization efforts. Two case studies show billions of dollars of savings through use of xAPI for training programs in the US Navy and government career development.
xAPI State of the State: xAPI for Reals
- 1. xAPI For Reals: The State of xAPI in 2018 Aaron E. Silvers xAPI Camp: DevLearn 23 October 2018 aaron@makingbetter.us
- 2. In 2018, It’s Difficult to Scale Productivity with xAPI (But We Know What We Need…) • Available Talent Who Work with xAPI to a Set of Shared Expectations • A Way to Automate Complicated, Contextual Data Operations Unique to xAPI • Policies that Make xAPI Fit-for-Market
- 3. So Today, We’ll Share Out the Results of Our 2018 Research Efforts • xAPI-related Knowledge, Skills & Abilities for Learning Professionals • xAPI Profiles • xAPI: The Specification, The US Department of Defense Instruction, The IEEE Standardization
- 4. xAPI-related Knowledge, Skills & Abilities for Learning Professionals
- 5. Take a Picture: Two 2018 Case Studies in xAPI’s ROI $1 Billion $1.6 Billion The US Navy appropriated $1B spend to modernize enlisted training programs, converting content and upgrading systems to leverage xAPI for data-driven "Ready Relevant Learning,” saving $1.3B by 2021. The Sailor 2025 Program The Office of Personnel Management FY19 Budget designates $1.6B to modernize career development across US Government to leverage xAPI to enable data-driven approaches to Learning & Development. USA Learning https://news.usni.org/2018/02/23/ready-relevant-learning-pushing-out-to-the-fleet https://www.opm.gov/about-us/budget-performance/budgets/ congressional-budget-justification-fy2019.pdf
- 6. What’s Do People Need to Know to Rock xAPI Projects that Make Data? • A knowledge of xAPI’s fundamentals • The development and maintenance of an xAPI data strategy • A knowledge of market-available Learning Record Providing solutions and their capabilities
- 7. A Common Challenge with xAPI… • Organizations derive the most value from xAPI when the data generated from one source aligns with similar data from other sources. • In practice, the tools and content widely available on the market generate data that follows the structure required of xAPI, but how xAPI is implemented to create data varies. • This results in data that is technically interoperable (same data structure) but lacks semantic interoperability (same meaning): two “completion” statements from two different authoring tools might not be recognized as similar.
- 8. Department of Defense Implementation Plan for Advanced Distributed Learning. Office of the Deputy Undersecretary of Defense (Readiness & Training), 1999, prhome.defense.gov/ portals/52/documents/rfm/readiness/docs/adl_stratplan.pdf. “In short, the strategy is to: pursue emerging network-based technologies; create common standards that will enable reuse and interoperability of learning content; lower development costs; promote widespread collaboration that can satisfy common needs; enhance performance with next-generation learning technologies; work closely with industry to influence COTS product development cycle; and establish a coordinated implementation process.”
- 9. Investigating Performance Do I currently have this Knowledge and these Skills and Abilities? If not, what am I willing to do to obtain these KSAs? Effron, Janet Laane, and Sean Putman. Investigating Performance: Design and Outcomes with xAPI. Edited by Megan Bowe, MakingBetter, 2017.
- 10. Get Ready for 14 Slides of All New, SRSLY Serious Competencies (nobody’s joking)
- 11. Knowledge: xAPI’s Fundamentals Skill/Ability Performance Measure Understands/Explains Observed by Supervisor to a) engage in educational opportunities and b) explain xAPI fundamentals Required elements of an xAPI activity statement Optional elements of an xAPI activity statement
- 12. Knowledge: xAPI’s Fundamentals Skill/Ability Performance Measure Describes Observed by Supervisor to a) engage in educational opportunities and b) explain xAPI fundamentalsVerbs, Activities, Activity Types, Attachment Usage Types, Document Resources, and Extensions that may be required of the Design
- 13. Knowledge: xAPI’s Fundamentals Skill/Ability Performance Measure Determines Observed by Supervisor to a) engage in educational opportunities and b) explain xAPI fundamentals A well-written activity statement from poorly written statements What should go into an Extension
- 14. Knowledge: Development and Maintenance of an xAPI Data Strategy Skill/Ability Performance Measure Understands/Explains Established ontology and/or taxonomy used in the organization Observed by Supervisor to produce or maintain a conformant xAPI Profile Observed by Enterprise-level Data Architecture Stakeholders that the xAPI Profile conforms to or supports required taxonomy/ontology conventions
- 15. Knowledge: Development and Maintenance of an xAPI Data Strategy Skill/Ability Performance Measure Identifies Observed by Supervisor to produce or maintain a conformant xAPI Profile Observed by Enterprise-level Data Architecture Stakeholders that the xAPI Profile conforms to or supports required taxonomy/ontology conventions Key interactions in the Design Key performance indicators in the Design Established xAPI vocabulary, activities and Profiles required in the Design
- 16. Knowledge: Development and Maintenance of an xAPI Data Strategy Skill/Ability Performance Measure Defines Observed by Supervisor to produce or maintain a conformant xAPI Profile Observed by Enterprise-level Data Architecture Stakeholders that the xAPI Profile conforms to or supports required taxonomy/ontology conventions xAPI activity statements related to key interactions Groupings or patterns of xAPI activity statements that evidence key performance indicators A maintenance plan that addresses evolutionary changes and graceful retirement of vocabulary no longer used
- 17. Knowledge: Development and Maintenance of an xAPI Data Strategy Skill/Ability Performance Measure Develops/Maintains Observed by Supervisor to produce or maintain a conformant xAPI Profile Observed by Enterprise-level Data Architecture Stakeholders that the xAPI Profile conforms to or supports required taxonomy/ontology conventions An xAPI Profile conformant to the xAPI Profile specification
- 18. Knowledge: Market-available Learning Record Provider Solutions and Capabilities Skill/Ability Performance Measure Understands/Explains Validated by third-party services that the activity statements generated by the Learning Record Provider conform to the xAPI specification Confirmed by Enterprise-level Data Architecture Stakeholders that the generated activity statements support or conform to Design requirements Commercial, Off-the-Shelf Tools on the market that potentially support the Design requirements Open-source code libraries that potentially support the Design requirements
- 19. Knowledge: Market-available Learning Record Provider Solutions and Capabilities Skill/Ability Performance Measure Identifies Validated by third-party services that the activity statements generated by the Learning Record Provider conform to the xAPI specification Confirmed by Enterprise-level Data Architecture Stakeholders that the generated activity statements support or conform to Design requirements Activity statements and any other xAPI-related capabilities generated by the given Learning Record Provider
- 20. Knowledge: Market-available Learning Record Provider Solutions and Capabilities Skill/Ability Performance Measure Evaluates Validated by third-party services that the activity statements generated by the Learning Record Provider conform to the xAPI specification Confirmed by Enterprise-level Data Architecture Stakeholders that the generated activity statements support or conform to Design requirements Generated activity statements against defined needs
- 21. Common Roles & Phases in Agile xAPI Projects that Generate Data Role Manage Design Develop Deliver Product Owner Business Analyst Instructional Designer User Experience (UX) Designer Engineer (Back-end) Engineer (Front-end) Engineer (QA) Delivery Manager
- 22. Quality Controls and Applicable KSAs: Manage Quality Controls KSAs Roles Develop the Business Case Knowledge of xAPI’s Fundamentals • Product Owner • Business Analyst Perform SWOT Analysis Create the Opportunity Statement Define Project Objectives Develop the Project Scope • Knowledge of xAPI’s Fundamentals • Development and Maintenance of an xAPI Data StrategyComplete the Project Plan Finalize Project Charter Knowledge of xAPI’s Fundamentals Develop a Process Flowchart • Knowledge of xAPI’s Fundamentals • Development and Maintenance of an xAPI Data Strategy
- 23. Quality Controls and Applicable KSAs: Design Quality Controls KSAs Roles Translate Requirements into Critical-to-Quality Components Development and Maintenance of an xAPI Data Strategy • Business Analyst • Instructional Designer • UX Designer Conduct Benchmarking Knowledge of Market-Available Learning Record Provider Solutions and Capabilities • Product Owner • Business Analyst • Instructional Designer • UX Designer Reduce the Set of Potential Design Concepts Development and Maintenance of an xAPI Data Strategy • Product Owner • Instructional Designer • UX Designer Evaluate Potential Design Concepts Development and Maintenance of an xAPI Data Strategy • Product Owner • Business Analyst • Instructional Designer • UX Designer Evaluate the Prototyped Design • Knowledge of xAPI’s Fundamentals • Development and Maintenance of an xAPI Data Strategy • Knowledge of Market-Available Learning Record Provider Solutions and Capabilities • Product Owner • Business Analyst • Instructional Designer • UX Designer
- 24. Quality Controls and Applicable KSAs: Develop Quality Controls KSAs Roles Evaluate the work product functions to design specifications • Knowledge of xAPI’s Fundamentals • Development and Maintenance of an xAPI Data Strategy • Product Owner • Business Analyst • UX Designer • Instructional Designer • Engineer (Back-end) • Engineer (Front-end) • Engineer (QA)
- 25. Quality Controls and Applicable KSAs: Deliver Quality Controls KSAs Roles Compare Delivery Approaches Development and Maintenance of an xAPI Data Strategy • Product Owner • Delivery Manager Confirm the Product Meets Business Requirements • Product Owner • Engineer (QA) • Delivery Manager Confirm the Product Achieves Strategic Goals and Objectives • Product Owner • Business Analyst • Delivery Manager
- 26. That’s a lot… is it worth it? When we’re dealing with high stakes performance (Clinical Medicine, Legal Compliance, Military Operations, Safety), the level of professionalism and effort needed is intense… When high stakes performance matters, the level of effort required is less for a greater return on investment (ROI).
- 27. In Case It’s Buried by Knowledge… Two 2018 Case Studies in xAPI’s ROI $1 Billion $1.6 Billion The US Navy appropriated $1B spend to modernize enlisted training programs, converting content and upgrading systems to leverage xAPI for data-driven "Ready Relevant Learning,” saving $1.3B by 2021. The Sailor 2025 Program The Office of Personnel Management FY19 Budget designates $1.6B to modernize career development across US Government to leverage xAPI to enable data-driven approaches to Learning & Development. USA Learning https://news.usni.org/2018/02/23/ready-relevant-learning-pushing-out-to-the-fleet https://www.opm.gov/about-us/budget-performance/budgets/ congressional-budget-justification-fy2019.pdf
- 28. xAPI Profiles
- 29. What challenges with SCORM did xAPI address?
- 30. What challenges with xAPI does xAPI Profiles address? • Aligns Data Generated by Multiple Sources • Scales xAPI’s Best Practices • Reduces the Skill Level Required to Produce High Quality Learning Data • Allows for Version Control and Retrospective Analysis
- 31. What challenges with xAPI does xAPI Profiles address… for YOU? • Supports better, more accurate reporting of learning experiences involving multiple learning activities. • Embeds xAPI’s best practices into workflows, making it easier to work well with xAPI.
- 32. Case Study: Elsevier Transition- to-Practice Almost half of all newly graduated registered nurses quit within the first two years. A new product being developed leverages multiple approaches to identify newly graduated registered nurses who are at-risk. These activities yield targeted feedback for nursing managers about engagement, competency (and gaps) to help nurses feel more confident and more supported, and improve on that attrition rate.
- 33. Artisanal, Hand-Crafted Data • I have to look up what vocabulary already exists that may be related to the goals of our content, before we have all the content. • Next, I have to draft model statements that our content, assessments, surveys and other apps should be making. • After reviewing the approach with stakeholders, I have to validate that these statements actually conform well to the xAPI spec. • Then, after documenting all this, developers need to translate the specifications I write into actual code, which has to be tested multiple times to make sure the data generated is to-spec.
- 34. WHAT ARE WE TRACKING? …is the question I imagine you’re asking or gonna ask me.
- 35. Our First Prototype uses xAPI to Track… • Content-Specific eLearning Events • Assessment Events • Feature Usage Events • Generalized eLearning Events • Navigation • Media Events • Reading Events
- 36. BETTER xAPI DATA IS HANDCODED (TODAY)!
- 37. Current State: Vocabulary (not quite xAPI Profiles) Vocab Server - https://xapi.vocab.pub The approaches to publishing and using linked data are the same as xAPI Profiles, but there are differences between “vocabulary” and profiles. xAPI Profiles have more requirements and metadata than what’s available through the Vocab Server, the interface to ADL’s ontology set. • ADL has a vocabulary and it also has an xAPI Profile for SCORM. The SCORM profile can use the ADL vocabulary and other vocabularies in addition to custom data elements, statement templates/patterns defined within the profile. • Medbiquitous has a core vocabulary, but plan to build several different profiles based on their various medical scenarios or use cases. They have already built one profile called Virtual Patient.
- 38. Current State: xAPI Profiles The xAPI Profile specification has two main parts. • One section describes how a profile should be documented with metadata describing an xAPI Concept. Multiple xAPI Concepts make up one profile. This section of the xAPI Profile specification offers guidance on how vocabulary should linked together with metadata across different profiles. • The other section of the xAPI Profile specification details conformance requirements for minimal functionality from a profile server. This section requires that a profile server be able support: • Statement template validation • Statement pattern validation 7% 93% The current Vocal Server meets only 3 of 38 identified requirements for an xAPI Profile Server
- 40. xAPI: The IEEE Standard
- 41. Four Stages of IEEE Standard Development 1. Project Initiation & Standard Development: Projects are initiated by existing Working Groups or by external groups who propose new standardization projects. =The bulk of the work in creating a standard involves developing a draft standard that is ready for the formal IEEE approval process. This work is carried out by the individual working groups listed in the Navigation panel to the left. Work is done via email lists, teleconferences, and face-to-face meetings as needed. Any materially interested person may participate. LTSC membership is required for posting to the mailing lists and accessing working group and study group documents and IEEE Standards Association membership is required for voting. Membership is only available on an individual basis – organizations may send individual representatives but may not join as “entities.” 2. Standards Approval: When a Working Group decides it is ready (and the LTSC Sponsor Executive Committee approves), a draft standard is put into the formal IEEE Sponsor Ballot process. This is managed by and requires membership in the IEEE 3. Standards Association: Requires membership in either the IEEE Computer Society, the IEEE or both. Please see the membership page for more details. 4. Standards Maintenance: Maintaining a standard is the responsibility of the Working Group that produced it. Please contact the appropriate Working Group chair if you would like to comment on an existing standard.
- 42. STANDARDIZATION! P92741 (xAPI) Work Group is a spinoff of the work of the xAPI Technical Working Group. There are three components – • P92741.1 (xAPI Specification) • P92741.2 (xAPI Profile Spec) • P92741.2.* (Individual Profiles)
- 43. Where will xAPI be in 2019? in 2020? xAPI 2019 2020
- 44. Thank You The State of xAPI in 2018 Aaron E. Silvers xAPI Camp: DevLearn 23 October 2018 aaron@makingbetter.us