Bx D Presentation Cstd Hamilton

Blended By Design Creating and Sustaining Meaningful Blended Learning Programs Calvin McKnight Director, Solution Services SkillSoft Corporation

Workshop Goals To begin developing: an executable & measurable blueprint and draft project plan for a blended learning program in which available multi-modal learning assets are assembled to effectively impact an identified performance objective by methodically addressing the needs of adult learners using sound learning objectives to drive your evaluation and marketing strategy SkillSoft

Agenda WELCOME and INTRODUCTIONS BLENDED LEARNING CONCEPTS BLUEPRINTS FOR BLENDED LEARNING DEVELOPING YOUR BLENDED LEARNING PROGRAM PUTTING IT ALL TOGETHER SkillSoft

Welcome and Introductions Please share… Your Name? Your Organization and Role? Your Experience with Blended Learning? Share one thing that you have learned in the past week… SkillSoft

Twenty-first Century Business Imperatives Shifting Competitive Landscape Speed to Market Demands Globalization of the Workforce Corporate Uncertainty Changing Employee Demographics Technology-accelerated Business Process Mission-critical Training SkillSoft

“ In today's business environment, finding better ways to learn will propel organizations forward. Strong minds fuel strong organizations. We must capitalize on our natural styles and then build systems to satisfy needs.” From: Conner, M. L. "How Adults Learn." Ageless Learner, 1997-2004. SkillSoft

The Learning Organization Evolves to Meet Today’s Business Needs Stage 3 Strategic Aligned Learning Stage 1 Supplemental Ad Hoc Learning Stage 4 Systematic Integrated Learning Stage 5 Optimized Performance Learning Stage 2 Targeted Managed Learning SkillSoft

Examples of Learning Growth Indicators Stage 3 Strategic Aligned Learning Stage 1 Supplemental Ad Hoc Learning Stage 4 Systematic Integrated Learning Stage 5 Optimized Performance Learning Stage 2 Targeted Managed Learning Performance and Business Impact Usage and Adoption Informal Learning Assets Formal Learning Assets Effective Development Efficient Deployment SkillSoft

Advanced Blended Learning Strategies Facilitate This Evolution Blended by Design Basic Blended Learning Stage 3 Strategic Aligned Learning Stage 1 Supplemental Ad Hoc Learning Stage 4 Systematic Integrated Learning Stage 5 Optimized Performance Learning Stage 2 Targeted Managed Learning SkillSoft

Definitions Blended Learning: Combines instructional modalities (or Delivery Media) Combines instructional methods Combines on-line and face-to-face instruction Blended Learning Systems: Combines face-to-face instruction with computer-mediated instruction

Definitions (cont’d) Applying different forms of instructional methods (Classroom, e-Learning, collaboration, simulations, etc.) Combining delivery technology (Internet, CD-ROM, etc.) Blended Learning has also been defined as a combination of at least four different delivery elements: Adapted ‘Blended Learning: Let's Get Beyond the Hype’ By Dr. Margaret Driscoll Technical Non-Technical 3. Mixing teaching approaches (behavioral, cognitive and constructive) 4. Integrating formal learning activities with actual job activities and informal learning.

Discussion How does your organization define Blended Learning? SkillSoft

Strengths and Weaknesses Instructor Led E-Learning Strength Networking Peer interaction Instant Q&A Schedule time Flexibility Tutoring Schedule Away from desk Limited audience Travel Instructor leaves Not consistent Weakness Strength Anytime anywhere Where you want it Audience unlimited Don’t need to go anywhere Always available for review Consistent message Weakness No Networking Limited interaction No one to Q&A Do it tomorrow Not Flexible Lack of a tutor

Understand the Issues with BLT Why would someone want to take a 5 day ILT course rather than a 3 day Blended Learning Course over two months? What reasons do they give?

Hearing Issues with Blended Learning I don’t learn that way I can’t learn by computer I want to get away from work I want to get away from my boss Cheaper doesn’t mean better You can’t fit the same training into less time Not used to pulling information I need someone there to hold my hand Where is the networking? I want more peer to peer interaction I want a live body to answer questions I need to get away from telephones Too many interruptions I know it will not work I need 5 days to absorb 5 days of training No lunches Where are the Donuts?

Blended Learning Means Many Things Distance Education e-Learning Integrated Mixed Media Synchronous Asynchronous e-ILT Facilitated Learning Self-Directed Learning Learning by Doing

New Focus for Blended Solutions “ Blended learning focuses on optimizing achievement of learning objectives by applying the right learning technologies to match the right personal learning style to transfer the right skills to the right person at the right time” SkillSoft

Why Blended Learning? Improved Pedagogy Increased Access and Flexibility Increased Cost Effectiveness Social Interaction Personal Agency Ease of Revision From: The Handbook of Blended Learning SkillSoft

Benefits of Blended Learning Cognitive Rehearsal Emphasis on Context Rather Than Content Opportunity for Value Sorting Support for Longitudinal Learning Social Learning Tacit and Unstructured Learning From: “The Blended Learning Imperative” by Elliott Masie SkillSoft

Blended Learning Models Activity-Level Blending Distinct face-to-face and CM activities used as part of an activity at the discretion of the learner Course-Level Blending Distinct face-to-face and CM activities used as part of a course at the discretion of the learner Program-Level Blending Participants choose a mix between face-to-face and online courses or a combination between the two is prescribed by the program Institutional-Level Blending A concerted effort is made to enable learners to take advantage of both ends of the spectrum

Categories of Blended Learning Systems From: The Handbook of Blended Learning Allows for a radical transformation of the pedagogy (learners actively constructing knowledge in ways that were not possible without technology) Transforming Blends Incremental changes to the pedagogy without changing the way that teaching/learning occurs Enhancing Blends Focus on addressing issues of access and convenience (i.e. greater flexibility and different learning modalities) Enabling Blends

Challenges of Designing Blended Learning The Role of Live Interaction Role of Learner Choice and Self-Regulation Models for Support and Training Digital Divide Cultural Adaptation Balance Between Innovation and Production From: The Handbook of Blended Learning

Blueprints for Blended Learning

Eight Criteria for Building Blended Learning Programs Program Type Information Distribution Critical Information Transfer Skills and Competency Development Certification Program Culture-building Goals Audience Characteristics Budget Limitations Access to Resources Time Pressures Content Complexity Technology Platform From: Bersin (2006)

Blended Learning Program Definition Worksheet Consider the following: Business Initiative Program Goal Desired Performance Outcome Target Audience and Characteristics Program Details (Program Name, Timeframe for Delivery, etc.) Success Criteria Method of Measurement SkillSoft

How Business Professionals Learn Formal Learning Informal Learning - 70% of workplace learning* Peers Reports Career growth Mistakes Managers Manuals New skills Successes Coaches Books Degree Experience Mentors Knowledge Base Credential On-the-job Experts Internet Search Certification Practice & Use Ask Others Consult Reference Material Attend Formal Training SkillSoft

Learner-centric Learning The best expression of blended learning is in the context of blending formal and informal strategies from a learner-centric point of view. SkillSoft

Learning Styles Reading Seeing Hearing Watching Doing Teaching SkillSoft

Learning Style Inventory David Kolb (2005) Reflective Observation Active Experimentation Concrete Experience Abstract Conceptualization SkillSoft Converging Solving problems Making decisions Reasoning deductively Defining problems Being logical Accommodating Getting things done Leading Taking Risks Initiating Being adaptable and practical Assimilating Planning Creating models Defining problems Developing theories Being patient Diverging Being imaginative Understanding people Recognizing problems Brainstorming Being open-minded

The 8-Phase Adult Learning Model

How Adults Learn Tell Me Show Me Let Me Prepare Me Connect Me Support Me Coach Me Check Me SkillSoft

PREPARE ME to learn Communicate the benefits the learner will receive when he has mastered the new skill. This creates an openness to learning, because the learner perceives a potential personal gain in applying the skill you want him to master. This critical first phase affects the attitude with which the learner regards the discomfort of change. SkillSoft Prepare Me

PREPARE ME I Ideas and Strategies include: Explain the process Provide job and learning aids Schedule orientation and practice sessions Conduct a team-building session Use a virtual meeting tool Set up a communication protocol SkillSoft Prepare Me

TELL ME what is important Provide the learner with content that will cover basic principles, processes and concepts of your subject. During this phase, facts and information are conveyed, often in lecture format. It’s always important to establish or reinforce the knowledge base of your audience. Tell Me Prepare Me

TELL ME Ideas and Strategies include: Gaining attention (through storytelling, etc.) Inform learners of objectives Stimulate recall of prior learning (associate new information with prior knowledge) Separate content into smaller chunks Use visuals and animations Tell Me Prepare Me

SHOW ME what is expected Clearly demonstrate or illustrate the idea or procedure to the learner. Place the skill into a real-life context so they can understand how it can be applied. The demonstration Phase should utilize several senses so the learner can see, hear and possibly experience the actual event. Tell Me Show Me Prepare Me

SHOW ME Ideas and Strategies include: Basic demonstrations on real-life tasks, problems and scenarios Model behaviors and skills using strategies appropriate for the content Debrief to clarify key teaching points Chunk content into smaller pieces Mix it up for interest and effectiveness Tell Me Show Me Prepare Me

LET ME do it myself A natural ‘next Phase’ is to allow the learner to practice the new skill. This allows the learner to build confidence in applying and reapplying their newly acquired skills in a lifelike situation. At this stage of learning, a safe environment for practice is critical to remove anxiety and clear the way for skills retention. Tell Me Show Me Let Me Prepare Me

LET ME Ideas and Strategies include: Try out learning frequently and appropriately Choose practice strategies appropriate for the content and level of learning Use simulations when possible Tell Me Show Me Let Me Prepare Me

CHECK ME to validate Give feedback on tests and practice activities as soon as possible in order to confirm comprehension. Positive feedback builds the confidence needed to apply new skills in real situations. Constructive criticism sharpens the learner’s skills, by addressing learning gaps before they become habits. Tell Me Show Me Let Me Prepare Me Check Me

CHECK ME Ideas and Strategies include: Assess certification objectives Assess learning objectives Use pre-tests Use post-tests Consider other forms of assessment Tell Me Show Me Let Me Prepare Me Check Me

SUPPORT ME with resources Empower learners to continue their own development by recommending resources and materials to extend the learning experience. It is imperative that reference materials are organized so as to be easily searchable and retrievable. Tell Me Show Me Let Me Prepare Me Support Me Check Me

SUPPORT ME Ideas and Strategies include: Re-use assets from formal learning Expect and support correction of mistakes Expect and support expansion of learning Provide access to additional reference or support materials Tell Me Show Me Let Me Prepare Me Support Me Check Me

Have experts and mentors available to the learner during both the formal and informal phases of learning. This allows the learner to transition skills from knowledge alone to application. The designated coaches may be supervisors, more experienced peers or, in the case of technical subjects, experts in their field. COACH ME as I apply new skills Tell Me Show Me Let Me Prepare Me Support Me Coach Me Check Me

Ideas and Strategies include: Support the coaches as well as the coachees Meet regularly Tie coaching to development plans Incorporate job aids and other tools COACH ME Tell Me Show Me Let Me Prepare Me Support Me Coach Me Check Me

Provide the learner with access to a team or community that allows him to work with others who practice the same skills. Together, peer teams share ideas and experiences that build collective confidence. As they practice new skills, innovation and creativity benefit every individual. CONNECT ME to my peers Tell Me Show Me Let Me Prepare Me Connect Me Support Me Coach Me Check Me

Ideas and Strategies include: Optimize the use of technology Encourage ad hoc meetings Stress the value of online communities Don’t forget the non-technology based opportunities CONNECT ME Tell Me Show Me Let Me Prepare Me Connect Me Support Me Coach Me Check Me

Discussion What phases does your organization do particularly well? In which phases would you like to focus more of your efforts? 4 7 Tell Me Show Me Let Me Prepare Me Connect Me Support Me Coach Me Check Me

How does knowledge of the 8 Phases guide you in constructing Blended Learning programs? It helps you choose The right learning assets The right learning modality The right delivery mechanism The appropriate technology

Sample Blended Learning Program Course Content Job Aids Simulations Recorded Testimonial Online Chat Room Test Prep Virtual Meetings Mentors Prepare Me Tell Me Show Me Let Me Connect Me Coach Me Support Me Check Me

Means of Delivery Job aids Online Mentors Mentoring Coach me Job aids, Online books, courses Web sites, Phone, Chat Meetings with Experts Support me, Connect me Study Guides and Assessments Assessments and Feedback Observation, Tests, Feedback Check me Simulations and Case Studies Practice, OJT, Experts Labs, Practice, OJT, Experts Let me Courses, Online books, Papers and Recordings Presentations Presentations Tell me, Show me Self-Paced Virtual Face to Face Learning Phase/Method

The Blended By Design Blueprint

The Blended by Design Blueprint A structured process to guide you through the design of your blended learning program by addressing the eight dimensions of adult learning. SkillSoft

Blueprint Elements Learning Objective Type Learning Asset Medium Means of Delivery Existing Resources New Content Development Development Responsibility Content Subject Matter Expert SkillSoft

Learning Objective Type A learning objective is a brief, clear statement of what the learner should be able to do as a result of training/learning. An objective should look at how the learning relates to successful completion of the task or job. A well-written objective is specific and measurable. Determine if an objective meets these criteria: Understandable Trainable Demonstrable Attainable

Writing Measurable Objectives Three Parts of a Measurable Objectives Measurable objectives must include: 1. Performance (i.e. Behavior) - What should the student be able to do? Performance clearly states what the student is expected to be able to do or produce. Example: Stay under water for 10 minutes 2. Condition: Under what condition(s) should the student be able to do this? Conditions eliminate guesswork by the learner; add detail by describing each of the condition’s needs; make accurate measurement of student performance easier. Example: Given a list of key manager-related drivers that impact performance, discuss those you consider to be the most important 3. Criteria (i.e. Standard): How well must it be done? State the level of desired performance. Criterion should name appropriate ability, not a minimum level. Examples of criteria include speed, quantity, quality, or accuracy. Example: Be able to run the hundred-yard dash on dry, level track in 10 seconds

Developing Your Blended Learning Program

Three Major Steps in Building Blended Learning Programs Define the program strategy, learning plan, and activities (DEFINE) Build content components (DESIGN) Launch, Deliver and Manage the Program (IMPLEMENT) From: Bersin (2006)

Step 1: Defining Program Strategy (DEFINE) Stages of Instructional Design Identification of instructional goals Subordinate skill analysis Learner and context analysis Construction of performance objectives Identification of assessment techniques Identification of instructional strategies Determination of appropriate delivery media Course development Evaluation Dick and Carey ISD Model

Step 1: Defining Program Strategy (DEFINE) Clearly Define Your Training Objectives Identify the Skill Gaps of Learners and the Characteristics of your Audience Match the content needed to achieve your training objectives with the most appropriate mix of learning methods Incorporate sound instructional design that takes into account how adults learn Leverage Subject Matter Experts

Internal Assessment Does the leadership of your organization support blended learning? Does your organization have the technology infrastructure in place to support blended learning? If not, do you have the support to build it? Have staff members been identified to develop and manage your blended learning program? What is the total cost of the technologies involved in implementing your blended learning project?

Target Audience Assessment What are the knowledge and skill gaps of your learners? What is the size and distribution of your audience? Are your learners motivated to participate in and complete the program? How will you encourage your learners to make time to participate?

Learning Content Assessment What are the key learning objectives that need to be taught? What instructional strategies and methods will best support your learning objectives? What level of interactivity does the learning material require? What methods of instruction will best meet the needs of your learners? Can you make updates to the content of your blended learning programs quickly and effectively? How will you assess and track learner progress and completion? How quickly does the training need to be up and running?

Step 2: Building Content Components (DESIGN) Some objectives are best taught in a traditional classroom (kinesthetic task-based objectives) Determine the need for making experts available in a live format (without face-to-face interaction) Objectives requiring memorization and knowledge absorption might lend themselves best to self-directed approaches (i.e. lecture format with minimal questions or clarifications) Always approach a blended learning initiative from a fresh perspective and don’t assume it will take less time to redesign an existing traditional program than it would to design a brand new blended program From: “Why Blended Learning Hasn’t Yet Fulfilled Its Promises” by Jennifer Holman

Step 3: Program Management and Delivery (IMPLEMENT) Establish support from leadership and managers Communicate importance and urgency of learning program Teach learners how to learn online Provide appropriate technology and technical support to learners Keep learners engaged and motivated by facilitating and supporting the program

Step 3: Program Management and Delivery (IMPLEMENT) The most important phase, as you are delivering value to the business. Program management elements might include: Marketing and Launching the Program Recruiting and Training Coaches Identifying roles for the learner’s Managers Learner notification and managing cohorts Assigning SMEs to help facilitate and support learners Tracking and monitoring discussion rooms and exercises Answering questions and providing daily updates to learners Reporting on completion or certification

Step 3: Program Management and Delivery (IMPLEMENT) (cont’d) To prepare to manage a blended program, facilitators should: Participate in as many blended learning experiences as possible Learn how to use all of the technologies Learn how to facilitate the individual technologies Practice managing the blend

Roadblocks to Blended Learning Initiatives DEFINE and DESIGN Creating programs without using a formal design process Assuming that redesigning an existing program is easier than starting from scratch (oversimplification) Stringing together stand-alone components into a learning path instead of truly weaving learning experiences together From: “Why Blended Learning Hasn’t Yet Fulfilled Its Promises” by Jennifer Holman

Roadblocks to Blended Learning Initiatives (cont’d) IMPLEMENT Overemphasizing the live components and undervaluing the self-directed components of the blend Lack of experience on the part of the training team in facilitating the blend No formal training for the implementation team Lack of organizational understanding and support for blended programs Inexperienced learners who have not been taught how to learn online

Organizational Readiness for Blended Learning Being Ready Provide Technology Training to facilitators, participants, and technical support staff Ensure participant desktops have necessary hardware, software and bandwidth Train facilitators to manage the blend effectively Provide a learner orientation to teach participants how to be successful online learners in your organization Being Willing Publish success stories to motivate participants Create early successes for participants by starting with less intensive blends

Organizational Readiness for Blended Learning (cont’d) Being Able Participants need to be able to feel as if they can be successful at desktop learning and will not be interrupted by coworkers or managers Participants need to be recognized for successful completion by the learning being recognized in performance reviews Facilitators need the time to prepare to teach and manage the self-directed components of the blend Designers need time to pilot and continuously revise designs to ensure the program meets it stated objectives

Future Trends and Predictions Mobile Blended Learning Greater visualization, individualization, and hands-on learning Self-determined blended learning Increased connectedness, community, and collaboration Increased authenticity and on-demand learning Linking work and learning Changed calendaring Blended learning course designations Changed instructor roles The emergence of blended learning specialists

Future Trends and Predictions (cont’d) Authentic Learning Real-Time Workflow Learning Work-Embedded Learning More Emphasis on Interaction Learning in Virtual Environments

SkillSoft Blended Learning Services Determine Customer’s Unique Needs Needs assessments Participant selection surveys Define success criteria Select the Right Learning Assets Proprietary SkillSoft Third party ILT Create and Manage a Program Schedule Business needs Management needs Participant needs

BLS Program Management Communication/Marketing Obtain support Executives Management Participants Set expectations Logistics Kick-off presentations Program Guides First-level support Monitor and Motivate Progress reports Personal contact via phone/email

BLS Programs Industry Certification Cisco; CCNA, CCNP, CCDP, etc. Microsoft; MCP, MCSE, MCSA, MCDBA, etc. PMP, CAPM Lean Six Sigma ITIL v2,v3 Proprietary Certifications Business/Professional Skills Leadership Management Sales Customer Service

Resources The Blended Learning Book (2004) by Josh Bersin The Blended Learning Handbook (2006) edited by Curtis Bonk and Charles Graham Informal Learning (2007) by Jay Cross “ The Eight Key Steps of Blended Learning Whitepaper” (2005) by SkillSoft Corporation “ Writing Measurable Objectives” (2003) by SkillSoft Corporation

Questions? Thank you! Calvin McKnight Director, Solution Services ph: 212.233.5004 email: calvin_mcknight@skillsoft.com

Evaluation Framework Adapted from: The E-Learning Fieldbook by Nick van Dam

How do you know you’re RIGHT? What outcomes are you expecting? Who are program stakeholders? What do they expect? How will you know when you’ve achieved targeted outcomes? Think about outcomes first Stay on track to achieve outcomes SkillSoft

Research shows that Evaluation should… Be objective and directed at important outcomes Identify the important elements of the training program Match the organizational philosophy Be reasonable Focus on both the outcomes and the process.

Successful Evaluation Requires: A commitment to training and an important and central business function rather than an optional or nonessential activity; The adoption of a disciplined and active planning approach rather than a reactive manage style distracted by putting out fires; The allocation of valuable management time to a careful consideration of what is to be achieved and measured, and how to measure it, before training is planned; The commitment of time and resources to detailed analysis.

Benefits of Evaluation Broadly, evaluation can tell you whether what you have done (summative) has worked or if what you are working on (formative) is working. It confirms that financial resources have targeted identified priorities Determine if desired improvements in individual performance have been achieved. Helps improve performance if objectives are not achieved. Clearly shows the impact of training on the organization and provides information about performance to use in planning future training.

What is your evaluation strategy?

Who are Donald & Jack? Reaction Learning Job Impact Organizational Impact ROI

Kirkpatrick Evaluation Levels 1. Reaction How the learner (customer) responds to the training experience

REACTION Donald L. Kirkpatrick Determine what you want to find out. Design a form that will quantify reactions. Encourage written comments and suggestions. Attain an immediate response rate of 100%. Seek honest reactions. Develop acceptable standards. Measure reactions against the standards and take appropriate action. Communicate the reactions, as appropriate.

A Training Survey Example Training Satisfaction Were you prepared to perform tasks required on your job? O Completely O Very Well O Adequately O Somewhat O Not At All Was there sufficient practice included in the training? O Plenty O More Than Enough O Adequate O Needed More O Too Little Were the training materials clear and to the point? O Very Good O Better Than Average O Average O Poor O Very Poor Would you recommend this course to others? O Highly Recommend O Recommend O Neutral O Not Recommend Are you confident in your performance following training? O Very O Reasonably O Somewhat O Not Very O Not At All Did this training meet your expectations? O Completely O Mostly O Somewhat O Less Than Expected O Did Not Meet

Kirkpatrick Evaluation Levels 2. Learning Proof that all training objectives are met

LEARNING Donald L. Kirkpatrick Use a control group, if feasible. Evaluate knowledge, skills, or attitudes both before and after the training. Attain a response rate of at least 60%, if possible. Use the results of the evaluation to take appropriate action.

Requirements for Measuring Learning Your learning outcomes must be stated in measurable training objectives (visible performances). You must evaluate your entry-level learner by using a preassessment made up of valid items that measure the training objectives. You must provide practice during instruction that allows the learner to increase their level of knowledge and skills in order to master the training objectives. You must post-assess the measurable objectives with valid items that allow the learner to demonstrate mastery of the training objectives.

Kirkpatrick Evaluation Levels 3. Behavior Proof that learner performance has changed because of training

BEHAVIOR Donald L. Kirkpatrick Use a control group, if feasible. Allow enough time for a change in behavior to take place. Survey or interview one or more of the following groups: trainees, their bosses, their subordinates, and others who often observe trainees’ behaviors on the job. Choose an adequate sample Repeat the evaluation at appropriate times. Consider the cost of evaluation versus the potential benefits.

Requirements for Measuring Behavior If possible, you must assess “on-the-job” skills before you provide training on new skills or ask learners to estimate before training performance Then, introduce new skills through training. The training experience (practice and assessment) should include real-world application of the content. You should assess the learner on facts, concepts, and procedures learned during the training. When possible, you should assess training transfer in the actual job setting. You can use focus groups or post-training surveys to see if training content is being used on the job.

Kirkpatrick Evaluation Levels 4. Results Productivity, decreased costs, higher quality, better time to market

RESULTS Donald L. Kirkpatrick Use a control group, if feasible. Allow enough time for results to be achieved. Measure both before and after training, if feasible. Repeat measurement at appropriate times. Consider the cost of evaluation versus the potential benefits. Be satisfied with the evidence if it is not possible to attain absolute proof.

Requirements for Measuring Results You should have both qualitative and quantitative data showing the effectiveness of the training and the application of this training to real job tasks. You must measure the same areas before and after training to show a demonstrable difference.

Phillips ROI Evaluation Jack J. Phillips “ Return on Investment in Training and Performance Improvement Programs” Reaction Level 1 (Smile Sheets) Learning Level 2 (Meet Objectives) Behavior Level 3 (Adopt New Behaviors) Results Level 4 (Increase Performance) Return on Investment Level 5

Bx D Presentation Cstd Hamilton

More Related Content

What's hot (20)

Similar to Bx D Presentation Cstd Hamilton (20)

Bx D Presentation Cstd Hamilton

Editor's Notes