Experiment on Knowledge Acquisition
Download as PPTX, PDF2 likes709 views
The document outlines the objectives of a user experience review for the AuraWiki project, focusing on the authoring of 'universal truths' (UTs) from textbook sentences by subject matter experts. It details the training and process for creating UTs, emphasizing the importance of context-independent, precise statements that convey single facts. The document also includes evaluation methods for the functionality of the application and feedback from users to enhance the authoring process.
Experiment on Knowledge Acquisition
- 1. Subject Matter Expert - User Experience Review AuraWiki Jesse Wang, Wil Smith Oct 2012
- 2. Project Goals “Crowd Source UT Authoring” Can Subject Matter Experts Author Useful Universal Truths? Can We Speed Up Encoding a Textbook with Input from Subject Matter Experts? Can We Create a UT Authoring Portal for Multiple Textbooks? Can Existing Social Networks Provide Subject Matter Experts Capable of UT Authoring? Could Gamification be Applied to An Existing Portal to Add Non-Domain Experts?
- 3. Meeting Goals This is an end user meeting and we are doing it to hear your comments on the current system. The team wants to explain the authoring process, and demonstrate the current application state. Before we continue development on the project we need to review how the application, in the unfinished state, performs with our targeted user base. MID PROJECT FEEDBACK. We do these meetings when releasing production products to ensure the final application complies with customer needs and is not “the engineering best way”.
- 4. Review Schedule Phase Method Time required Introduction & Training • Goals of Project Halo and Aura 0:45 Wiki • Training for UT Authoring • What is a UT? • What is a concept / context? • How do I formulate UTs • System demonstration & hands-on Assignment: UT Authoring • You work by yourself using our 1:30 system • Support is provided by us Questionnaire & Discussion • Online questionnaire 0:45 • Group discussion
- 5. Subject Matter Expert - User Experience Review How do we author a universal truth?
- 6. From Sentence to Universal Truths What is a UT? How to write UTs for a sentence What it a concept? How to choose concepts for a UT
- 7. What is a UT? There is no exact definition of “Universal Truth”, but a good UT… expresses a single fact (which can be compound, but must be indivisible). is as simple as possible, but not simpler (to quote Einstein). is context-independent, i.e. meaningful on its own, without surrounding sentences or other context. is universally true. For example, a UT could start with "All Plants..." but not with "Some plants...". is unambiguous. is precise.
- 8. How to Write UTs for a Sentence For each relevant sentence, write one or more UTs There is no one correct set of UTs for a given sentence. The source sentences are open to interpretation, and it is up to you to determine what the most appropriate UTs are The interpretation of the source sentences, and the paraphrasing into UTs, should be guided by your knowledge of the subject matter (and by common sense) The following methods are not exhaustive. When in doubt, refer back to the definition of a “good UT”.
- 9. How to Write UTs for a Sentence In the simplest case, the sentence has only one UT, and it is exactly the same as the UT. Sentence: Matter is made up of elements. UT: Matter is made up of elements.
- 10. How to Write UTs for a Sentence Sometimes part of a sentence is irrelevant, even though the sentence as a whole has been deemed relevant. In such cases the irrelevant part is simply dropped. Sentence: Some symbols are derived from Latin or German; for instance, the symbol for sodium is Na. UT: Sodium has the symbol Na.
- 11. How to Write UTs for a Sentence Often, a sentence contains several facts. In this case, the sentence needs to be split into several UTs. Sentence: Pure sodium is a metal, and pure chlorine is a poisonous gas. UT: Pure sodium is a metal. UT: Pure chlorine is a poisonous gas.
- 12. How to Write UTs for a Sentence When a reference like "this", "that", "these", "those", "which", etc occurs in a source sentence, they should be replaced with the referent (the word that "this","that", etc refers to). Sentence: Organisms are composed of matter, which is defined as anything that takes up space and has mass UT: Organisms are composed of matter UT: Matter has mass and volume Here, "which" clearly refers to "Organisms", and "is defined as" is unnecessarily verbose, and thus simplified.
- 13. How to Write UTs for a Sentence Sometimes the referent is in a previous sentence in the same paragraph. Sentence: These selective channels affect the rate at which water moves osmotically across the membrane UT: Aquaporins are selective channels UT: Aquaporins affect the rate at which water moves osmotically across the membrane
- 14. How to Write UTs for a Sentence There are many other words that are used as linguistic flourishes or to bind sentences in a paragraph together and give the text some "flow". Such words ("another" in the example below) should be eliminated, but can sometimes be used to extract or infer "incidental" information from a sentence. Sentence: Water (H20), another compound, consists of the elements hydrogen (H) and oxygen (O) in a 2:1 ratio UT: Water consists of hydrogen and oxygen in a 2:1 ratio UT: Water is a compound UT: Hydrogen has symbol H UT: Oxygen has symbol O Here, the first UT contains the "main" information of the sentence, and the last three are "incidental".
- 15. How to Write UTs for a Sentence Ideally, a UT should not contain words like "sometimes", "usually", "most", "some", "generally", etc. There are a couple of ways to handle them. One way is to generalize the weaker statement into a universal statement, usually just by dropping the word in question. Sentence: Branching generally enables plants to harvest sunlight for photosynthesis more effectively. UT: Branching enables plants to harvest sunlight for photosynthesis more effectively.
- 16. How to Write UTs for a Sentence Another method of eliminating "some" etc is to instantiate the sentence by providing a name for the case where the sentence is true. Sentence: Some prokaryotes even contain a row of tiny magnetic particles that allow the cells to orient in Earth's magnetic field UT: Magnetotactic bacteria contain a row of tiny magnetic particles that allow the cells to orient in Earth's magnetic field Here we replaced "some prokaryotes" with "magnetotactic bacteria". We only do this when the new name occurs somewhere in the textbook.
- 17. How to Write UTs for a Sentence "some", etc are allowed when they do not refer to the main concept of the sentence (see "Choosing Concepts for a UT", below). For example, this is a good UT: UT: In vascular plants, some substances may use more than one route The UT as a whole states a fact about all vascular plants (the main concept), and "some" refers to substances within all vascular plants. One way to remember this distinction is to keep in mind that Universal Truths are just that -- universally true statements, i.e. statements that are always true of some type of entity, event, etc. If it is possible, we can also use Instantiation (see previous slide) in this type of case.
- 18. How to Write UTs for a Sentence When "for example", "such as", etc occurs in a sentence, encode the specific information in the example. Sentence: Some plants have horizontally oriented leaves; others, such as grasses, have leaves that are vertically oriented. UT: Grass has vertically oriented leaves As discussed above, the "some" part of the sentence is problematic.
- 19. How to Write UTs for a Sentence In other cases, the "for example" does not add any new information. Paragraph: We symbolize atoms with the same abbreviation used for the element that is made up of those atoms. For example, the symbol C stands for both the element carbon and a single carbon atom. Here, the second sentence does not add any information, since its informational content is subsumed by the general "rule" of the first sentence (assuming we have already encoded that carbon has the symbol C).
- 20. How to Write UTs for a Sentence Conditional statements should usually be rephrased to a non- conditional form. Sentence: If we place a cell in a solution that is hypotonic to the cell, water will enter the cell faster than it leaves UT: A cell in a hypotonic solution will gain water faster than it loses it Here, the condition has been changed into a context, "in a hypotonic solution". See also the section on "Contextual Universals" below.
- 21. Review: What is a UT? Or rather a good UT… expresses a single fact (which can be compound, but must be indivisible). is as simple as possible is context-independent, i.e. meaningful on its own, without surrounding sentences or other context. is universally true. For example, a UT could start with "All Plants..." but not with "Some plants...". is unambiguous. is precise.
- 22. Summary: Methods of UT Authoring Copy, remove (partial, irrelevant info), splitting Reference resolution (from context) Incidental extraction e.g. “Water, another compound, …” Specialization, generalization Instantiation E.g. “some prokaryotes” “magnetotactic bacteria” Example-encoding E.g. “some plants…” “grass” Condition to context
- 23. What is a Concept? A concept is a noun or a noun phrase that describes a type of entity or event, in singular form You may need to create a new concept for a UT if an appropriate one does not already exist. Good concept names: Cell, Eukaryotic Cell, Cellular Respiration Bad concept names: Cells, The eukaryotic cell, A plant
- 24. Choosing Concepts for a UT The main concept of a UT is the entity or event that the UT is about. It is usually the grammatical subject of the UT UT: All plant cells have a nucleus Main concept: Plant Cell Note that "All plant cells" is not a good concept, because it is not in singular form. UT: Photosynthesis produces oxygen Main concept: Photosynthesis
- 25. Choosing Concepts for a UT Some UTs are true in a specific context. In such cases, also choose a contextual concept –contextual concept is OPTIONAL The contextual concept usually follows a word such as "in", "during", "while", etc. UT: The phospholipids in a membrane move laterally Main concept: Phospholipid Contextual concept: Membrane Note that "the" or "a" is never part of the concept. UT: The oxygen produced during photosynthesis is used during cellular respiration Main concept: oxygen Contextual concept: Photosynthesis UT: Cellular respiration in eukaryotes happens in the mitochondria Main Concept: Cellular Respiration Contextual Concept: Eukaryote
- 26. Bring your Best, then Move On We hope you got good understanding of what you’re supposed to do Give it your best shot, it does NOT need to be perfect Do not spend too much time on one sentence/UT, move on if it’s too hard Feel free to work on sentences or UT others started Feel free to ask us if you’re really confused or not sure
- 27. Subject Matter Expert - User Experience Review Aura wiki tutorial
- 28. Our Solution: Aura Wiki Contents: A few chapters of Campbell Biology Without figures and using a simplified layout Purpose Use the power of crowdsourcing and online collaboration Let users „translate“ sentences from the textbook into UTs.
- 29. Sample wiki page for the textbook
- 30. Sample wiki page for a whole section
- 31. Sample wiki page for a sentence
- 32. Creating UTs The translation of sentences to UTs is done on the sentence page. The sentence page can display the paragraph surrounding the sentence (or even more context) if needed for understanding and formulating the UT. Under the heading Universal Truths you can see all UTs that exist for this sentence and you can add yor own.
- 33. Sample UTs presented in the sentence page
- 34. Creating UTs – Add new UT
- 35. Creating UTs – Input Form Input form for new UT. First two inputs are required.
- 36. Creating UTs - Autocompletion Autocompletion is automatically triggered and suggests existing concepts
- 37. Subject Matter Expert - User Experience Review Subject Matter Expert Assignment
- 38. The Assignment Now it is your turn Go to your User page There is a list of sentences assigned to you Once you have added UTs they will be displayed there together with the sentences (after refreshing the page) At the top of your user page, you will find links to help- texts, some sample UTs and the questionnaire for after the session.
- 39. The Assignment This link always brings you back to your user page Help documents and a list of example UTs Sentences (and UTs)
- 40. The Assignment IMPORTANT This evaluation is conducted in order to assess the system, not to assess your work as a SME. You are supporting the evaluation, but you are not a test subject! Please work with the system at your own convenience. You do not need to achieve any quantitative or qualitative goal. (This is of course no excuse for being lazy ) You have pencils and paper available – if you encounter any problems or have ideas you think are worth mentioning, feel free to take a short note so you can bring it up during the discussion later If there are questions please ask them anytime
- 41. Subject Matter Expert Subject Matter Expert - User Experience Review Review
- 42. Questionnaire Go to your user page There is a link to the online questionnaire at the top In the first question, select the username you were assigned today Fill out the entire questionnaire. Don’t forget to submit at the end
- 43. Discussion – System Functionality Did you have trouble following the UT authoring workflow? If so, where and why? What changes would you apply to the existing functionality and/or workflows to increase productivity and ease of use? What new functionality could you think of that might have a positive influence on your work?
- 44. Discussion – UT Authoring Did you understand the notion of UT and concept? Did the training sufficiently explain the meaning of UT and concept? Do you feel the training you received in the beginning and the help materials on the site were sufficient for executing your task? If not, what would you improve? Which areas need better coverage?
- 45. Subject Matter Expert - User Experience Review Conclusion
- 46. Thank you. Contact: Jesse Wang (jessew@vulcan.com) William Smith (wills@vulcan.com )