Computing for Human Experience [v3, Aug-Oct 2010]

1. Ohio Center of Excellence on Knowledge-Enabled Computing (Kno.e.sis)NCSA Director’s Seminar, UIUC, Oct10. Talk at IBM Almaden, HP Labs, DERI-Galway, City Uni of Dublin, KMI-Open U, AugSep10.

2. Ohio Center of Excellence on Knowledge-Enabled Computing (Kno.e.sis)scientiapotentiaestKnowledge is Power Francis Bacon, 1597…established and popularized deductive methodologies for scientific inquiry2

3. Kno.e.sis’ leadership in semantic processing will contribute to basic theory about computation and cognitive systems, and address pressing practical problems associated with productive thinking in the face of an explosion of data.Kno.e.sis: from information to meaning.Kno.e.sis Vision3

5. Significant InfrastructureWhole-Body Laser Range ScannerVERITASstereoscopic 3D visualizationNMRAVL5

6. Exceptional Regional CollaborationAt least 6 active projects with AFRL/WPAFB

7. Human Effectiveness Directorate

8. Sensors Directorate 6

9. Exceptional National CollaborationUniv. of Georgia, Stanford, Purdue, OSU, Ohio U., Indiana U. UC-Irvine, Michigan State U., Army

10. Microsoft, IBM, HP, Yahoo 7

11. ExceptionalInternational CollaborationW3C Member: WSDL-S/SAWSDL, SA-REST, SSN, …

12. U. Manchester, TU-Copenhagen, TU-Delft, DERI (Ireland), Max-Planck Institute, U. Melbourne, CSIRO, DA-IICT (India)8

13. FundingCurrent active funds of ~$10 million (supporting research of 15 faculty and 45+ funded grad students & postdocs).NIH, NSF, AFRL, …MSR, IBM-R, HP Labs, Google9

14. FacultyOver 2,000 citations per faculty, around 1,000 refereed publications – comparable to any excellent group; granted 74 PhDs, $50 million in cumulative funding

15. Prof. Sheth among the most cited Computer Science authors in the world today(h-index 65, 10th in citation in WWW area:cf Microsoft Academic Search)

16. Prof. Bennett & Flach’s paper declared as one of most influential papers published in over 50 years in Journal of Human Factors; Prof. Raymer’s paper was cited in a US Supreme Court decision

17. Kno.e.sis has attracted top-notch faculty

18. High quality funding: NIH, NSF, AFRL…..innovation grants: Microsoft Research, Google, IBM Research, HP labs

19. Entrepreneurship experience – launched several companies10

20. World Class StudentsMeena Nagarajan gave a keynote at an international workshop– unheard of for a PhD student.

21. Satya, Cory, Karthikorganized international workshops

22. Satya joining CWRU (tenure track), Meena – IBM Almaden

23. Six of the senior PhD students: 80+ papers, 40+ program committees, contributed to winning NIH and NSF grants.

24. Students interned at & collaborated with the very best places: Microsoft Research, Yahoo! Research, IBM Research, HP Labs, NLM, Accenture Labs, …and filed for 6 patents in 3 years11

25. Computing for Human Experience: SemanticsempoweredSensors, Services, and Social Computing on ubiquitous WebSemantic Provenance: Trusted Biomedical Data IntegrationAmit ShethLexisNexis Ohio Eminent ScholarWright State University, Dayton OHhttp://knoesis.orgThanks: Meena, Cory, Kats & Kno.e.sis team

26. For Semantic-skepticsMicrosoft purchased Powerset in 2008Apple purchased Siri [Apr 2010]“Once Again The Back Story Is About Semantic Web”Google buys Metaweb [June 2010]...” Google Snaps Up Metaweb in Semantic Web Play”FacebookOpenGraph, Twitter annotation …”another example of semantic web going mainstream” “Google, Twitter and Facebook build the semantic web”RDFa adoption ….Search engines (esp Bing) are about to introduce domain models and (all) use of background knowledge/structured databases with large entity basesKno.e.sis is the largest US academic group in terms of # of faculty and PhD students in Semantic Web/Web 3.0 area (semantics enhanced services, cloud, social and sensor computing/Webs)13

27. Semantic Search etc.A Bit of HistorySYSTEM AND METHOD FOR CREATING A SEMANTIC WEB AND ITS APPLICATIONS IN BROWSING, SEARCHING, PROFILING, PERSONALIZATION AND ADVERTISING [Filed 3/2000, Granted 5/2001] More in this 2000 keynote: Semantic Web and Information Brokering: Opportunities, Commercialization and Challenges14

28. 15

29. Semantics as core enabler, enhancer @ Kno.e.sis16

30. imagine

31. imagine when

32. meets Farm Helper

33. with thisLatitude: 38° 57’36” NLongitude: 95° 15’12” WDate: 10-9-2007Time: 1345h

34. that is sent to Sensor Data ResourceStructured Data ResourceWeather ResourceAgri DBSoil Survey Weather DataServices ResourceLocationDate /TimeGeocoderWeather dataLawrence, KSLat-LongFarm HelperSoil InformationPest information …

35. and

36. Six billion brains

37. imagination today

38. impacts our experience tomorrow

40. 27Seamless integration of technology with life*“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life ... Machines that fit the human environment instead of forcing humans to enter theirs...” Mark Weiser, The Computer for the 21st Century (Ubicomp vision)“…technology that will allow us to combine what we can do on the Internet with what we do in the physical world.” Ian Pearson in Big data: The next Google http://tarakash.com/guj/tool/hug2.html* From Devices to Ambient Intelligence

41. 28But we are not just talking aboutIntelligent DesignUbicomp: Mark Wisner and othersIntelligence @ Interface: Gruber –“the system knows about us, our information, and our physical environment. With knowledge about our context, an intelligent system can make recommendations and act on our behalf.”(c) 2007 Thomas Gruber

42. 29What is CHE? Beyond better human interactionComputing for human experience will employ a suite of technologies to nondestructively and unobtrusively complement and enrich normal human activities, with minimal explicit concern or effort on the humans’ part.FeaturesSeamless - nondestructive and unobtrusive, with minimal explicit concern or effort on part of humansanticipatory, knowledgeable, intelligent, implicit, ubiquitousEncompasses: Mobile (ubiquitous) Web, Sensor (multisensory and participatory) Web, Social Web (collective intelligence and wisdom of the crowd), multimodal

43. 30Learning from a number of exciting visions

44. Web (and associated computing) is evolvingWeb ofpeople, Sensor Web - social networks, user-createdcasualcontent - 40 billionsensorsWeb of resources - data, service, data, mashups - 4 billionmobilecomputingWeb of databases - dynamically generated pages - web query interfacesWeb of pages - text, manually created links - extensive navigationComputing for Human ExperienceWeb as an oracle / assistant / partner - “ask the Web”: using semantics to leverage text + data + services - Powerset Enhanced Experience,Tech assimilated in life2007Situations,EventsSemantic TechnologyUsedObjectsPatternsKeywords1997

45. Four enablers of CHEBridging the Physical/Digital DivideElevating Abstractions That Machines & Humans Understand: signals to observations to perceptionFrom Social Perception to Semantics (meaningful to other humans/observers and machine – shared, computable; crowd sourcing)Semantics at an Extraordinary ScaleMore in Computing for Human Experience, IEEE IC, Jan-Feb 2010.32

46. 33Physical-Cyber divide is narrowingPsyleron’s Mind-Lamp (Princeton U), connections between the mind and the physical world. Neuro Sky's mind-controlled headset to play a video game.IoTEmotion SensorsWearable SensorsBody Area NetworksSixth Sense- Gesture Computing and wearable device with a projector for deep interactions with the environment

47. 34Sensors everywhere ..sensing, computing, transmitting2009: 1.1 billion PCs, 4 billion mobile devices, 40+ billion mobile sensors (Nokia: Sensing the World with Mobile Devices)6 billion intelligent sensorsinformed observers, rich local knowledgeChristmas Bird Count

48. 35Relevant happenings todaythat all objects, events and activities in the physical world have a counterpart in the Cyberworld(IoT)multi-facted context of real world is captured in the cyberworld(multilevel & citizen sensors/participatory sensing)each object, event and activity is represented with semantic annotations (semantic sensor web)for a chosen context, with an ability to explicate and associate variety of relationships and events (background knowledge, Relationship Web, EventWeb)

49. appropriate reasoning and human/social interaction are available and applied, insights extracted (semantic web, social semantic web, experiential computing)

50. Activity anticipated/answers obtained/ decisions reached/communicated/appliedElevating Abstractionsthat Machines & Humans Understand: signals to observations to perception that lead to semantics (provide meaning and understanding)

51. People Web(human-centric)Sensor Web(machine-centric)Observation(senses)Observation(sensors)Perception(analysis)Perception(cognition)Communication(language)Communication(services)

52. Enhanced Experience (humans & machines working in harmony)ObservationPerceptionCommunicationSemantics for shared conceptualization and interoperability between machine and humanSemantics to improve communicationabout shared spaces, events,…

53. Semantic Sensor Web Infrastructure

54. 40Semantically Annotated O&M<om:Observation> <om:samplingTime><gml:TimeInstant>...</gml:TimeInstant></om:samplingTime> <om:procedurexlink:role="http//www.w3.org/2009/Incubator/ssn/ontologies/SensorOntolgy.owl#Sensor“xlink:href="http//www.w3.org/2009/Incubator/ssn/ontologies/SensorOntolgy.owl#sensor_xyz"/> <om:observedPropertyxlink:href="http//www.w3.org/2009/Incubator/ssn/ontologies/SensorOntolgy.owl#temperature"/> <featureOfInterestxlink:href="http://sws.geonames.org/5758442/"/> <om:resultuom="http//www.w3.org/2009/Incubator/ssn/ontologies/SensorOntolgy.owl#fahrenheit">42.0</om:result></om:Observation>

55. Semantic Sensor ML – Adding Ontological MetadataDomainOntologyPersonCompanySpatialOntologyCoordinatesCoordinate SystemTemporalOntologyTime UnitsTimezone41Mike Botts, "SensorML and Sensor Web Enablement," Earth System Science Center, UAB Huntsville

56. Kno.e.sis’ Semantic Sensor Web42

57. Scenario of context (Semantic Sensor Web demo)More: Semantic Sensor Web

58. Active Perception (Sensing – Observation – Perception) and role of ontologies and background knowledge for Situational Awareness

59. To enable situation awareness on the Web, we must utilize abstractions capable of representing observations and perceptions generated by either people or machines.Webobserveperceive“real-world”45

60. For example, both people and machines are capable of observing qualities, such as redness.observesObserverQuality* Formally described in a sensor/observation ontology46

61. 47With the help of sophisticated inference, both people and machines are also capable of perceiving entities, such as apples.perceivesPerceiverEntity* Formally described in a perception ontology

62. 48The ability to perceive is afforded through the use of background knowledge. For example, knowledge that apples are red helps to infer an apple from an observed quality of redness. Quality* Formally described in a domain ontologyinheres inEntity

63. 49The ability to perceive efficiently is afforded through the cyclical exchange of information between observers and perceivers. Observer* Traditionally called the Perception Cycle (or Active Perception)sendsfocussends perceptPerceiver

64. 50Integrated together, we have an abstract model – capable of situation awareness – relating observers, perceivers, and background knowledge.observesObserverQualitysends perceptsendsfocusinheres inperceivesPerceiverEntityWhat is new? Relevant background knowledge/ontologies are increasingly available or possible to create. Domain independent ontologies are being developed or exist… Web, scale….

65. Let’s review an example execution of the perception cycle, utilizing background knowledge from the weather domain.51

66. Background knowledge from weather ontologyQualityinheresInEntityBlizzardFreezing TemperatureNot Freezing TemperatureFlurrySnow PrecipitationRain StormRain PrecipitationNo PrecipitationRain ShowerHigh Wind SpeedClearLow Wind Speed52

67. Example execution of the Perception Cyclesensor-observation ontologyobserved qualitiesobserveshigh wind speedsnow precipitationObserverQualityFocus: Percept:inheres inperceptual theoryEntityPerceiverperceives clear blizzard flurry rain shower rain stormperception ontology53

68. Perceiver sends ‘wind-speed’ focus to observerobserved qualitiesobserveshigh wind speedsnow precipitationObserverQualityFocus:wind-speed Percept:inheres inperceptual theoryperceives clear blizzard flurry rain shower rain stormPerceiverEntity54

69. Observer observes ‘high wind-speed’observed qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:wind-speed Percept:inheres inperceptual theoryperceives clear blizzard flurry rain shower rain stormPerceiverEntity55

70. Observer sends ‘high wind-speed’ percept to perceiverobserved qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:wind-speed Percept:high wind-speedinheres inperceptual theoryperceives clear blizzard flurry rain shower rain stormPerceiverEntity56

71. Perceiver perceives either ‘clear’, ‘blizzard’, or ‘rain storm’ observed qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:wind-speed Percept:high wind-speedinheres inperceptual theoryperceives clear blizzard rain stormPerceiverEntity57

72. Perceiver sends ‘precipitation’ focus to observerobserved qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:precipitationPercept:high wind-speedinheres inperceptual theoryperceives clear blizzard rain storm PerceiverEntity58

73. Observer observes ‘snow precipitation’observed qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:precipitationPercept:high wind-speedinheres inperceptual theoryperceives clear blizzard rain storm PerceiverEntity59

74. Observer sends ‘snow precipitation’ percept to perceiverobserved qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:precipitationPercept:snow precipitationinheres inperceptual theoryperceives clear blizzard rain storm PerceiverEntity60

75. Perceiver perceives ‘blizzard’observed qualitiesobserveshigh wind-speedsnow precipitationObserverQualityFocus:precipitationPercept:snow precipitationinheres inperceptual theoryperceives blizzardPerceiverEntity61

76. Reference OntologiesSensor-Observation Ontology SSN-XG Sensor Ontology [1]

77. Kuhn’s Ontology of Observation [2]

78. etc. [3]observesinheres inDomain Ontology: WeatherKno.e.sis’s Weather Ontology (derived from NOAA) [5]sends perceptsends focusPerception OntologyKno.e.sis’s Perception Ontology [4]perceives62

79. Implementation of Perception CycleTrustStrengthened Trust63

80. Trust - Background(What?) *Trust is the psychological state comprising a willingness to be vulnerable in expectation of a valued result.

81. (Why?) To act on a decision, it’s important to have confidence in the information from which it was derived.

82. (How?) Through reputation, past behavior can be used to predict future behavior.*Ontology of Trust, Huang and Fox, 2006Josang et al’s Decision Trust

83. Trust values converge with the increasing number of observations.What next?Trillions+ of observations (data) from human body to all devices and everything on the Globe -> What’s meaningful to human experience?66Body Sensor NW: http://www.enterprise.mtu.edu/im/projects.htmlhttp://www.opengeospatial.org/ogc/markets-technologies/swe

84. mumbai, india

85. november 26, 2008

86. another chapter in the war against civilization

87. and

90. the world saw itThrough the eyes of the people

91. the world read itThrough the words of the people

92. PEOPLE told their stories to PEOPLE

93. A powerful new era in Information dissemination had taken firm ground

94. Making it possible for us tocreate a global network of citizensCitizen Sensors – Citizens observing, processing, transmitting, reporting

95. Social Perceptions, Observations, (semantic) Sense Making

96. Varied Social Perceptions lend to Varied ObservationsThe Health Care Reform Debate

97. Zooming in on Washington

98. Summaries of Citizen ReportsRT @WestWingReport: Obama reminds the faith-based groups "we're neglecting 2 live up 2 the call" of being R brother's keeper on #healthcare

99. Zooming in on Florida

100. Summaries of Citizen Reports

101. Perception -> Observations -> Sense MakingSocial Components of content dictate how we perceive and process informationTextual ContentLatent crowd characteristics from contentSpatial, Temporal parametersWhen, where the message originatedPoster demographicsAge, gender, socio-economic status..

102. Spatio Temporal and Thematic analysisWhat else happened “near” this event location?What events occurred “before” and “after” this event?Any message about “causes” for this event?

103. Spatial View….Which tweets originated from an address near 18.916517°N 72.827682°E?

104. Temporal ViewWhich tweets originated during Nov 27th 2008,from 11PM to 12 PM

105. Giving usTweets originated from an address near 18.916517°N, 72.827682°E during time interval27th Nov 2008 between 11PM to 12PM?

106. More meaningful spatio-temporal-thematic analysisPreserve social perceptions behind social dataExtracting key phrases that describe an eventSeparate user observations by time and spaceExtract summaries / key phrases / n-gramsWeight local to global, most recent to least recent

107. TWITRIS : Twitter+TetrisOur attempt to help you keep up with citizen observations on TwitterWHAT are people saying, WHEN, from WHEREPuts citizen reports in context for you by overlaying it with news, wikipedia articles!90

108. Twitris demo(Search “Twitris” on YouTube)

109. 92

110. THEME – Understanding Casual TextGathering and processing social observationsChallenges with Casual textInformal, Domain Dependent slangs, misspellings, non-grammaticalRedundancy (everyone is tweeting the same thing)Variability (everyone is saying the same thing in many ways) off-topic noise

111. Context – Importance and Challenges94 Weather conditions from tweets using key words (http://smalltalkapp.com/).Context – Importance and Challenges95 Selecting Georgia, we get the tweets used to derive weather conditions. Primarily use Keywords without background knowledge.

112. Context – Importance and Challenges96a nickname for Hip-Hop/R&B singer Chris Brown"Country Sunshine" is the name of a popular country song written by Dottie West in 1973Not snowing now!

113. Using Domain KnowledgeUsing Domain Knowledge to Overcome challenges with informal user-generated contentSupplement statistical NLP / ML algorithms and techniquesDaniel Gruhl, Meenakshi Nagarajan, Jan Pieper, Christine Robson, AmitSheth, Multimodal Social Intelligence in a Real-Time Dashboard System, to appear in a special issue of the VLDB Journal on "Data Management and Mining for Social Networks and Social Media"

114. Knowledge of domain helps in collecting and analyzing social observationsInformal text – www.twitter.com

115. Common Thematic Analysis Tasks on User-generated ContentEntity identification and disambiguationContext poor, InformalDomain models to aid disambiguation

116. Geocoder(Reverse Geo-coding)Address to location database18 Hormusji Street, ColabaVasantViharImage Metadatalatitude: 18° 54′ 59.46″ N, longitude: 72° 49′ 39.65″ EStructured Meta ExtractionNariman HouseIncome Tax OfficeIdentify and extract information from tweetsSpatio-Temporal Analysis

117. Domain Ontologies provide additional ContextInformal text, Context-poor utterances…Supplement NL features used for NER with information from Domain models

118. Common Thematic Analysis Tasks on User-generated ContentOpinion Expressions“Your new album is wicked”Shallow NL ParseLook up : UrbanDictionary (slang dictionary, glossary and orientations)Your/PRP$ new/JJ album/NN is/VBZ wicked/JJ

119. Common Thematic Analysis Tasks on User-generated ContentSpam / Off-topic EliminationSpecial type of spam: related to topic, not to application’s interestsMusic Popularity applicationsSpam: Paul McCartney’s divorce; Rihanna’s Abuse; Madge and JesusSelf-Promotions “check out my new cool sounding tracks..” Same (music) domain, similar keywords, harder to tell apartStandard Spam“Buy cheap cellphones here..”

120. Spam Elimination using previous knowledge annotation cuesAggregate functionPhrases indicative of spam (regular expressions)Rules over previous annotator results if a spam phrase, artist/track name and a positive sentiment were spotted, the comment is probably not spam!

121. Social Data in ContextPresenting social data in context is an important aspect of sense making

122. Example -- Social Media in ContextSOYLENT GREEN and the HEALTH CARE REFORMPerceptions -> Observations -> Sense making -> Perceptions

123. People-Content-Network AnalysisSentimentIntentionCulturalBehavioralInformation DiffusionInfluenceGroup formulation107

124. Merging sensor and social dataan image taken from a raw satellite feed108

125. Realistic scenarioan image taken by a camera phone with an associated label, “explosion.” 109

126. Realistic scenarioTextual messages (such as tweets) using STT analysis110

127. Realistic scenarioCorrelating to get

128. Create better views (smart mashups)

129. Continuous SemanticsIncreasingly popular social, mobile, and sensor webs exhibit these characteristics spontaneous (arising suddenly)follow a period of rapid evolution, involving real-time or near real-time data, which requires continuous searching and analysis. many distributed participants with fragmented and opinionated informationaccommodate diverse viewpoints involving topical or contentious subjects. feature context colored by local knowledge as well as perceptions based on different observations and their sociocultural analysis. 113

130. The circle of knowledge life on the Web 114Physical/Scientific principles

131. Historical Facts,

132. Content created by humans and through social processesWhere does the Domain Knowledge come from?For relatively static domains:Expert and committee based ontology creation … works in some domains (e.g., biomedicine, health care,…)Community maintained knowledge-bases, dictionaries, … (musicbrainz, IMDB, ….)For rapidly evolving domains:How to create models that are “socially scalable”?How to organically grow and maintain this model?

133. KNOWLEDGE EXTRACTION by crowd-sourcing“Human Cognition” AND Psychology AND Neuroscience?116

134. Harvesting Community Knowledge & Scientific Corpus“Human Cognition” AND Psychology AND Neuroscience117

135. Human Performance &Cognition Ontology“Human Cognition” AND Psychology AND NeuroscienceC. Thomas, P. Mehra, R. Brooks and A. Sheth. Growing Fields of Interest -Using an Expand and Reduce Strategy for Domain Model Extraction. 2008 IEEE/WIC/ACM Intl Conf on Web Intelligence and Intelligent Agent Technology, Sydney, 2008,

136. 119From: Continuous Semantics … IEEE Internet Computing, Nov-Dec, 2010

139. Significant capabilities to come:Richer forms of relationships (relationships are at the heart of semantics)

140. Richer support for events and situations

141. Significant advances in semantics and knowledge-enriched linking of cyber-physical and social-technical systems123EventWeb [Jain], RelationshipWeb [Sheth]Suppose that we create a Web in whichEach node is an event or object Each node may be connected to other nodes using Referential: similar to common links that refer to other related information. Spatial and temporal relationships.Causal: establishing causality among relationships. Relational: giving similarity or any other relationship. Semantic or Domain specific:Familial ProfessionalGenetics,…Adapted from a talk by Ramesh Jain

142. is_advised_bypublishesResearcherPh.D StudentResearch Paperpublished_inpublished_inAssistant ProfessorProfessorJournalConferencehas_locationLocationMoscone Center, SFOMay 28-29, 2008Spatio- temporalCausalImage MetadataAttended Google IOEventDomain SpecificAmitShethKarthik GomadamIs advised byRelationalDirectskno.e.sis

143. Events in Iran Election, 2009125

144. SearchIntegrationAnalysisDiscoveryQuestion AnsweringSituational AwarenessDomain ModelsPatterns / Inference / ReasoningRDBRelationship WebMeta data / Semantic AnnotationsMetadata ExtractionMultimedia Content and Web dataTextSensor DataStructured and Semi-structured data

145. SearchIntegrationDomain ModelsStructured text (biomedical literature)AnalysisDiscoveryQuestion AnsweringPatterns / Inference / ReasoningInformal Text (Social Network chatter)Relationship WebMeta data / Semantic AnnotationsMetadata ExtractionMultimedia Content and Web dataWeb Services

146. 128Online and offline worldsComputational abstractions to represent the physical world’s dynamic nature Merging online and offline activitiesConnecting the physical world naturally with the online worldWhat are natural operations on these abstractions? How do we detect these abstractions based on other abstractions and multimodal data sources?

147. 129Objects to EventsIf we move from this object mode to an event modeA single user action or request or sensory observation could act as a cue for getting all (multi-modal) information associated with an eventIf conditions change, systems could even modify their behavior to suit their changing view of the world Today text is most prevalent, with increasing but disparate (non-integrated) image and video data, but human experience is event based (at higher levels of abstractions) formed based on multi-sensory, multi-perception (at lower level of abstraction) observations

148. 130On our way…We are already seeing efforts toward this larger goalSocial connections, interests, locations, alerts, commentMobile phone to social compass: LOOPT.comImage credit - www.movilae.com

149. 131On our way…Internet of ThingsInternet of Things: “A world where inanimate objects communicate with us and one another over the network via tiny intelligent objects” - Jean Philippe Vasseur, NSSTG SystemsImage credit - www.forbes.com

150. ExperienceDirect Observation ofor Participation inEvents as a basis of knowledge

151. Entities and EventsEvent Entity NameDurationNameLocationAttributesData-streamsAttributesProcessesAdjacent StatesRelated LinksProcesses(Services)Objects and Entitiesare static.Events are dynamic.Thanks – Ramesh Jain

152. Strategic Inflection PointsEvents on Web(Experience)Documents on Web(Information)ImmersiveExperienceContextualSearchUbiquitous DevicesSemanticSearchUpdates and alertsKeyword Search1995200020052010Thanks – Ramesh Jain

153. 135Challenges – Complex EventsFormal framework to model complex situations and composite events Those consisting of interrelated events of varying spatial and temporal granularity, together with their multimodal experiencesWhat computational approaches will help to compute and reason with events and their associated experiences and objects ?

154. Example 1. Sensors observe environmental phenomena and nearby vegetation. 2. Observation analysis determines potential situation and effects. Through abductive reasoning, observation analysis perceives a possible storm as the best explanation hypothesis for observed phenomena.

155. Through predictive deductive reasoning, observation analysis determines the effect on the crops, including the potential for the poisoning of the soil from salt carried from the ocean in the wind.

156. Through query against a knowledge base of the agriculture domain, observation analysis determines that the best remedy

157. for saline soil is to “leach” the soil with excess irrigation water in order to ‘push’ the salts below the crop root zone,

158. for sodic soil is to add gypsum before leaching.Example 1. Sensors observe environmental phenomena and nearby vegetation. 2. Observation analysis determines potential situation and effects. 3. System alerts nearby farmers of situation and possible remedy.4. Farmer goes outside and looks at the sky and crops.5. Farmer perceives high-winds and dark rain clouds over the ocean view and agrees with system perception. 6. Farmer calls children and neighbors to help take the necessary precautions to save the vegetables.

159. 138From the Semantic Web CommunitySeveral key contributing research areasOperating Systems, networks, sensors, content management and processing, multimodal data integration, event modelling, high-dimensional data visualization ….Semantics and Semantic technologies can play vital role In the area of processing sensor observations, the Semantic Web is already making stridesUse of core SW capabilities: knowledge representation, use of knowledge bases (ontologies, folkonomies, taxonomy, nomenclature), semantic metadata extraction/annotation, exploiting relationships, reasoning

160. 139THERE IS MORE HAPPENING AT KNO.E.SIShttp://knoesis.orgAlso check out demos, systems athttp://knoesis.wright.edu/library/demos/

161. Kno.e.sis Members – a subset140

162. Influential WorksV Bush, As We May Think, The Atlantic, July 1945. [Memex, trail blazing]Mark Weiser, The Computer for the Twenty-First Century, Scientific American, Sept 1991, 94-10. [The original vision paper on ubicomp. Expansive vision albeit technical aspects focused on HCI with networked tabs, pads and boards.]V. Kashyap and A. Sheth, Semantics-based information brokering. Third ACM Intl Conf on Information and Knowledge Management (CIKM94), Nov 29 - Dec 02, 1994. ACM, New York, NY. [semantics based query processing (involving multiple ontologies, context, semantic proximity) across a federated information sources across the Web]Abowd, Mynatt, Rodden, The Human Experience, Pervasive computing, 2002. [explores Mark Wisner’s original ubicomp vision]Jonathan Rossiter , Humanist Computing: Modelling with Words, Concepts, and Behaviours, in Modelling with Words, Springer, 2003, pp. 124-152 [modelling with words, concepts and behaviours defines a hierarchy of methods which extends from the low level data-driven modelling with words to the high level fusion of knowledge in the context of human behaviours]Ramesh Jain, Experiential computing. Commun. ACM 46, 7, Jul. 2003, 48-55. AmitSheth, Sanjeev Thacker, and Shuchi Patel, Complex Relationship and Knowledge Discovery Support in the InfoQuilt System, VLDB Journal, 12 (1), May 2003, 2–27. [complex semantic inter-domain (multi-ontology) relationships including causal relationships to enable human-assisted knowledge discovery and hypothesis testing over Web-accessible heterogeneous data]141

163. AmbjörnNaeve: The Human Semantic Web: Shifting from Knowledge Push to Knowledge Pull. Int. J. Semantic Web Inf. Syst. 1(3): 1-30 (2005) [discusses conceptual interface providing human-understandable semantics on top of the ordinary (machine) Semantic Web]Ramesh Jain, Toward EventWeb. IEEE Distributed Systems Online 8, 9, Sep. 2007. [a web of temporally related events… informational attributes such as experiential data in the form of audio, images, and video can be associated with the events]The Internet of Things, International Telecommunication Union, Nov 2005.Other Closely Related publicationsAmitSheth and MeenaNagarajan, Semantics empowered Social Computing, IEEE Internet Computing, Jan-Feb 2009.AmitSheth, Cory Henson, and SatyaSahoo, "Semantic Sensor Web," IEEE Internet Computing, July/August 2008, p. 78-83. AmitSheth and Matthew Perry, “Traveling the Semantic Web through Space, Time and Theme,” IEEE Internet Computing, 12, (no.2), February/March 2008, pp.81-86. AmitSheth and CarticRamakrishnan, “Relationship Web: Blazing Semantic Trails between Web Resources,” IEEE Internet Computing, July–August 2007, pp. 84–88.142

164. Interested in more background?Computing for Human ExperienceContinuous Semantics to Analyze Real-Time DataSemantic Modeling for Cloud ComputingCitizen Sensing, Social Signals, and Enriching Human ExperienceSemantics-Empowered Social ComputingSemantic Sensor Web Traveling the Semantic Web through Space, Theme and Time Relationship Web: Blazing Semantic Trails between Web Resources SA-REST: Semantically Interoperable and Easier-to-Use Services and MashupsSemantically Annotating a Web ServiceContact/more details: amit @ knoesis.orgPartial Funding: NSF (Semantic Discovery: IIS: 071441, Spatio Temporal Thematic: IIS-0842129), AFRL and DAGSI (Semantic Sensor Web), Microsoft Research and IBM Research (Analysis of Social Media Content),and HP Research (Knowledge Extraction from Community-Generated Content).

Computing for Human Experience [v3, Aug-Oct 2010]

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Computing for Human Experience [v3, Aug-Oct 2010] (20)

Recently uploaded (20)

Computing for Human Experience [v3, Aug-Oct 2010]