Ontopia tutorial

Ontopia TutorialTMRA 2010-09-29Lars Marius Garshol & Geir Ove Grønmo

AgendaAbout youwho are you?About OntopiaThe productThe futureParticipating in the project

Brief history1999-2000private hobby project for Geir Ove2000-2009commercial software sold by Ontopia ASlots of international customers in diverse fields2009-open source project

The projectOpen source hosted at Google CodeContributorsLars Marius Garshol, BouvetGeir Ove Grønmo, BouvetThomas Neidhart, SpaceAppsLars Heuer, SemagiaHannes Niederhausen, TMLabStig Lau, BouvetBaard H. Rehn-Johansen, BouvetPeter-Paul Kruijssen, MorpheusQuintin Siebers, MorpheusMatthias Fischer, HTW Berlin

Recent workOntopia/Liferay integrationMatthias Fischer & LMGVarious fixes and optimizationseveryoneTropics (RESTful web service interface)SpaceAppsPorting build system to Maven2Morpheus

Architecture and modulesProduct overview

The big pictureAuto-class.A.N.otherA.N.otherOtherCMSsA.N.otherA.N.otherDB2TMPortlet supportOKPTMSyncEngineCMSintegrationData integrationEscenicTaxon.importOntopolyWebservice

The engineCore APITMAPI 2.0 supportImport/exportRDF conversionTMSyncFulltext searchEvent APItolog query languagetolog update languageEngine

The backendsIn-memoryno persistent storagethread-safeno setupRDBMStransactionspersistentthread-safeuses cachingclusteringRemoteuses web serviceread-onlyunofficialEngineMemoryRDBMSRemote

DB2TMUpconversion to TMsfrom RDBMS via JDBCor from CSVUses XML mappingcan call out to JavaSupports synceither full rescanor change tableTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote

TMRAPWeb service interfacevia SOAPvia plain HTTPRequestsget-topicget-topic-pageget-tologdelete-topic...TMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote

Navigator frameworkServlet-based APImanage topic mapsload/scan/delete/createJSP tag libraryXSLT-likebased on tologJSTL integrationTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote

Automated classificationUndocumentedexperimentalExtracts textautodetects formatWord, PDF, XML, HTMLProcesses textdetects languagestemming, stop-wordsExtracts keywordsranked by importanceuses existing topicssupports compound termsTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote

VizigatorVizOntopolyGraphical visualizationVizDesktopSwing app to configurefilter/style/...VizletJava applet for webuses configurationloads via TMRAPuses “Remote” backendTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote

OntopolyVizOntopolyGeneric editorweb-based, AJAXmeta-ontology in TMOntology designercreate types and fieldscontrol user interfacebuild viewsincremental devInstance editorguided by ontologyTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote

Typical deploymentViewingapplicationEngineUsersDBBackendOntopolyFrameworksEditorsDBTMRAPDB2TMHTTPDBExternal applicationApplication server

Core APIsnet.ontopia.topicmaps.core.*Fairly direct mapping from TMDMTopicIFAssociationIFTopicMapIF...Set/get methods reflect TMDM properties

TopicIFInterface, not a classgetTopicNames()addTopicName(TopicNameIF)removeTopicName(TopicNameIF)getOccurrences() + add + removegetSubjectIdentifiers() + add + removegetItemIdentifiers() + add + removegetSubjectLocators() + add + removegetRoles()getRolesByType(TopicIF)

Core interfacesTopicMapStoreIFTopicMapIFTopicIFAssociationIFTopicNameIFOccurrenceIFAssociationRoleIFVariantNameIF

How to get a TopicMapIFCreate one directlynew net...impl.basic.InMemoryTopicMapStore()Load one from fileusing an importer (next slide)Connect to an RDBMScovered laterUse a topic map repositorycovered later

TopicMapReaderIFimport net.ontopia.topicmaps.core.TopicMapIF;import net.ontopia.topicmaps.core.TopicMapReaderIF;import net.ontopia.topicmaps.utils.ImportExportUtils;public class TopicCounter { public static void main(String[] argv) throws Exception { TopicMapReaderIF reader = ImportExportUtils.getReader(argv[0]); TopicMapIF tm = reader.read(); System.out.println("TM contains " + tm.getTopics().size() + " topics"); }}[larsga@c716c5ac1 tmp]$ java TopicCounter ~/data/bilder/privat/metadata.xtmTM contains 17035 topics[larsga@c716c5ac1 tmp]$

The utility classesA set of classes outside the core interfaces that perform common tasksa number of these utilities are obsolete now that tolog is hereThey are all built on top of the core interfacesSome important utilitiesImportExportUtils creates readers and writersMergeUtils merges topics and topic mapsPSI contains important PSIsDeletionUtils cascading delete of topicsDuplicateSuppressionUtils removes duplicatesTopicStringifiers find names for topics

Topic Maps repositoryUses a set of topic maps sources to build a set of topics mapstopic maps can be looked up by IDMany kinds of sourcesscan directory for files matching patterndownload from URLconnect to RDBMS...Configurable using an XML filetm-sources.xmlUsed by Navigator Framework

Event APIAllows clients to receive notification of changesMust implement TopicMapListenerIF static class TestListener extends AbstractTopicMapListener { public void objectAdded(TMObjectIF snapshot) { System.out.println("Topic added: " + snapshot.getObjectId()); } public void objectModified(TMObjectIF snapshot) { System.out.println("Topic modified: " + snapshot.getObjectId()); } public void objectRemoved(TMObjectIF snapshot) { System.out.println("Topic removed: " + snapshot.getObjectId()); } }

Using the API // register to listen for events TestListener listener = new TestListener(); TopicMapEvents.addTopicListener(ref, listener); // get the store through the reference so the listener is registered ref.createStore(false); // let's add a topic System.out.println("Off we go"); TopicMapBuilderIF builder = tm.getBuilder(); TopicIF newbie = builder.makeTopic(tm); System.out.println("Let's name this topic"); builder.makeTopicName(newbie, "Newbie topic"); // then let's remove it System.out.println("And now, the exit"); DeletionUtils.remove(newbie); System.out.println("Goodbye, short-lived topic");[larsga@dhcp-98 tmp]$ java EventTest bkclean.xtm Off we goTopic added: 3409Let's name this topicTopic modified: 3409Topic modified: 3409And now, the exitTopic removed: 3409Goodbye, short-lived topic

For more informationSee the Engine Developer's Guidehttp://www.ontopia.net/doc/current/doc/engine/devguide.html

Persistence & transactionsRDBMS backend

RDBMS backendStores Topic Maps in an RDBMSgeneric schemaaccess via JDBCProvides full ACIDtransactionsconcurrency...Supports several databasesOracle, MySQL, PostgreSQL, MS SQL Server, hsqlClustering support

Core API implementationImplements same API as in-memory impltheoretically, a switch requires only config changeLazy loading of objectsobjects loaded from DB as neededConsiderable internal cachingfor performance reasonsSeparate objects for separate transactionsin order to provide isolationShared cache between transactions

ConfigurationA Java property fileSpecifiesdatabase typeJDBC URLusername + passwordcache settingsclustering settings...

jdbcspyA built-in SQL profilerUseful for identifying cause of performance issues

tologA logic-based query languagea mix of Prolog and SQLeffectively equivalent to DatalogTwo partsqueries (data retrieval)updates (data modification)Developed by Ontopianot an ISO standardeventually to be replaced by TMQL

tologThe recommended way to interact with the dataAPI programming is slow and cumbersometolog queries perform betterAvailable viaJava APIWeb service APIForms interface in Omnigatortolog queries return API objects

Finding all operas by a composer Collection operas = new ArrayList(); TopicIF composer = getTopicById("puccini"); TopicIF composed_by = getTopicById("composed-by"); TopicIF work = getTopicById("work"); TopicIF creator = getTopicById("composer"); for (AssociationRoleIF role1 : composer.getRolesByType(creator)) { AssociationIF assoc = role1.getAssociation(); if (assoc.getType() != composed_by) continue; for (AssociationRoleIF role2 : assoc.getRoles()) { if (role2.getType() != work) continue; operas.add(role2.getPlayer()); } }

Finding all operas by a composercomposed-by(puccini : composer, $O : work)?composed-by($C : composer, tosca : work)?composed-by($C : composer, $O : work)?composed-by(puccini : composer, tosca : work)?

FeaturesAccess all aspects of a topic mapGeneric queries independent of ontologyAND, OR, NOT, OPTIONALCountSortLIMIT/OFFSETReusable inference rules

Chaining predicates (AND)Predicates can be chainedborn-in($PERSON : person, $PLACE : place),located-in($PLACE : containee, italy : container)?The comma between the predicates means ANDThis query finds all the people born in ItalyIt first builds a two-column table of all born-in associationsThen, those rows where the place is not located-in Italy are removed(Note that when the PLACE variable is reused above that means that the birthplace and the location must be the same topic in each match)Any number of predicates can be chainedTheir order is insignificantActually, the optimizer reorders the predicatesIt will start with located-in because it has a topic constant

Thinking in predicatesMost of you are probably used to functions, which work like this:

function(arg1, arg2, arg3) -> result

Predicates, however, are in a sense bidirectional, because of the way the pattern matching works

predicate(topic : role1, $VAR : role2)

predicate($VAR : role1, topic : role2)

The order of the roles are, on the other hand, insignificant

predicate($VAR : role2, topic : role1)ProjectionSometimes queries make use of temporary variables that we are not really interested inThe way to get rid of unwanted variables is projectionSyntax: select $variable1, $variable2, ... from <query>?The query is first run, then projected down to the requested variables

The instance-of predicateinstance-of has the following form:

NOTE: the order of the arguments is significant

Like players, instance and class may be specified in two ways:

instance-of makes use of the superclass-subclass associations in the topic map

this means that composers will be considered musicians, and musicians will be considered personsCities with the most premieresusing o for i"http://psi.ontopedia.net/"select $CITY, count($OPERA) from instance-of($CITY, o:City), { o:premiere($OPERA : o:Work, $CITY : o:Place) | o:premiere($OPERA : o:Work, $THEATRE : o:Place), o:located_in($THEATRE : o:Containee, $CITY : o:Container) } order by $OPERA desc?

All non-hidden photosselect $PHOTO from instance-of($PHOTO, op:Photo) not(ph:hide($PHOTO : ph:hidden)), not(ph:taken-at($PHOTO : op:Image, $PLACE : op:Place), ph:hide($PLACE : ph:hidden)), not(ph:taken-during($PHOTO : op:Image, $EVENT : op:Event), ph:hide($EVENT : ph:hidden)), not(ph:depicted-in($PHOTO : ph:depiction, $PERSON : ph:depicted), ph:hide($PERSON : ph:hidden))?

DemoShow running queries in OmnigatorAlso show query tracingShakespeare/* #OPTION: optimizer.reorder = false */

tologspytolog query profilershares code with jdbcspy

Using the query engine APIThe query engine API is really simple to useget a QueryProcessorIF objectrun a query in the QueryProcessorIF and get a QueryResultIFloop over the results and use themclose the result objectgo back to step 2, or do something elseThere are two different QueryProcessorIF implementations

the API lets you write code without worrying about that, however

the two implementations behave identicallyRunning a query with the APITopicMapIF tm = ...;QueryProcessorIF processor = QueryUtils.getQueryProcessor(tm);QueryResultIF result = processor.execute(“instance-of($P, person)?”);try { while (result.next()) { TopicIF person = (TopicIF) result.getValue(0); // do something useful with 'person' } } finally { result.close();}

Advanced optionsIt is possible to parse a query once, and then run it many times

the processor returns a ParsedQueryIF object, which can be executed

parameters can be passed to the query on each execution

It is possible to make declarations and use them across executionsUsing a parsed queryParsedQueryIF parsedQuery = processor.parse(“instance-of($P, %TYPE%)?”);Map params = Collections.singletonMap(“TYPE”, person);QueryResultIF result = parsedQuery.execute(params);try { while (result.next()) { // ... } } finally { result.close();}

QueryWrapperDesigned to make all of this easierQueryWrapper qw = new QueryWrapper(tm);TopicIF topic = qw.queryForTopic(...);List topics = qw.queryForList(...);List<Person> people = qw.queryForList(..., mapper);

tolog updatesGreatly simplifies TM modificationAlso means you can do modification without API programminguseful with RDBMS topic mapsuseful with TMs in running web serversBy performing a sequence of updates, just about any change can be madePotentially allows much more powerful architecture

DELETEStatic formdelete lmgDynamic formdelete $person from instance-of($person, person)Delete a valuedelete subject-identifier(topic, “http://ex.org/tst”)

MERGEStatic formMERGE topic1, topic2Dynamic formMERGE $p1, $p2 FROM instance-of($p1, person), instance-of($p2, person), email($p1, $email), email($p2, $email)

INSERTStatic formINSERT lmg isa person; - “Lars Marius Garshol” .Dynamic formINSERT tmcl:belongs-to-schema(tmcl:container : theschema, tmcl:containee: $c) FROM instance-of($c, tmcl:constraint)

INSERT againINSERT ?y $psi . event-in-year(event: $e, year: ?y)FROM start-date($e, $date), str:substring($y, $date, 4), str:concat($psi, "http://psi.semagia.com/iso8601", $y)

UPDATEStatic formUPDATE value(@3421, “New name”)Dynamic formUPDATE value($TN, “Ontopia”) FROM topic-name(oks, $TN)

More informationLook at sample queries in Omnigatortolog tutorialhttp://www.ontopia.net/doc/current/doc/query/tutorial.htmltolog built-in predicate referencehttp://www.ontopia.net/doc/current/doc/query/predicate-reference.html

Conversion from RDBMS dataDB2TM

DB2TMUpconversion of relational dataeither from CSV files orover JDBCBased on an XML file describing the mappingvery highly configurableSupport forall of Topic Maps (except variants)value transformationssynchronization

Standard use casePull in data from external sourceturn it into Topic Maps following some ontologyEnrich itusually manually, but not necessarilyResync from source at intervals

DB2TM exampleOntopia+=United NationsBouvet<relation name="organizations.csv" columns="id name url"> <topic type="ex:organization">
<item-identifier>#org${id}</item-identifier>
<topic-name>${name}</topic-name>
<occurrence type="ex:homepage">${url}</occurrence>
</topic></relation>

Creating associations <relation name="people.csv" columns="id given family employer phone"> <topic id="employer"> <item-identifier>#org${employer}</item-identifier> </topic> <topic type="ex:person"> <item-identifier>#person${id}</item-identifier> <topic-name>${given} ${family}</topic-name> <occurrence type="ex:phone">${phone}</occurrence> <player atype="ex:employed-by" rtype="ex:employee"> <other rtype="ex:employer" player="#employer"/> </player> </topic> </relation>

Value transformations <relation name="SCHEMATA" columns="SCHEMA_NAME"> <function-column name='SCHEMA_ID' method='net.ontopia.topicmaps.db2tm.Functions.makePSI'> <param>${SCHEMA_NAME}</param> </function-column> <topic type="mysql:schema"> <item-identifier>#${SCHEMA_ID}</item-identifier> <topic-name>${SCHEMA_NAME}</topic-name> </topic> </relation>

Running DB2TMjava net.ontopia.topicmaps.db2tm.Executecommand-line toolalso works with RDBMS topic mapsnet.ontopia.topicmaps.db2tm.DB2TMAPI class to run transformationsmethods "add" and "sync"

More informationDB2TM User's Guidehttp://www.ontopia.net/doc/current/doc/db2tm/user-guide.html

Synchronizing with other sourcesTMSync

TMSyncConfigurable module for synchronizing one TM against anotherdefine subset of source TM to sync (using tolog)define subset of target TM to sync (using tolog)the module handles the restCan also be used with non-TM sourcescreate a non-updating conversion from the source to some TM formatthen use TMSync to sync against the converted TM instead of directly against the source

How TMSync worksDefine which part of the target topic map you want,Define which part of the source topic map it is the master for, andThe algorithm does the rest

If the source is not a topic mapTMSyncconvert.xsltSimply do a normal one-time conversionlet TMSync do the update for youIn other words, TMSync reduces the update problem to a conversion problemsource.xml

The City of Bergen usecaseNorge.noServiceUnitPersonLOSCity of BergenLOS

TMRAP basicsAbstract interfacethat is, independent of any particular technologycoarse-grained operations, to reduce network trafficProtocol bindings existplain HTTP bindingSOAP bindingSupports many syntaxesXTM 1.0LTMTM/XMLcustom tolog result-set syntax

get-topicRetrieves a single topic from the remote servertopic map may optionally be specifiedsyntax likewiseMain useto build client-side fragments into a bigger topic mapto present information about a topic on a different server

get-topicParametersidentifier: a set of URIs (subject identifiers of wanted topic)subject: a set of URIs (subject locators of wanted topic)item: a set of URIs (item identifiers of wanted topic)topicmap: identifier for topic map being queriedsyntax: string identifying desired Topic Maps syntax in responseview: string identifying TM-Views view used to define fragmentResponsetopic map fragment representing topic in requested syntaxdefault is XTM fragment with all URI identifiers, names, occurrences, and associationsin default view types and scopes on these constructs are only identified by one <*Ref xlink:href=“...”/> XTM elementthe same goes for associated topics

get-topic-pageReturns link information about a topicthat is, where does the server present this topicmainly useful for realizing the portal integration scenarioresult information contains metadata about server setup

get-topic-pageParametersidentifier: a set of URIs (subject identifiers of wanted topic)subject: a set of URIs (subject locators of wanted topic)item: a set of URIs (item identifiers of wanted topic)topicmap: identifier for topic map being queriedsyntax: string identifying desired Topic Maps syntax in responseResponse is a topic map fragment[oks : tmrap:server = "OKS Samplers local installation"][opera : tmrap:topicmap = "The Italian Opera Topic Map"] {opera, tmrap:handle, [[opera.xtm]]}tmrap:contained-in(oks : tmrap:container, opera : tmrap:containee)tmrap:contained-in(opera : tmrap:container, view : tmrap:containee)tmrap:contained-in(opera : tmrap:container, edit : tmrap:containee)[view : tmrap:view-page %"http://localhost:8080/omnigator/models/..."][edit : tmrap:edit-page %"http://localhost:8080/ontopoly/enter.ted?..."][russia = "Russia” @"http://www.topicmaps.org/xtm/1.0/country.xtm#RU"]

get-tologReturns query resultsmain use is to extract larger chunks of the topic map to the client for presentationmore flexible than get-topiccan achieve more with less network traffic

get-tologParameterstolog: tolog querytopicmap: identifier for topic map being queriedsyntax: string identifying desired syntax of responseview: string identifying TM-Views view used to define fragmentResponseif syntax is“tolog”an XML representation of the query resultuseful if order of results matterotherwise, a topic map fragment containing multiple topics is returnedas for get-topic

add-fragmentAdds information to topic map on the serverdoes this by merging in a fragmentParametersfragment: topic map fragmenttopicmap: identifier for topic map being added tosyntax: string identifying syntax of request fragmentResultfragment imported into named topic map

update-topicCan be used to update a topicadd-fragment only adds informationupdate sets the topic to exactly the uploaded informationParameterstopicmap: the topic map to updatefragment: fragment containing the new topicsyntax: syntax of the uploaded fragmentidentifier: a set of URIs (subject identifiers of wanted topic)subject: a set of URIs (subject locators of wanted topic)item: a set of URIs (item identifiers of wanted topic)Update happens using TMSync

delete-topicRemoves a topic from the serverParametersidentifier: a set of URIs (subject identifiers of wanted topic)subject: a set of URIs (subject locators of wanted topic)item: a set of URIs (item identifiers of wanted topic)topicmap: identifier for topic map being queriedResultdeletes the identified topicincludes all names, occurrences, and associations

tolog-updateRuns a tolog update statementParameterstopicmap: topic map to updatestatement: tolog statement to runRuns the statement & commits the change

HTTP binding basicsThe mapping requires a base URLe.g http://localhost:8080/tmrap/This is used to send requestshttp://localhost:8080/tmrap/method?param1=value1&...GET is used for requests that do not cause state changesPOST for requests that doResponses returned in response body

Exercise #1: Retrieve a topicUse the get-topic request to retrieve a topic from the serverbase URL is http://localhost:8080/tmrap/find the identifying URI in Omnigatorjust print the retrieved fragment to get a look at itNote: you must escape the “#” character in URIsotherwise it is interpreted as the anchor and not transmitted at allescape sequence: %23Note: you must specify the topic map IDotherwise results will only be returned from loaded topic mapsin other words: if the topic map isn’t loaded, you get no results

Solution #1 (in Python)import urllibBASE = "http://localhost:8080/tmrap/tmrap/"psi = "http://www.topicmaps.org/xtm/1.0/country.xtm%23RU"inf = urllib.urlopen(BASE + "get-topic?identifier=" + psi)print inf.read()inf.close()

Solution #1 (response)
<topicMap xmlns="http://www.topicmaps.org/xtm/1.0/"
xmlns:xlink="http://www.w3.org/1999/xlink">
<topic id="id458">
<instanceOf>
<subjectIndicatorRef xlink:href="http://psi.ontopia.net/geography/#country"/>
</instanceOf>
<subjectIdentity>
<subjectIndicatorRef xlink:href="http://www.topicmaps.org/xtm/1.0/country.xtm#RU"/>
<topicRef xlink:href="file:/.../WEB-INF/topicmaps/geography.xtmm#russia"/>
</subjectIdentity>
<baseName>
<baseNameString>Russia</baseNameString>
</baseName>
</topic>

Processing XTM with XSLTThis is possible, but unpleasantthe main problem is that the XML is phrased in terms of Topic Maps, not in domain termsthis means that all the XPath will talk about “topic”, “association”, ... and not “person”, “works-for” etcThe structure is also complicatedthis makes queries complicatedfor example, the XPath to traverse an association looks like this://xtm:association [xtm:member[xtm:roleSpec / xtm:topicRef / @xlink:href = '#employer'] [xtm:topicRef / @xlink:href = concat('#', $company)]] [xtm:instanceOf / xtm:topicRef / @xlink:href = '#employed-by']

TM/XMLNon-standard XML syntax for Topic Mapsdefined by Ontopia (presented at TMRA’05)implemented in the OKSXSLT-friendlymuch easier to process with XSLT than XTMcan be understood by developers who do not understand Topic Mapsdynamic domain-specific syntaxes instead of generic syntaxpredictable (can generate XML Schema from TM ontology)

TM/XML example<topicmap ... reifier="tmtopic"> <topicmap id="tmtopic"> <iso:topic-name><tm:value>TM/XML example</tm:value> </iso:topic-name> <dc:description>An example of the use of TM/XML.</dc:description> </topicmap> <person id="lmg"> <iso:topic-name><tm:value>Lars Marius Garshol</tm:value> <tm:variant scope="core:sort">garshol, lars marius</tm:variant> </iso:topic-name> <homepage datatype="http://www.w3.org/2001/XMLSchema#anyURI" >http://www.garshol.priv.no</homepage> <created-by role="creator" topicref="tmtopic" otherrole="work"/> <presentation role="presenter"> <presented topicref="tmxml"/> <event topicref="tmra05"/> </presentation> </person></topicmap>

tmphotoCategoryPersonPhotoEventLocationhttp://www.garshol.priv.no/tmphoto/A topic map to organize my personal photoscontains ~15,000 photosA web gallery runs on Ontopiaon www.garshol.priv.no

tmtoolshttp://www.garshol.priv.no/tmtools/OrganizationAn index of Topic Maps toolsorganized as shown on the rightAgain, web application for browsingscreenshots belowPersonSoftwareproductPlatformCategoryTechnology

The person pageBoring! No content.

get-illustrationA web service in tmphotoreceives the PSI of a personthen automatically picks a suitable photo of that personBased onvote score for photos,categories (portrait),other people in photo...The service returnsa topic map fragment with links to the person page and a few different sizes of the selected photohttp://www.garshol.priv.no/blog/183.html

get-illustrationHmmm. Scores, categories, people in photo, ...Do you have a photo ofhttp://psi.ontopedia.net/Benjamin_Bock ?http://www.garshol.priv.no/tmphoto/get-illustration?identifier=http://psi.on....tmphototmtoolsTopic mapfragment

Points to noteNo hard-wiring of linksjust add identifiers when creating people topicsphotos appear automaticallyif a better photo is added later, it’s replaced automaticallyNo copying of datano duplication, no extra maintenanceVery loose bindingnothing application-specificHighly extensibleonce the identifiers are in place we can easily pull in more content from other sources

My blogHas more content aboutpeople (tmphoto & tmtools),events (tmphoto),tools (tmtools),technologies (tmtools)Should be available in those applications

SolutionMy blog posts are taggedbut the tags are topics, which can have PSIsthese PSIs are used in tmphoto and tmtools, tooThe get-topic-page request lets tmphoto & tmtools ask the blog for links to relevant postsgiven identifiers for a topic, returns links to pages about that topichttp://www.garshol.priv.no/blog/145.html

get-topic-pageDo you have pages abouthttp://psi.ontopedia.net/TMRA_2008 ?http://www.garshol.priv.no/blog/get-topic-page?identifier=http://psi.on....BlogtmphotoTopic mapfragmentTopics linking toindividual blog posts

Making web applicationsNavigator Framework

Ontopia Navigator FrameworkJava API for interacting with TM repositoryJSP tag librarybased on tologkind of like XSLT in JSP with tolog instead of XPathhas JSTL integrationUndocumented partsweb presentation componentssome wrapped as JSP tagswant to build proper portlets from them

How it worksWeb server with JSP containere.g. Apache TomcatJSP pageBrowserTopic MapEngineTagLibrariesJSP pageBrowserJSP pageBrowserJSP pageBrowserTopic Map

makes up nearly the entire framework

used to extract information from topic maps

lets you execute tolog queries to extract information from the topic map

looping and control flow structures

separates layout and structure from content

optional, but recommendedHow the tag libraries workThe topic map engine holds a registry of topic maps

collected from the tm-sources.xml configuration file

each topic map has its own id (usually the file name)

Each page also holds a set of variable bindings

each variable holds a collection of objects

objects can be topics, base names, locators, strings, ...

some tags set the values of variables, while others use themBuilding a JSP pageThe <%@ taglib ... %> tags declare your tag libraries

Tells the page which tag library to include and binds it to a prefix

Prefixes are used to qualify the tags (and avoid name collisions)

Use the <tolog:context> tag around the entire page

The "topicmap" attribute specifies the ID of the current topic map

The first time you access the page in your browser the page gets compiled

If you modify the page then it will be recompiled the next time it is accessedhttp://www.ontopia.net/operamap

Navigator tag library example <%-- assume variable 'composer' is already set --%>Operas: <tolog:foreach query=”composed-by(%composer% : composer, $OPERA : opera), { premiere-date($OPERA, $DATE) }?”> <li> <a href="opera.jsp?id=<tolog:id var="OPERA"/>” ><tolog:out var="OPERA"/></a> <tolog:if var="DATE"> <tolog:out var="DATE"/> </tolog:if> </li></tolog:foreach>

Possible configurationApplication directorieswebappsmyApp/*.jspomnigator/WEB-INF/config/*.xmli18n/topicmaps/*.xtm, *.ltmweb.xml

The navigator configuration filesweb.xml

where to find the other files, plus plug-ins

tells the navigator where to find topic maps

configuration of the log4j logging

More details in the "Configuration Guide" documentMore informationNavigator Framework Configuration Guidehttp://www.ontopia.net/doc/current/doc/navigator/config.htmlNavigator Framework Developer's Guidehttp://www.ontopia.net/doc/current/doc/navigator/navguide.htmlNavigator Framework Tag Library Referencehttp://www.ontopia.net/doc/current/doc/navigator/tolog-taglib.html

What is automated classification?Create parts of a topic map automaticallyusing the text in existing content as the sourcenot necessarily 100% automatic; user may help outA hard tasknatural language processing is very complexresult is never perfectHowever, it’s possible to achieve some results

Why automate classification?Creating a topic map requires intellectual effortthat is, it requires work by humansHuman effort = costadded value must be sufficient to justify the costin some cases eitherthe cost is too high, orthe value added is too limitedThe purpose of automation is to lower the costthis increases the number of cases where the use of Topic Maps is justified

Automatable tasksProjectPersonDepartmentWorked onWorked onJane Doeworked onemployed inXYZ ProjectIT groupOntologyharddepends on requirementsone time onlyInstance datahardusually exists in other sourcesDocument keywordseasierfrequent operationusually no other sources

Two kinds of categorizationBroad:Environment, Crisis managementNarrow:Water, Norway, drought, Drought Act, Cloud seeding, Morecambe BayBroad categorizationcategories are broadly definedinclude many different subjectsNarrow categorizationuses very specific keywordseach keyword is a single subject

What it doesExtract keywords from contentgoal is to use these for classificationNot entity recognitionwe only care about identifying what the content is aboutUses statistical approachno attempt at full formal parsing of the text

Steps of operationIdentify formatthen, extract the textIdentify languagethen, remove stop wordsstem remaining wordsClassifycan use terms from preexisting Topic Mapsexploits knowledge of the languageReturn proposed keywords

Example of keyword extractiontopic maps 1.0metadata 0.57subject-based class. 0.42Core metadata 0.42faceted classification 0.34taxonomy 0.22monolingual thesauri 0.19controlled vocabulary 0.19Dublin Core 0.16thesauri 0.16Dublin 0.15keywords 0.15

Example #2Automated classification 1.0 5Topic Maps 0.51 14XSLT 0.38 11compound keywords 0.29 2keywords 0.26 20Lars 0.23 1Marius 0.23 1Garshol 0.22 1...

So how could this be used?To help users classify new documents in a CMS interfacesuggest appropriate keywords, screened by user before approvalAutomate classification of incoming documentsthis means lower quality, but also lower costGet an overview of interesting terms in a document corpusclassify all documents, extract the most interesting termsthis can be used as the starting point for building an ontology(keyword extraction only)

Example user interfaceThe user creates an articlethis screen then used to add keywordsuser adjusts the proposals from the classifier

Interfacesjava net.ontopia.topicmaps.classify.Chew<topicmapuri><inputfile>produces textual output onlynet.ontopia.topicmaps.classify.SimpleClassifierclassify(uri, topicmap) -> TermDatabaseclassify(uri) -> TermDatabase

Supported formats and languagesXML (any schema)HTML (non-XML)PDFWord (.doc, .docx)PowerPoint (.ppt, .pptx)Plain textEnglishNorwegian

Visualization of Topic MapsVizigator

The VizigatorGraphical visualization of Topic MapsTwo partsVizDesktop: Swing desktop app for configurationVizlet: Java applet for web deploymentConfiguration stored in XTM file

The uses of visualizationNot really suitable for navigationdoesn't work for all kinds of dataGreat for seeing the big picture

The VizigatorThe Vizigator uses TMRAPthe Vizlet runs in the browser (on the client)a fragment of the topic map is downloaded from the serverthe fragment is grown as neededServerTMRAP

Embedding the VizletSet up TMRAP serviceAdd ontopia-vizlet.jarAdd necessary HTML <applet code="net.ontopia.topicmaps.viz.Vizlet.class" archive="ontopia-vizlet.jar"> <param name="tmrap" value="/omnigator/plugins/viz/"> <param name="config" value="/omnigator/plugins/viz/config.jsp?tm=<%= tmid %>"> <param name="tmid" value="<%= tmid %>"> <param name="idtype" value="<%= idtype %>"> <param name="idvalue" value="<%= idvalue %>"> <param name="propTarget" value="VizletProp"> <param name="controlsVisible" value="true"> <param name="locality" value="1"> <param name="max-locality" value="5"></applet>

OmnigatorGeneric Topic Maps browservery useful for seeing what's in a topic mapthe second-oldest part of OntopiaContains other features beyond simple browsingstatisticsmanagement consolemergingtolog querying/updatesexport

Ontology designer and editorOntopoly

OntopolyA generic Topic Maps editor, in two partsontology editor: used to create the ontology and schemainstance editor: used to enter instances based on ontologyFeaturesworks with both XTM files and topic maps stored in RDBMS backendsupports access control to administrative functions, ontology, and instance editorsexisting topic maps can be importedparts of the ontology can be marked as read-only, or hidden

Ontology designerCreate ontology based ontopic, association, name, occurrence, and role typesSupports iterative ontology developmentmodify and prototype the ontology until it's rightSupports ontology annotationadd fields to topic types, for exampleSupports viewsdefine restricted views of certain topic types

Instance editorConfigured by the ontology editorshows topics as defined by the ontologyHas several ways to pick associationsdrop-down listby searchfrom hierarchyAvoids conflictspages viewed by one user are locked to others

Ontopoly is embeddableThe Ontopoly instance editor can be embeddedbasically, the main panel can be inserted into another web applicationuses an iframeRequires only ID of topic being editedcan also be restricted to a specific viewMakes it possible to build easier-to-use editorsso users don't have to learn all of Ontopoly

Adding content featuresCMS integrations

CMS integrationThe best way to add content functionality to Ontopiathe world doesn’t need another CMSbetter to reuse those which already existSo far two integrations existEscenicOfficeNet Knowledge Portalmore are being worked on

ImplementationA CMS event listenerthe listener creates topics for new CMS articles, folders, etcthe mapping is basically the design of the ontology used by this listenerPresentation integrationit must be possible to list all topics attached to an articleconversely, it must be possible to list all articles attached to a topichow close the integration needs to be here will vary, as will the difficulty of the integrationUser interface integrationit needs to be possible to attach topics to an article from within the normal CMS user interfacethis can be quite trickySearch integrationthe Topic Maps search needs to also search content in the CMScan be achieved by writing a tolog plug-in

Articles as topicsis aboutElectionsNew city council appointedGoal: associate articles with topicsmainly to say what they are abouttypically also want to include other metadataNeed to create topics for the articles to do thisin fact, a general CMS-to-TM mapping is neededmust decide what metadata and structures to include

Mapping issuesArticle topicswhat topic type to use?title becomes name? (do you know the title?)include author? include last modified? include workflow state?should all articles be mapped?Folders/directories/sections/...should these be mapped, too?one topic type for all folders/.../.../...?if so, use associations to connect articles to foldersuse associations to reproduce hierarchical folder structureMultimedia objectsshould these be included?what topic type? what name? ...

Two styles of mappingsArticles as articlesTopic represents only the articleTopic type is some subclass of “article”“Is about” association connects article into topic mapFields are presentationaltitle, abstract, bodyArticles as conceptsTopic represents some real-world subject (like a person)article is just the default content about that subjectType is the type of the subject (person)Semantic associations to the rest of the topic mapworks in department, has competence, ...Fields can be semanticname, phone no, email, ...

Article as articleArticle about building of a new schoolIs about association to “Primary schools”Topic type is “article”

Article as conceptArticle about a sports hallArticle really represents the hallTopic type is “Location”Associations tocity borough

Ontopia/LiferayAn integration with the Liferay CMS and portal is in progresspresented Friday 1130-1150 in Schiller 2

The projectA new citizen’s portal for the city administrationstrategic decision to make portal main interface for interaction with citizensas many services as possible are to be moved onlineBig projectstarted in late 2004, to continue at least into 2008~5 million Euro spent by launch date1.7 million Euro budgeted for 2007Topic Maps development is a fraction of this (less than 25%)Many companies involvedBouvet/OntopiaAvenirKPMGKarabinEscenic

Simplified original ontologyService catalogEscenic (CMS)LOSFormArticlenearlyeverythingCategoryServiceSubjectDepartmentBoroughExternalresourceEmployeePayroll++

Ontopia tutorial

More Related Content

What's hot (20)

Similar to Ontopia tutorial (20)

More from Lars Marius Garshol (20)

Recently uploaded (20)

Ontopia tutorial