Ontopia Code Camp
- 1. Ontopia Code CampTMRA 2009-11-11Lars Marius Garshol & Geir Ove Grønmo
- 2. AgendaAbout youwho are you?what do you want from the code camp?About OntopiaThe productThe futureParticipating in the projectWriting some code!
- 4. Brief history1999-2000private hobby project for Geir Ove2000-2009commercial software sold by Ontopia ASlots of international customers in diverse fields2009-open source project
- 5. The projectOpen source hosted at Google CodeContributorsLars Marius Garshol, BouvetGeir Ove Grønmo, BouvetThomas Neidhart, SpaceAppsLars Heuer, SemagiaHannes Niederhausen, TMLabStig Lau, BouvetBaard H. Rehn-Johansen, BouvetPeter-Paul Kruijssen, MorpheusQuintin Siebers, Morpheus
- 6. Current activity (toward 5.1)tolog updatesadded by LMGVarious fixes and optimizationsby everyoneToma implementation (in sandbox)by ThomasTMQL implementation (in sandbox)?by Sven Krosse
- 7. Architecture and modulesThe product
- 8. The big pictureAuto-class.A.N.otherA.N.otherOtherCMSsA.N.otherA.N.otherDB2TMPortlet supportOKPXML2TMEngineCMSintegrationData integrationEscenicTaxon.importOntopolyWebservice
- 9. The engineCore APITMAPI 2.0 supportImport/exportRDF conversionTMSyncFulltext searchEvent APItolog query languagetolog update languageEngine
- 10. Query EngineImplementation of Ontopia’s tolog language (based on Prolog and SQL)Allows powerful queries on the topic map data structureSimplifies application development and improves performanceExample:select $B, count($A) from instance-of($B, city),{ premiere($A : opera, $B : place) | premiere($A : opera, $C : place), located-in($C : containee, $B : container) } order by $A desc?returns all B's and the corresponding number of A's whereB is a city ANDEITHER B is the place where A was premieredOR the place where A was premiered is located in B in decreasing order of ATMSyncConfigurable module for synchronizing one TM against anotherdefine subset of source TM to sync (using tolog)define subset of target TM to sync (using tolog)the module handles the restCan also be used with non-TM sourcescreate a non-updating conversion from the source to some TM formatthen use TMSync to sync against the converted TM instead of directly against the source
- 11. How TMSync worksDefine which part of the target topic map you want,Define which part of the source topic map it is the master for, andThe algorithm does the rest
- 12. If the source is not a topic mapTMSyncconvert.xsltSimply do a normal one-time conversionlet TMSync do the update for youIn other words, TMSync reduces the update problem to a conversion problemsource.xml
- 13. The City of Bergen usecaseNorge.noServiceUnitPersonLOSCity of BergenLOS
- 14. The backendsIn-memoryno persistent storagethread-safeno setupRDBMStransactionspersistentthread-safeuses cachingclusteringRemoteuses web serviceread-onlyunofficialEngineMemoryRDBMSRemote
- 15. RDBMS BackendAllows the Engine to use topic maps stored in a relational databaseBased on a generic topic map schemaNecessary when working with very large topic mapsTransparent to applicationsFeaturesAutomatically loads data when neededCaches frequently used dataFull support for RDBMS transactionsSupports tolog-to-SQL compilationStatistical reports for performance tuningPlatform supportOracle, MySQL, PostgreSQL, MS SQL ServerTest suite available for verifying compatibility with other JDBC-enabled RDBMSes
- 16. DB2TMUpconversion to TMsfrom RDBMS via JDBCor from CSVUses XML mappingcan call out to JavaSupports synceither full rescanor change tableTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote
- 17. DB2TM exampleOntopia+=United NationsBouvet<relation name="organizations.csv" columns="id name url"> <topic type="ex:organization"> <item-identifier>#org${id}</item-identifier> <topic-name>${name}</topic-name> <occurrence type="ex:homepage">${url}</occurrence> </topic></relation>
- 18. TMRAPWeb service interfacevia SOAPvia plain HTTPRequestsget-topicget-topic-pageget-tologdelete-topic...TMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote
- 19. Navigator frameworkServlet-based APImanage topic mapsload/scan/delete/createJSP tag libraryXSLT-likebased on tologJSTL integrationTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote
- 20. Ontopia Navigator FrameworkJava API for interacting with TM repositoryJSP tag librarybased on tologkind of like XSLT in JSP with tolog instead of XPathhas JSTL integrationUndocumented partsweb presentation componentssome wrapped as JSP tagswant to build proper portlets from them
- 22. Navigator tag library example <%-- assume variable 'composer' is already set --%><p><b>Operas:</b><br/><tolog:foreach query=”composed-by(%composer% : composer, $OPERA : opera), { premiere-date($OPERA, $DATE) }?”> <li> <a href="opera.jsp?id=<tolog:id var="OPERA"/>” ><tolog:out var="OPERA"/></a> <tolog:if var="DATE"> <tolog:out var="DATE"/> </tolog:if> </li></tolog:foreach></p>
- 23. Elmer Preview
- 27. Automated classificationUndocumentedexperimentalExtracts textautodetects formatWord, PDF, XML, HTMLProcesses textdetects languagestemming, stop-wordsExtracts keywordsranked by importanceuses existing topicssupports compound termsTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote
- 28. Example of keyword extractiontopic maps 1.0metadata 0.57subject-based class. 0.42Core metadata 0.42faceted classification 0.34taxonomy 0.22monolingual thesauri 0.19controlled vocabulary 0.19Dublin Core 0.16thesauri 0.16Dublin 0.15keywords 0.15
- 29. Example #2Automated classification 1.0 5Topic Maps 0.51 14XSLT 0.38 11compound keywords 0.29 2keywords 0.26 20Lars 0.23 1Marius 0.23 1Garshol 0.22 1...
- 30. So how could this be used?To help users classify new documents in a CMS interfacesuggest appropriate keywords, screened by user before approvalAutomate classification of incoming documentsthis means lower quality, but also lower costGet an overview of interesting terms in a document corpusclassify all documents, extract the most interesting termsthis can be used as the starting point for building an ontology(keyword extraction only)
- 31. Example user interfaceThe user creates an articlethis screen then used to add keywordsuser adjusts the proposals from the classifier
- 32. VizigatorVizOntopolyGraphical visualizationVizDesktopSwing app to configurefilter/style/...VizletJava applet for webuses configurationloads via TMRAPuses “Remote” backendTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote
- 33. The VizigatorGraphical visualization of Topic MapsTwo partsVizDesktop: Swing desktop app for configurationVizlet: Java applet for web deploymentConfiguration stored in XTM file
- 36. The VizigatorThe Vizigator uses TMRAPthe Vizlet runs in the browser (on the client)a fragment of the topic map is downloaded from the serverthe fragment is grown as neededServerTMRAP
- 37. OntopolyVizOntopolyGeneric editorweb-based, AJAXmeta-ontology in TMOntology designercreate types and fieldscontrol user interfacebuild viewsincremental devInstance editorguided by ontologyTMRAPNavDB2TMClassifyEngineMemoryRDBMSRemote
- 38. OntopolyA generic Topic Maps editor, in two partsontology editor: used to create the ontology and schemainstance editor: used to enter instances based on ontologyBuilt with the Web Editor Frameworkworks with both XTM files and topic maps stored in RDBMS backendsupports access control to administrative functions, ontology, and instance editorsexisting topic maps can be importedparts of the ontology can be marked as read-only, or hidden
- 41. CMS integrationThe best way to add content functionality to Ontopiathe world doesn’t need another CMSbetter to reuse those which already existSo far two integrations existEscenicOfficeNet Knowledge Portalmore are being worked on
- 42. ImplementationA CMS event listenerthe listener creates topics for new CMS articles, folders, etcthe mapping is basically the design of the ontology used by this listenerPresentation integrationit must be possible to list all topics attached to an articleconversely, it must be possible to list all articles attached to a topichow close the integration needs to be here will vary, as will the difficulty of the integrationUser interface integrationit needs to be possible to attach topics to an article from within the normal CMS user interfacethis can be quite trickySearch integrationthe Topic Maps search needs to also search content in the CMScan be achieved by writing a tolog plug-in
- 43. Articles as topicsis aboutElectionsNew city council appointedGoal: associate articles with topicsmainly to say what they are abouttypically also want to include other metadataNeed to create topics for the articles to do thisin fact, a general CMS-to-TM mapping is neededmust decide what metadata and structures to include
- 44. Mapping issuesArticle topicswhat topic type to use?title becomes name? (do you know the title?)include author? include last modified? include workflow state?should all articles be mapped?Folders/directories/sections/...should these be mapped, too?one topic type for all folders/.../.../...?if so, use associations to connect articles to foldersuse associations to reproduce hierarchical folder structureMultimedia objectsshould these be included?what topic type? what name? ...
- 45. Two styles of mappingsArticles as articlesTopic represents only the articleTopic type is some subclass of “article”“Is about” association connects article into topic mapFields are presentationaltitle, abstract, bodyArticles as conceptsTopic represents some real-world subject (like a person)article is just the default content about that subjectType is the type of the subject (person)Semantic associations to the rest of the topic mapworks in department, has competence, ...Fields can be semanticname, phone no, email, ...
- 46. Article as articleArticle about building of a new schoolIs about association to “Primary schools”Topic type is “article”
- 47. Article as conceptArticle about a sports hallArticle really represents the hallTopic type is “Location”Associations tocity borough
- 48. events in the location
- 54. Two projects
- 55. The projectA new citizen’s portal for the city administrationstrategic decision to make portal main interface for interaction with citizensas many services as possible are to be moved onlineBig projectstarted in late 2004, to continue at least into 2008~5 million Euro spent by launch date1.7 million Euro budgeted for 2007Topic Maps development is a fraction of this (less than 25%)Many companies involvedBouvet/OntopiaAvenirKPMGKarabinEscenic
- 56. Simplified original ontologyService catalogEscenic (CMS)LOSFormArticlenearlyeverythingCategoryServiceSubjectDepartmentBoroughExternalresourceEmployeePayroll++
- 59. The portal
- 61. NRK/SkoleNorwegian National Broadcasting (NRK)media resources from the archivespublished for use in schoolsintegrated with the National CurriculumIn productiondelayed by copyright wranglingTechnologiesOKSPolopoly CMSMySQL databaseResin application server
- 62. Curriculum-based browsing (1)CurriculumSocial studiesHigh school
- 63. Curriculum-based browsing (2)Gender roles
- 64. Curriculum-based browsing (3)Feminist movement in the 70s and 80sChanges to the family in the 70sThe prime minister’s husbandChildren choosing careersGay partnerships in 1993
- 65. One video (prime minister’s husband)MetadataSubjectPersonRelatedresourcesDescription
- 67. ImplementationDomain model in JavaPlain old Java objects built onOntopia’s Java APItologJSP for presentationusing JSTL on top of the domain modelSubversion for the source codeMaven2 to build and deployUnit tests
- 68. What we’d like to seeThe future
- 69. The big pictureAuto-class.A.N.otherA.N.otherOtherCMSsA.N.otherA.N.otherDB2TMPortlet supportOKPXML2TMEngineCMSintegrationData integrationEscenicTaxon.importOntopolyWebservice
- 70. CMS integrationsThe more of these, the betterCandidate CMSsLiferay (being worked on at Bouvet)Alfresco (might be started soon)MagnoliaInspera (possible project here)JSR-170 Java Content RepositoryCMIS (OASIS web service standard)
- 71. Portlet toolkitSubversion contains a number of “portlets”basically, Java objects doing presentation taskssome have JSP wrappers as wellExamplesdisplay tree viewlist of topics filterable by facetsshow related topicsget-topic-page via TMRAP componentNot ready for prime-time yetundocumentedincomplete
- 72. Ontopoly plug-insPlugins for getting more data from externalsTMSync import pluginDB2TM pluginSubj3ct.com pluginadapted RDF2TM pluginclassify plugin...Plugins for ontology fragmentsmenu editor, for example
- 73. TMCLNow implementableWe’d like to seean object model for TMCL (supporting changes)a validator based on the object modelOntopoly import/export from TMCL (initially)refactor Ontopoly API to make it more portableOntopoly ported to use TMCL natively (eventually)
- 74. Things we’d like to removeOSL supportOntopia Schema LanguageWeb editor frameworkunfortunately, still used by some major customersFulltext searchthe old APIs for this are not really of any use
- 75. Management interfaceImport topic maps (to file or RDBMS)
- 76. What do you think?Suggestions?Questions?Plans?Ideas?
- 77. Setting up the developer environmentGetting started
- 78. If you are using Ontopia......simply download the zip, thenunzip,set the classpath,start the server, ......and you’re good to go
- 79. If you are developing Ontopia...You must haveJava 1.5 (not 1.6 or 1.7 or ...)Ant 1.6 (or later)Ivy 2.0 (or later)SubversionThencheck out the source from Subversionsvn checkout http://ontopia.googlecode.com/svn/trunk/ ontopia-read-onlyant bootstrapant dist.jar.ontopiaant testant dist.ontopia
- 80. BewareThis is fun, becauseyou can play around with anything you wante.g, my build has a faster TopicIF.getRolesByTypeyou can track changes as they happen in svnHowever, you’re on your ownif it fails it’s kind of hard to say whymaybe it’s your changes, maybe notFor production use, official releases are best
- 82. Our goalTo provide the best toolkit for building Topic Maps-based applicationsWe want it to beactively maintained,bug-free,scalable,easy to use,well documented,stable,reliable
- 83. Our philosophyWe want Ontopia to provide as much useful more-or-less generic functionality as possibleNew contributions are generally welcome as long asthey meet the quality requirements, andthey don’t cause problems for others
- 84. The sandboxThere’s a lot of Ontopia-related code which does not meet those requirementssome of it can be very useful,someone may pick it up and improve itThe sandbox is for these piecessome are in Ontopia’s Subversion repository,others are maintained externallyTo be “promoted” into Ontopia a module needsan active maintainer,to be generally useful, andto meet certain quality requirements
- 85. CommunicationsJoin the mailing list(s)!http://groups.google.com/group/ontopiahttp://groups.google.com/group/ontopia-devGoogle Code pagehttp://code.google.com/p/ontopia/note the “updates” feed!Bloghttp://ontopia.wordpress.comTwitterhttp://twitter.com/ontopia
- 86. CommittersThese are the people who run the projectthey can actually commit to Subversionthey can vote on decisions to be made etcEveryone else canuse the software as much as they want,report and comment on issues,discuss on the mailing list, andsubmit patches for inclusion
- 87. How to become a committerParticipate in the project!that is, get involved firstlet people get to know you, show some commitmentOnce you’ve gotten some way into the project you can ask to become a committerbest if you have provided some patches firstUnless you’re going to commit changes there’s no need to be a committer
- 88. Finding a task to work onReport bugs!they exist. if you find any, please report them.Look at the open issuesthere is always testing/discussion to be doneLook for issues marked “newbie”http://code.google.com/p/ontopia/issues/list?q=label:NewbieLook at what’s in the sandboxmost of these modules need workScratch an itchif there’s something you want fixed/changed/added...
- 89. How to fix a bugFirst figure out why you think it failsThen write a test casebased on your assumptionmake sure the test case fails (test before you fix)Then fix the bugfollow the coding guidelines (see wiki)Then run the test suiteverify that you’ve fixed the bugverify that you haven’t broken anythingThen submit the patch
- 90. The test suiteLots of *.test packages in the source tree3148 test cases as of right nowtest data in ontopia/src/test-datasome tests are generators based on filessome of the test files come from cxtm-tests.sf.netRun withant testjava net.ontopia.test.TestRunner src/test-data/config/tests.xml test-group
- 91. Source tree structurenet.ontopia.utils various utilitiestest various test support codeinfoset LocatorIF code + cruftpersistence OR-mapper for RDBMS backendproduct cruftxml various XML-related utilitiestopicmaps next slides
- 92. Source tree structurenet.ontopia.topicmaps.core core engine APIimpl engine backends + utilsutils utilities (see next slide)cmdlineutils command-line toolsentry TM repositorynav + nav2 navigator frameworkquery tolog enginevizclassify db2tmwebed cruft
- 93. Source tree structurenet.ontopia.topicmaps.utils* various utility classesltm LTM reader and writerctm CTM readerrdf RDF converter (both ways)tmrap TMRAP implementation
- 94. Let’s write some code!
- 95. The engineThe core API corresponds closely to the TMDMTopicMapIF, TopicIF, TopicNameIF, ...Compile withant init compile.ontopia.class files go into ontopia/build/classesant dist.ontopia.jar # makes a jar
- 96. The importersMain class implements TopicMapReaderIFusually, this lets you set up configuration, etcthen uses other classes to do the real workXTM importersuse an XML parsermain work done in XTM(2)ContentHandlersome extra code for validation and format detectionCTM/LTM importersuse Antlr-based parsersreal code in ctm.g/ltm.gAll importers work via the core API
- 97. Fixing a real bugThere is a failing test case in the TM/XML importerSo let’s fix that right now...
- 98. Find an issue in the issue tracker(Picking one with “Newbie” might be good, but isn’t necessary)Get set upcheck out the source codebuild the coderun the test suiteThen dig inwe’ll help you with any questions you haveAt the end, submit a patch to the issue trackerremember to use the test suite!