ONTOCOM SemTech
- 1. Development effort estimation for large scale business ontologies Presentation Dr. Elena Simperl Dr. Christoph Tempich 20.05.2008 Member of
- 2. Content 1. Motivation 2. Framework 3. Cost Driver 4. Case Study 5. Conclusion 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Page 1
- 3. Development effort estimation for large scale business ontologies Management Summary Ontocom is a framework to help you estimate the effort related to the building of an ontology. It make accurate predictions and can be improved with data from your team. Ontologies are the key enablers for knowledge exchange across the Web Ontocom is a framework to estimate the effort related to ontology building Ontocom comes with a process for effort estimation a formula and a tool calculating the estimations and a methodology to adjust the estimations to a particular company. Ontocom takes the size, the domain, the development complexity, the expected quality and the experience of the staff as input factors 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Ontocom estimates ontology building costs with a 30% accuracy in 80% of the cases We successfully applied the methodology in an ontology development project within a large German telecommunication operator as part of a SOA project. Page 2
- 4. 1. Content Page 3 Motivation 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT
- 5. Motivation Key challenges for telecommunication operators For telco IT organizations, the migration from today´s vertical, silo-type architecture towards a flexible, NGN compliant, horizontal architecture is a disruptive step. Traditional Telco Stove Pipe Architecture Future NGN IT Layered Architecture Customer Customer Voice Mobile Data Integrated, convergent Services CRM, Sales, Order Management CRM, Sales CRM, Sales CRM, Sales Order Mgt Order Mgt Order Mgt Fulfillment Assurance Billing Assurance Fulfillment Assurance Fulfillment Assurance Fulfillment Billing Billing Billing 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT NGN service delivery Fixed Network Access Mobile Network Service Service Service Access Delivery Delivery Delivery vertical architecture horizontal architecture Page 4
- 6. Motivation Reference Architecture Many operators base this architectural change on a Service Oriented Architecture (SOA) in which an ontology enables communication between different applications. Customer Portal Partner Portal Service Factory Customer Management 3rd Party Exposure Order Problem Billing SDP Management Management Policy Enforcement Invoicing Network based Computing Infrastructure Service Elem. Ontology Security Federated ID BPM Master Data Mgmt. ERP Service Bus Connectivity Control Provisioning Management Service Creation Network Dynamic Resource IP/MPLS Transport Resource Discovery 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Control Service Assurance Resource Activation Access Resource Management Service Management Customer Equipment Supplier Management Network Factory Virtualization, ATCA, Blades, Storage Page 5
- 7. Motivation Implementation Timeline The implementation of the new architecture is a long time effort and an accurate esti- mation of development efforts is the pre-requisite for the creation of a good project plan. Elements to align Aligned Project Plan Feb Mar Apr Activity 06 07 08 09 10 11 12 13 14 15 16 17 CRM IT Process IT Data application Resource IT Process 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Data Process Service Ontology model IT Process Data Page 6
- 8. Motivation Ontocom Ontocom is a framework to estimate the effort to develop an ontology. Ontocom consists of a process, a formula and a methodology. Objective Elements 5 ? Process Ontocom 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Formula Methodology CHARTPOOL_A4.PPT Page 7
- 9. Content 2. Framework Process Formula Example Methodology 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Page 8
- 10. Ontocom Framework Cost estimation in ontology engineering processes The project manager of an ontology development project estimates the effort for building, documenting and evaluating the ontology during the requirements analysis. Requirements analysis motivating scenarios, use cases, existing solutions, Knowledge acquisition effort estimation, competency questions, application requirements Documentation Evaluation Conceptualization conceptualization of the model, integration and extension of existing solutions 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Implementation implementation of the formal model in a representation language Page 9
- 11. Ontocom Methodology Process Applying Ontocom is easy and follows a five step process. The project manager defines the different parameters based on the process guidelines which are part of the framework. Step 1 Step 2 Step 3 Step 4 Step 5 Eval uatio Eval Eval Eval n of uatio uatio uatio devel n of Size n of Effort n of op- expe Estimation pers estimation dom ment cted onne ain com quali l plexi ty 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT ty Page 10
- 12. Ontocom Framework Effort Estimation Formula The formula uses information collected in the ontology development process and of historical information collected from previous projects to make the effort estimation. . Parametric Effort Estimation Method PM = A * (Size ) * ∏ CD i B Person Normaliza- Size of the Cost Month tion Factor Ontology Drivers 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Learning Factor Result Input from project manager Input from methodology Page 11
- 13. Ontocom Methodology Example The parameters associated with the different cost drivers are predefined in our calculation tool. Effort Estimation Formula Person Size of the Cost Month Ontology Drivers Quality of personnel Development complexity very high very high 6,9 PM = 500 Entities * high X high 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT X average average low low very low very low Page 12
- 14. Ontocom Methodology Methodology to adapt Ontocom to your company For a high accuracy of the model we calculated the parameters aggregating the experience of well over 40 ontology engineering projects. And counting. Model generation Data collection Data analysis Model Usage Model calibration Specify cost Collect data Analyze data Calibrate Evaluate Release drivers model model model Effort estimations 12.000 11.000 10.000 9.000 +/ -30% tolerance 8.000 average 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT 7.000 estimation 6.000 5.000 4.000 3.000 2.000 1.000 0 0 4 8 1 1 2 2 2 3 3 2 6 0 4 8 2 4 The accuracy of the model increases if it is adapted and calibrated with data from your own business. Page 13
- 15. 3. Content Page 14 Cost Drivers 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT
- 16. Cost drivers Model application Step 1: Size of the ontology Explanation Guidelines The size of the ontology. This includes all first class Determining the size of a prospected ontology is citizens of an ontology. Size is measured in kilo a challenging task in an early stage of the entities. ontology development process. All class definitions Existing domain ontologies can help to get a All attribute definitions rough capture. All relationship definitions 1. Search for existing domain ontologies. All rule definitions 2. Compare coverage of existing domain ontologies with the required level of detail Examples 3. Calculate expected size of the new ontology An ontology has 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT 500 classes 700 attributes 300 relations no rules This totals in 1.5 k entities. Page 15
- 17. Cost drivers Model application Step 2: Evaluation of the domain Explanation Guidelines The Domain Analysis Complexity accounts for DOMAIN those features of the application setting which Very Low: narrow scope, common-sense influence the complexity of the engineering knowledge, low connectivity outcomes. It consist of three sub categories: The domain complexity Very High: wide scope, expert knowledge, high connectivity The requirements complexity REQUIREMENTS The available information sources Very Low few, simple req. Examples Very High: very high number of req. with a high conflicting degree, high number of usability An ontology for the cooking domain, having a requirements 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT low number of requirements and a high number of available information sources has a very low INFORMATION SOURCES to low domain complexity. Very Low high number of sources in various An ontology for the chemistry domain, with a forms high number of requirements and a low number Very High none of available information sources has a high to very high domain complexity. Page 16
- 18. Cost drivers Model application Step 3: Evaluation of the development complexity Explanation Guidelines The Conceptualization Complexity accounts CONCEPTUALIZATION for the impact of a complex conceptual model Very Low: concept list on the overall costs Very High: instances, no patterns, considerable The Implementation Complexity takes into number of constraints consideration the additional efforts arisen from the usage of a specific implementation language IMPLEMENTATION Low: The semantics of the conceptualization Examples compatible to the one of the implementation language An ontology for a search application with an 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT thesaurus has a low development complexity. High: Major differences between the two An ontology for the chemistry domain, modeling reaction patterns has a high development complexity. Page 17
- 19. Cost drivers Model application Step 4: Evaluation of expected quality Explanation Guidelines The Evaluation Complexity accounts for the ONTOLOGY EVALUATION additional efforts eventually invested in Very Low: small number of tests, easily generating test cases and evaluating test generated and reviewed results. This includes the effort to document the ontology. Very High: extensive testing, difficult to generate and review Required reusability to capture the additional effort associated with the development of a REUSEABILITY reusable ontology, Very Low: Ontology is used for this application only Examples Very High: Ontology should be used across An ontology which is used for one application many applications as an upper level ontology 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT only without extensive testing has a low factor. An integration ontology which should be used across an entire organization or for many web users with high documentation requirements has a high or very high factor. Page 18
- 20. Cost drivers Model application Step 5: Evaluation of personnel Explanation Guidelines Ontologist/Domain Expert Capability accounts ONTOLOGIST/DOMAIN EXPERT CAPABILITY for the perceived ability and efficiency of the Very Low: 15% single actors involved in the process (ontologist and domain expert) as well as their teamwork Very High: 95% capabilities. ONTOLOGYIST/DOMAIN EXPERT EXPERIENCE Ontologist/Domain Expert Experience to mea- Very Low: 2 month (ontology) / 6 month sure the level of experience of the engineering (domain) team w.r.t. performing ontology engineering. Very High: 3 years (ontology) / 7 years Examples (domain) The new project member who has never worked 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT with ontologies nor has any experience with the domain has a very low expert experience. The project manager who has been working with ontologies for several years and is experienced in a certain field has a very high expert experience. Page 19
- 21. 4. Content Page 20 Case Study 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT
- 22. Case study Challenge We used the Ontocom process and formula to create a project plan for the development of an integration ontology. The challenge was to estimate the size of ontology. Step 1 Step 2 Step 3 Step 4 Step 5 Eval uatio Eval Eval Eval n of uatio uatio uatio devel n of Size n of Effort n of op- expe Estimation pers estimation dom ment cted onne ain com quali l plexi ty 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT ty Page 21
- 23. Case study Step 1: Size of the ontology For the integration ontology the TMForum has developed the Shared Information Data Model (SID). SID model overview Process Existing domain ontologies The TMForum is an industry organization of telecommunication operators and their vendors. The SID is a UML model formalizing the knowledge relevant to an operators business It has approx. 4.8 k UML elements 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Coverage of the domain Initial analysis shows that average coverage across all sub domain is at around 40% The estimate size is 12 k CHARTPOOL_A4.PPT entities. Page 22
- 24. Case study Step 1: Size of the ontology For a detailed estimation of the required person month to build the ontology in the respective sub domains we analyzed the number of classes in each sub domain. Market / Sales Market Strategy & Plan Market Segment Marketing Campaign Competitor Contact/Lead/Prospect Sales Statistic Sales Channel 52 Customer Customer Customer Interaction Customer Order Customer Statistic Customer Problem Customer SLA Applied Cust. Bill. Rate Customer Bill 74 Customer Bill Collection Customer Bill Inquiry Product Product Product Specification Strat. Prod. Portf. Plan Product Offering Product Performance Product Usage Statistic 66 Service Service Service Specification Service Applications Service Configuration Service Performance Service Usage Service Strategy & Plan Service Trouble 124 Service Test Resource Resource Resource Topology Resource Performance Resource Strat. & Plan 241 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Resource Specification Resource Configuration Resource Usage Resource Trouble Resource Test Supplier / Partner S/P Performance S/P Bill Supplier/Partner S/P Plan S/P Interaction S/P Product S/P Order S/P SLA S/P Problem S/P Statistic 8 S/P Bill Inquiry S/P Payment Enterprise Common Business (Under Construction) 25 Party Location Business Interaction Policy 431 Agreement Page 23
- 25. Case study Steps 2 - 5 We rated the remaining factors in accordance with the circumstances found in the project and following the guidelines of the framework. Step 2 Step 3 Step 4 Step 5 Evaluation of Evaluation of Evaluation of Evaluation of development domain expected quality personnel complexity The telecommuni- The development The expected The personnel cation domain has of the ontology quality was rated was rated slightly an above average has an average very high below average. domain complexity complexity. Due to the cross Experience with Although there are The ontology will organizational use ontology building many information be formalized as documentation was limited in the sources available an UML diagram has to very team. 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT detailed there are many It was not planned Modelers with different to model rules or We needed to experience in the requirements axioms evaluated the telecommunica- ontology tions industry are and it is a broad thoroughly. difficult to find domain. Page 24
- 26. Case study Effort estimation for the development of the data model: June 2007 The model size will increase slowly and it will take approximately 26 person month to develop the complete ontology if engineers can continuously work on it. Effort estimations Result Estimation: 12.000 We used a Excel tool in order to compute the 11.000 duration of the development.* We varied the 10.000 number of entities in order to show the increase 9.000 +/-30% tolerance in entities over time. no. of entities 8.000 Implications: 7.000 average estimation If modelers stay in the development team the 6.000 learning rate is quite high. 5.000 For the development of the entire model with 4.000 approx 12.000 UML elements we estimated an 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT 3.000 effort of 24 month. However, as it is a prediction 2.000 variations of around 30% are still within the 1.000 range of the model. 0 To introduce the first 600 elements plan six 0 5 10 15 20 25 30 35 month. person month *:The model was not adapted with data from the operator. Page 25
- 27. Case study Effort estimation for the development of the data model: June 2007 In order to get a rough estimation for the finalization of the different domains we divided the overall engineering effort into several sub tasks. Effort estimations average 12.000 estimation +/-30% tolerance Enterprise Supplier / Partner 11.000 Market / Sales 10.000 9.000 no. of entities 8.000 Product / Service 7.000 6.000 Customer/ Resource 5.000 4.000 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT 3.000 Common business 2.000 1.000 0 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 person month Page 26
- 28. Case study Comparison with actual numbers in February 2008 The actual effort is higher than expected. This is mainly due to frequent changes in the modeling team and to technical problems aligning the process and ontology model. Actual Effort Evaluation Changes in the development team: 12.000 The team consisted of in average 4 people. 11.000 10.000 The team structure changed quite often due to 9.000 management decisions Entities no. of entities 8.000 This required experienced modelers to train 7.000 newcomers 6.000 Aligning the process model with the ontology: 5.000 Tool support to define the data objects required 4.000 for activities in a process model is limited 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT 3.000 The original model does not account for the 2.000 integration of an ontology with a process model 1.000 Size 0 The estimate of the size of the ontology is 0 5 10 15 20 25 30 35 relatively good person month The project is ongoing Page 27
- 29. 4. Content Page 28 Conclusion 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT
- 30. Lessons learned Ontocom is a Previous experience with ontology building implies a high learning rate. This 1 straightforward can only be achieved if the ontology engineering team is constant. methodology to estimate the effort Identify existing taxonomies, models, classifications in order to get a rough 2 capture of the size of the ontology. related to ontology engineering. Evaluation and documentation of the model for a better reusability is the most The experience 3 time consuming part. incorporated in the methodology includes best practices for 4 A clear methodology is essential. ontology engineering. 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT The model can be 5 The target picture must be clear to increase efficiency. improved with calibration data from your company. Ontology development differs from data modeling and experts are difficult to 6 find. Page 29
- 31. Thank you. Find out more about Ontocom at: http://ontocom.ag-nbi.de/ Member of
- 32. Contact Dr. Elena Simperl Digital Enterprise Research Institute University of Innsbruck ICT Technologiepark Technikerstr. 21a 6020 Innsbruck (Austria) Phone: +43 512 507 96884 Fax: +43 512 507 9872 Mobile: +43 664 812 5236 e-Mail: elena.simperl@deri.at Dr. Christoph Tempich Detecon International GmbH 080413_CT_SEMANTIC TECHNOLOGY_V04.PPT Industry/Competence Practice IT Oberkasseler Str. 2 53227 Bonn (Germany) Phone: +49 228 700-1942 Fax: +49 228 700 – 2361 Mobile: +49 (151) 12720065 e-Mail: Christoph.Tempich@detecon.com Page 31