![Approach Verification
• Waterman [1985] suggested a
framework for assessing problem fit.
– It is composed of requirements,
justification, and appropriateness.](https://image.slidesharecdn.com/designingtheexpertsystem-121115090257-phpapp02/85/Designing-the-expert-system-11-320.jpg)
Designing the expert system
- 1. Designing Expert Systems • Selecting the appropriate Problem – Why build the expert system? • Need clear identification of the problem, expert, and users. – What is the payoff? • What is the return on investment in money, increased efficiency, or any other advantages. – What tools are available to build the system?
- 2. Designing Expert Systems – How much will it cost? • The cost is dependent upon the people, resources, and time devoted to its construction. • Also the costs of hardware and software as well as training need to be considered.
- 3. Problem definition Phase I Needs assessment Project Initialization Evaluation of alternative solutions Verification of an ES approach Feasibility study Cost-benefit analysis Managerial issues consideration Organization of development team Phase II Conceptual design and plan System analysis Development strategy and design Sources of knowledge Computing resources Build small prototype Phase III Testing, improving, expanding Rapid Prototyping Demonstrating and analyzing feasibility Completing design
- 4. Completing the Knowledge base Testing, evaluating, and improving the Phase IV knowledge base System Development Planning for integration Acceptance by users Installation, demonstration, deployment Phase V Orientation, training Implementation Security Documentation Integration, field-testing Operation Phase VI Maintenance and upgrades Post-implementation Periodic evaluation
- 5. Phase I • Phase I of building an expert system is the project initialization. – This phase requires: • Problem definition • Needs Assessment • Evaluation of alternative solutions • Verification of the ES approach • Cost-benefit analysis • Consideration of managerial issues – This phase ends when the project has been approved in principle.
- 6. Problem Definition • A clear problem definition will help to simplify the remaining tasks and generate a productive program. – The problem definition should include as much supporting material and information as possible.
- 7. Need Assessment • The Need assessment is a formal study to understand the problem or need.
- 8. Alternative Solutions • One should always consider if there is an alternative solution to a problem. – Lack of knowledge does not necessarily mean that there is a need for an expert system. – Can the information be made available from the expert in some other manner? – Can education and training be used to communicate the desired information?
- 9. Alternative Solutions – Can documentation of the knowledge be used? – Is this problem better suited for conventional programming techniques? – Can the information be purchased or obtained from the WWW, or some other external source?
- 10. Approach Verification • Even if there does not appear to be an alternative solution, an expert system may not be the best approach.
- 11. Approach Verification • Waterman [1985] suggested a framework for assessing problem fit. – It is composed of requirements, justification, and appropriateness.
- 12. Approach Verification • Waterman’s Requirements – The following 12 requirements must be met: • The task can not require common sense. • The task requires only cognitive, not physical, skills. • There must be one genuine expert who is willing to cooperate. • Experts involved can articulate their methods of problem solving.
- 13. Approach Verification • Experts involved can agree on the knowledge and the solution approach to the problem. • The task is not too difficult. • The task is well understood and is defined clearly. • The task definition is fairly stable. • Conventional (algorithmic) computer solution techniques are not satisfactory. • Incorrect or non-optimal results generated by the ES can be tolerated. • The task’s vocabulary has not more than a couple hundred concepts.
- 14. Approach Verification • The second phase of Waterman’s framework is Justification. – At least one of the following must be present to justify the ES development: • The solution to the problem has a high payoff. • The ES can preserve scarce human expertise. • Expertise is needed in many locations. • Expertise is needed in hostile and hazardous environments.
- 15. Approach Verification • The expertise improves performance or quality. • The system can be used for training. • The ES solution can be derived faster than that which a human can provide. • The ES is more consistent or accurate than a human.
- 16. Approach Verification • The final element to Waterman’s framework is Appropriateness. – Each of the following factors should be considered: • Have a symbolic structure and heuristics should be available for the solution. • Not be too easy nor too difficult for the human expert. • Have a manageable scope size.
- 17. Feasibility Study • Many times a feasibility study will be used to determine if an expert system should be developed. – Such a study should include the economic feasibility, technical feasibility, and operational feasibility and impacts.
- 18. Economic Feasibility • Economic feasibility considers – The cost of system development in an itemized fashion. – The cost of maintenance (usually much more). – The anticipated payoff. – A cash flow analysis. – A risk analysis.
- 19. Technical Feasibility • Technical feasibility covers: – Interface requirements – Networking issues – Availability of knowledge and data – Security of confidential knowledge – Knowledge representation scheme – Hardware/software availability and compatibility
- 20. Operational feasibility • The operational feasibility considers: – Availability of human and other resources – Priority as compared to other projects – Need assessment – Organizational and implementation issues – Management and user support – Availability of experts and knowledge engineers. – Legal and other constraints – Corporate culture – User environment
- 21. Cost-benefit Analysis • The purpose of cost-benefit analysis is to determine the viability of a project by looking at the costs versus the anticipated benefits. – This is hard to do for expert systems. – It is best to start this process with a well defined scope of the project and amount of time required for each participant.
- 22. Cost-benefit Analysis • The development costs that should be considered are: – Development tool cost – Any additional hardware cost – Cost to gather expertise from expert(s) – Cost of knowledge engineer and/or programmer. – Cost of time to test, debug, and maintain program. – Possibly, cost of outside consultants.
- 23. Cost-benefit Analysis • It is usually more difficult to assess the benefits. – Some benefits are intangible. – Some benefits can not be related to a single cause. – Some actions may occur over long periods of time and are therefore, difficult to predict. – The valuation of benefits should include an assessment of both quantity and quality. – Multiplicity of consequences can pose a major problem for quantification.
- 24. Managerial Issues • Managerial support is required to complete any project. Issues to consider are: – All parties and management are convinced of the project’s value. – Someone in top management is strongly sponsoring the project. – End-users are involved early, and receive proper support and training. – Availability of financing, and other resources. – Legal and other potential constraints.
- 25. Phase II • The second phase is the system analysis and design. – This includes: • Conceptualization and design • Development strategy • Sources of knowledge • Computing resources – This phase ends when there is an approved and complete project plan.
- 26. Conceptual Design • The conceptual design should provide a general idea of what the system will look like and how it will solve the problem.
- 27. Development Strategy • There are several strategies that can be used to develop the system. – Do it yourself. • This implies the organization has the skills and resources. – Hire an outside developer • Either a consulting firm or a consortium of companies.
- 28. Development Strategy – Participate in a joint venture • The vendor develops the system while testing its own products and services. • Sponsor a university research project. – Merge, acquire, or become a major stockholder in an AI company
- 29. Sources of Knowledge • The sources of knowledge are many, but the primary source will be a human expert. – A human expert should posses more complex knowledge than can be found in documented sources. – The more human expertise is required, the longer and more complicated the knowledge acquisition will be.
- 30. Sources of Knowledge • Questions that may need to be answered when selecting an expert are: – Should the expert be compensated for their contributions? – How will it be determined if the experts are truthfully describing the solutions? – How can experts be assured that they will not lose their jobs, or that the job will not be de-emphasized? – Are the experts concerned about other people in the organization who may suffer consequences of the new ES?
- 31. Sources of Knowledge • Attributes to look for in an expert: – Highly developed perceptual attention ability. – Awareness of the difference between relevant and irrelevant information. – Ability to simplify complexities. – Strong communication skills. – Knowledge of when to make exceptions. – A strong sense of responsibility for their choices. – Selectivity about which problems to solve.
- 32. Sources of Knowledge – Outward confidence in their decisions. – Ability to adapt to changing task conditions. – Highly developed content knowledge about their area. – Greater automaticity of cognitive processes (tasks are a habit to the expert.) – Ability to tolerate stress. – Capability to be more creative. – Ability to articulate their decisions processes.
- 33. Sources of Knowledge – Thorough familiarity with the domain, including task expertise built up over a long period of task performance. – Knowledge and reputation such that the system will have credibility and authority. – Commitment of a substantial amount of time to the development of the system. – Capability of communicating his or her knowledge, judgment, and experience.
- 34. Sources of Knowledge – Cooperative, easy to work with, and eager to work on the project. – Interest in computer systems, even if they are not a computer specialist.
- 35. Computing Resources • There are many software options for developing Expert Systems. – The higher-level the software the less programming that is required. But, it is less flexible.
- 36. Programming Languages • Expert systems can be developed using various languages. – The languages may be pure AI languages like PROLOG and LISP or may be more object oriented languages. – Object-oriented languages have been found useful given the complexity of problems. • Some problems can be simplified by using inheritance.
- 37. Programming Languages • The best programming languages tend to be those that combine procedure- oriented, rule-oriented, object-oriented, and data-oriented programming. – These tend to be more flexible.
- 38. Shells • A shell is basically an abstract application environment. – The advantage of shells is that knowledge can be easily represented. – Shells tend to be built once and then reused for many applications.
- 39. Shells Knowledge Base Shell Consultation KB Editor manager and Debugger Explanation KB Mangmt Inference Program Facilities Engine
- 40. Shells • While shells are easy to use there are also disadvantages: – They can be inflexible. – It may be difficult to fit into nonstandard problems and tasks. – Since they are end-user tools, they are subject to end-user computing problems.
- 41. Environments • Development environments usually support different ways of representing knowledge and handling inferences. • Hybrid System Environments create a programming environment that enhances building complex specific systems or complex tools.
- 42. Environments • Environments tend to be more specialized than languages so they can sometimes increase development productivity. • Environments also tend to be more flexible than shells.
- 43. Phase III • The third phase of expert system development is rapid prototyping and a demonstration prototype. – A prototype is a small-scale system that represents some of the knowledge captured. – Prototypes are important to establish the structure of the knowledge base before building more rules.
- 44. Phase III • The advantages of prototypes are: – They allow the developers to get a sense of the feasibility of a full application. – They provide the ability to study the effectiveness of the knowledge acquisition and representation. – They can be used to identify gaps or important problems in the proposed system. – They provide a tangible product early in the development.
- 45. Phase III – They provide an opportunity to demonstrate the system capabilities. – They allow for midcourse corrections based upon management, consulting expert, and user feedback. – They provide a system that can be tested. – They may provide a system that can be field tested. – They accelerate knowledge acquisition.
- 46. Phase III – They facility expert feedback and provide exceptions to the rules. – They make it easier to win over skeptics. – They help sustain the expert’s interest. – They can help build user support. – They can provide an idea of the value of the system. – They provide information about the initial problem domain definition.
- 47. Phase IV • The fourth development phase is the System Development. – During this phase the knowledge base is built and continuous testing, reviews, and improvements are carried out.
- 48. Building the KB • Building the KB involves acquiring and representing the knowledge in an appropriate computer format. – The first step is to organize the domain knowledge. • List all possible solutions, outcomes, answers, choices, or recommendations. – Then define the input facts by identifying and listing all the data that is required by the system.
- 49. Building the KB – It may be useful to develop an outline of the KB even though the potential solutions and outcomes are known. – A decision tree may be easily developed as some types of knowledge will organize itself into the tree structure. – Other types of knowledge will nicely organize into a matrix. – Finally, once the rules are written they can be entered into the KB.
- 50. Testing, Validating, etc. • Testing and evaluation should occur throughout the entire process. – Begin with the prototype and continue through the entire system development. – Evaluation questions. • What characteristics should be evaluated? • How should performance be evaluated? • How should the test problems be selected? • How should the system’s mistakes be evaluated?
- 51. Testing, Validating, etc. • Validation attempts to determine if the system was built correctly or whether the system does what is supposed to do with an acceptable level of accuracy. • Verification determines that the ES has been built to correctly implement the specifications.
- 52. Phase V • The fifth phase is implementation. – Some implementation considerations are: • The user’s acceptance of the system depends not only on ease of use and quality but also on behavioral and psychological considerations. • The system is ready for implementation when it reaches a minimum level of stability and quality. • Demonstrate as much as possible to the users. • Ensure that the deployment mode is chosen and verified. • User acceptance requires a high level of training.
- 53. Phase V • If the knowledge in the KB is proprietary, steps need to be taken to protect the knowledge. • Do not forget the documentation. • At the end of Phase V, if the system is a stand alone system, it is ready for field testing.
- 54. Implementation Pitfalls • Waterman lists many pitfalls during implementation and suggestions for avoiding such pitfalls: – The expert’s knowledge can become entwined with the rest of the program. Making it impossible to separate it from general knowledge about search and problem solving. • Use a rule-based organization.
- 55. Implementation Pitfalls – Even after a long knowledge acquisition process, the KB can still be incomplete,or not able to solve simple problems because fundamental concepts are missing or are ill-represented. • Such issues can be identified early by using incremental development and testing. – The development environment does not have built-in explanations and it is non- trivial to add such capability. • Design the system from the beginning with transparency.
- 56. Implementation Pitfalls – The system is slow and unwieldy because of a very large number of specific rules. • Where possible combine smaller rules into more general rules.
- 57. Implementation Suggestions • Work with a core set of representative problems and keep a library of cases that are presented to the system. • Separate domain specific knowledge from general problem solving knowledge. Keep the inference engine simple. • If a rule looks big, it probably should be broken down.
- 58. Implementation Suggestions • If several rules look very similar, look for an underlying domain concept and combine them. • Group rules into rule sets. • Adopt a set of programming style conventions to give the program a uniform look. – Implement similar functions the same way throughout the system
- 59. Implementation Suggestions • Do not be afraid to throw out prototypes and previous implementations. • Make sure the developers are trained in developing expert systems.
- 60. Phase VI • The sixth stage is the post-implementation. This phase includes: – Operation – Maintenance – Expansion – Evaluation
- 61. Operation • The operation of the system is concerned with: – Delivering the system – Providing user training
- 62. Maintenance • The maintenance of an expert system is never done. It is the nature of the beast. – The system will be revised on a regular basis. • Knowledge and understanding change. – Maintenance also includes the traditional tasks of fixing software and hardware bugs and system upgrades.
- 63. Expansion • Expert systems are always changing and expanding as new knowledge is added or old knowledge is updated. – It is best to schedule periodic reviews of the knowledge.
- 64. Evaluation • Periodic evaluation of the expert system is necessary, but may be needed less frequently if the users provide continuous feedback.
- 65. Evaluation – Questions that are asked: • What is the actual cost of maintaining the system compared to the benefits? • Is the maintenance sufficient to keep the knowledge up to date so that the system maintains a high level of accuracy? • Is the system accessible to all users? • Is acceptance by users increasing?
- 66. Why Expert Systems Fail? • The following are reasons that expert systems have failed either during development or after deployment: – The ES no longer solves a meaningful problem or can not provide a meaningful solution. – The maintenance costs are too expensive. – The system is not upgraded to continue to work with the computing environment.
- 67. Why Expert Systems Fail? – The company or industry change focus. – The task domain size was too large and could not be reasonably represented. – The users do not perceive the problem as critical. – The solutions are incorrect or inaccurate. – User resistance. – No one is willing to provide maintenance for the system. – Loss of key development personnel.
- 68. References • Expert Systems Principles and Programming, third edition by Giarratano and Riley; 1998. • Decision Support Systems and Intelligent Systems, sixth edition by Efraim Turban and Jay E. Aronson; 2000. • D. A. Waterman (1985). A guide to Expert Systems. Reading, MA: Addison-Wesley.