Table of Contents
Empirical Studies of Knowledge Acquisition - or -
Natasha and Mark do time at Leavenworth
Overview
Generations of Protégé systems at SMI
Protégé/2000
Knowledge-base development with Protégé/2000
Building knowledge bases: The Protégé
methodology
Protégé/2000 Ontology-editing tab
Generation of usable domain-specific KA tools
A great case for customized widgets: monitoring
nuclear power plants
PPT Slide
PPT Slide
Some Advances in Protégé/2000
Protégé-2000 adopts the OKBC knowledge model
The race to develop plug-ins
Swapping components
Protégé-2000 plug-ins
But how do we know we’re making progress?
Sisyphus experiments
What is needed
We found a captive audience in Kansas ...
What the rest of the talk is about
High-performance knowledge bases (HPKB) program
Two challenge problems
Why does SMI care about HPKB
Evaluating artificial-intelligence systems
Designing an experiment
Knowledge-acquisition experiment
The problem
Large-scale changes in military doctrine
Domain experts need to interact with knowledge bases
Specific goals for the experiment
Domain: Opposing-force unit organization
Information represented in the knowledge base
Protégé-2000
HPKB tab
Purpose of the experiment
Experiment methodology
Experiment time line
Tasks
Example of a task (task 4)
Preparing for evaluation
Evaluation criteria
Evaluating quality of knowledge entry
Knowledge-acquisition rate (Days 1-3)
KA rate improves substantially with learning
Knowledge base verification: finding errors
Quality of knowledge entry: wrong steps versus
correct steps
Removing the “hangover effect”
Task 6: enter a large amount of data
Error recovery rate
Creating new classes
Retention of skills
experiment: knowledge-acquisition rate
Retention of skills experiment
User satisfaction
Testing the hypothesis: Protégé-2000 versus
HPKB tab
Summary of results
Lessons learned
Lessons learned (2)
Lessons learned (3) |