KM Tools: NaCoDAE
Evaluation by: Kane Assemi
4-30-2001
NaCoDAE (the Navy Conversational
Decision Aids Environment) is a decision aids development
shell using conversational case-based reasoning built by members of the
Intelligent Decision Aids Group at NRL's Navy Center for Applied Research in
Artificial Intelligence (NCARAI). NaCoDAE is currently the primary focus of research in
the area of CBR technology for the DoD and was developed in an effort to mimic
CBR tools by Inference Corporation. It is based on conversational case-based
reasoning (CCBR) technology in which users interact with a case-based
reasoning (CBR) system, incrementally providing a problem description
that NaCoDAE matches with the cases (i.e., problem/solution pairs) stored in its
case library. NaCoDAE responds to this ongoing dialogue with two displays of
ranked items. One display contains the most similar stored cases' solutions. The
other display lists questions whose answers, if given, can be used to embellish
the problem description and further distinguish which of the ranked cases is
most similar to the user's current problem. The user has the option of answering
more questions or selecting a solution for their problem. The current version of NaCoDAE contains the following
functionality:
- Case Libraries: load, create, edit and save.
- Browsing: cases, problem descriptions (questions) and solutions (actions).
- Problem Solving: conduct interactive or simulated problem solving sessions
(conversations).
- Parameter Settings: load, browse, and edit.
- History: automatically collects and formats results on problem solving
sessions.
- Library Revision: automatically revise case libraries to case authoring
guidelines with CLIRE (Case LIbrary REvisor).
NaCoDAE is written as a standalone application in Java and can therefore run
on any system containing a Java Virtual Machine. The application can be obtained
via email for limited use by filling out a small form at the NaCoDAE
Project Home Page.
Knowledge for this system is stored in
text files as case libraries. Each case library contains a set of questions,
problems (cases) and solutions for a particular domain. These libraries may be
created or modified directly through the text file using a syntax provided in
the appendix of the user manual or more conveniently via the User Interface.
Creating case libraries via the text document can be quite tedious and
problematic unless using a script to translate from an existing alternate case
base. Modifications and case base authoring is intended to be handled via the
provided case library browser. Although this method is less tedious, allowing
more of a point-and-click approach, it is not entirely intuitive. Adding a new problem/solution pair to the system involves creating a case that
describes the problem, creating the action that should be taken to solve the
problem and identifying questions and their respective answers that lead to the
particular case. New questions may be required to lead up to the particular case
description. Unfortunately there is no mechanism for associating a
belief/disbelief or certainty factor with particular answers to questions. Also,
multiple and negated answers (using conditional operators AND, OR, and NOT) are
not permitted in case conditions. For example, Case #1 might be obtained if the
answer to Question #4 is 'red' or 'blue' but not 'green'. This form of case
authoring is not allowed. Additionally, after editing or adding a case library, the library
may be automatically revised by the CLIRE (Case LIbrary REvisor)
module to increase the quality of the case base (explained later in Knowledge
Retrieval).
As mentioned above, the case libraries
of the system are stored as text files, one file for each library. However, this
is intended to be transparent to someone using the system strictly for problem
solving, not case authoring. These files
are designed to be viewed using the aforementioned browser provided. Using
this browser, question, case and action objects can be easily viewed, added,
deleted or modified.
Fortunately a browser mode is provided
via the UI for viewing and editing question, case and action objects as opposed
to working directly with the text files. Unfortunately the objects are depicted
individually with their relations listed sequentially as text. A more
informative visualization might involve a more graphical representation using
nodes to represent the objects and lines which represent associations. This form
of visualization is used in another similar CBR application by Haley
Enterprises, Inc. and is much more informative than a strictly textual
representation.
In interactive mode the visualization is much more effective. After entering
a query in the convenient query box, a list of ranked questions is presented in
ascending order. Below this list of ranked questions is the list of ranked
similar cases in ascending order. Colored bars are used to clearly indicate
rankings. This setup allows the user to easily answer questions and clearly view
all implications from those answers. This mode is much more intuitive than the
browser mode for case authoring and is a good depiction of the necessary scope
of knowledge to be presented. Other CBR applications which I have evaluated use
a similar interface for the interactive mode.
Knowledge is retrieved from the system
in interactive mode via a query. This query can be in a conversational form
which is closer to natural language than using strictly keywords, but still
remains only a partial description. The query is compared to the description of
each case object for similarity and the highest ranked similar cases are
displayed to the user along with the highest ranked questions related to the
case. Parameter setting for the system can be used to change thresholds used in
the similarity function. Unfortunately I was unable to find the method by which
this similarity test is conducted. The query is further enhanced by answering
questions and narrowing down the possible cases while at the same time
increasing the similarity of an existing ranked case to that of the current
problem.In order to maximize effectiveness and efficiency, case libraries may
be run through the CLIRE (Case LIbrary REvisor) module.
This module is supposed to maximize the reuse of questions and eliminate
non-distinguished questions. However, it seems that this only applies to a very
small set of case libraries as identified by the documentation. Later versions
should take more advantage of the CLIRE module.
The NaCoDAE system runs not only in
interactive mode, but also in a simulated batch mode. In this simulated batch
mode, the application runs a series of interactive sessions, using a 'Rover' as
a simulated human to query the case base and answer questions leading towards
the solution of a problem. After processing these simulations, the application
reports back with a retrieval precision and a retrieval speed score for that particular
case library simulation run. Retrieval precision is measured by whether a
retrieved solution is appropriate for the user's problem and retrieval speed is
measured by the number of questions answered before solution retrieval. The
intended goal of this functionality is to give the user an idea of the quality
of a particular case base during case authoring. Low scores would indicate
that perhaps the case library is not inclusive or ineffective and requires more
work. Although parameter settings provided allow for manipulation of the
'Rover', unfortunately I fail to see how this 'Rover' could be very accurate in
depicting a human interaction with the case base without some serious
work.
This particular CBR application, as with
most CBR applications, is designed for the collaboration of knowledge. The
effectiveness of a CBR system is primarily driven by the completeness of the
case library which is intended to be expanded over time with new real-world
cases. However, since the application runs in standalone mode and is intended
for one user, it does not provide the facilities for collaboration of knowledge
from multiple users. One user would have to handle the collaboration externally
and then add all of the new cases manually. The application may be better suited
as a client/server application with multiple sessions allowed via multiple
clients and the case library managed by the server.
Case-Based systems are used
particularly for the collection of knowledge for an entity over time. As
mentioned above, the successfulness of the case library is dependent on the
addition of new real-world cases. The ability to add new cases to an existing
case library allows for a type of organizational memory. However, this capture
of corporate knowledge is not handled automatically by the system. Someone in
the corporation would be responsible for capturing this information and feeding
it to the system. The system is geared more towards querying existing corporate
knowledge.
Much work is still being done on the
NaCoDAE project, especially in the area of case library authoring and revision.
Future functionality includes integrating a database to provide thesaurus
support and additional inferencing capabilities. A goal of the system is to be
able to automatically answer questions that are implicitly answered within the
initial query.
Current NCARAI Projects involving NaCoDAE
Currently, the NCARAI is pursuing the development of NaCoDAE in the context of
the following projects:
- Intelligent
Search in the DON CIO's CD-ROM on Knowledge-Centric Organizations
- Integrated
Plan Authoring Tool Suite (ONR; FY01-)
- Inferencing
in Support of Active Templates (DARPA; FY00-03)
Previous Projects involving NaCoDAE
- Case-Based
Plan Authoring for Interactive Decision Support (ONR 6.2, Command,
Control, & Combat Systems)
- Collaboration
with ISLE MURI recipients on Intelligent Adaptive Agents (ONR
6.2, Artificial Intelligence)
- Affordability
Prediction and Modeling Analyses (ONR 6.1/6.2, Affordability Modeling)
- Active
Knowledge Management for Interactive Decision Support (ONR 6.2
Artificial Intelligence)
- Conversational
CBR Research (In a Cooperative Research and Development Agreement
(CRADA) with Inference
Corporation, 1996-98)
NaCoDAE used in Other Projects
NaCoDAE is distributed by request. Some folks have/are including it in their
projects or courses, which we list here. (Additions welcome.)
- Search
tool for a Out-of-Family Disposition Process (Professor Irma
Becerra-Fernandez & Student Claudia Alarcon, Florida International
University, a NASA-Kennedy proposed project)
- One
of the search tools for the SMART Knowledge Management Portal (Walter
Moleski & Ed Luczak, a NASA-Goddard project)
- Incident
Analysis (Professor Chris Johnson & Student Peter McElroy,
University of Glasgow)
- One
of the tools used in a three-day International Course on CBR (David
Leake & Boi Faltings, April 1999, Lausanne, Switzerland)
- Agent Storm (Joseph Giampapa and Katia Sycara, Carnegie Mellon University)
- Prototype Diagnosis System for Mood Disorders (Psychiatry) José
Barahona da Fonseca