Notes on Evaluation of CoAX, 23-Nov-00

Background: Prof Paul Cohen visited AIAI on 23/24th October 2000.
Since Paul is very well known for his work on evaluation and empirical
methods in AI, we thought it would be a good opportunity to discuss
possible ideas for evaluation of the Coalition TIE.

1. Two Sorts of Evaluation in CoAX

In CoAX, there are two sort of evaluation we should be doing. One is
to view CoAX as an application of the underlying agent technology (such
as the Grid) and use the data generated by CoAX as a way of testing the
underlying technology in a real application. We regard this sort of
evalution as very valuable and something we can offer to provide to the
CoABS program.

The other evaluation we should do is to evaluate CoAX and the ideas
underlying CoAX as entities in their own right. We spent most of our
time discussing the latter.

2. The CoAX Hypothesis

In CoAX, the central hypothesis we are trying to test is:

  The agent-based computing paradigm is a very good fit for the
  kind of computational support needed for coalition operations.

Additionally, we want to show that the CoAX approach gives us agility,
inter-operability and robustness.

3. Assumptions

We will assume that coalition members will bring software support tools
with them and that those tools will need to be connected in some way in
order to send information to each other.

We can further assume that, although integration may be a high cost
at the moment, this can be ignored - if it turns out that the CoAX
approach is the correct one, then the amount of work we do at the
moment to integrate systems will be much greater than the amount of
work that will be needed if the CoAX method becomes "the norm".

4. Three Evaluation Ideas for CoAX

A. Paul showed us a simple model of connecting agents, based on either
point-to-point connections or using bridging agents. Given the costs
of connecting point-to-point or to a bridging agent, the model can
quickly tell you which approach is optimal, and how many bridge agents
should be used.

B. For CoAX we considered a model of agents which are bounded by functional
domains, with each functional domain being characterised by a set of
axioms. These functional domains are groupings of agents that perform
a task, such that the amount of information they have to explain to 
each other can be minimised. The optimisation problem is, given a set
of tasks T and a set of agents A (where size_of(A) > size_of(T)), to
assign agents to domains so that the resulting system achieves all the
tasks with the minimum amount of information exchange.

C. We also discussed the idea of a game-theoretic formulation of the agent
system we are trying to build, based on either an agent model of connection
or point-to-point. The idea here is to show that the agent model is like
an auction, where everyone can see everyone - in game theoretic terms,
this is regarded as a fair system. Point-to-point connections where only
certain other agents are "visible" can be shown to breed distrust.

5. Brief Discussion

It is not clear to me (John L) that any of these ideas are the right way
to evaluate the CoAX hypothesis, though they all contain elements that
we may be able to use. If we accept our assumption that the systems must
be connected somehow, then 4A might be able to prove that an agent
arrangement is best, given which systems need to talk to which. If we
have functional domains in which "teams" of agents can cooperate to
achieve a task, 4B will be useful - but I'm not sure that CoAX can be
characterised in this way (yet). Given that our agent model has bounded
domains in which an agent may be restricted, I'm not sure we can get
much leverage out of 4C. Perhaps, given our needs for security and control
of (possibily rogue) agents, the CoAX arrangement of agents is the fairest
we can get?

Our central concerns are for agility, inter-operability and robustness.
We need to define what each of these mean - how do you distinguish an
agile approach from a non-agile approach? - and try to design experiments
(which could be pen-and-paper, such as the ones above, or actual runs of
the CoAX system) to show these apply.

Please send me your comments on these notes. Thanks!

John Levine (J.Levine@ed.ac.uk)
6th November 2000