The relevance of explanations of possibilities
to assessing competences
(DRAFT: Liable to change)
It is not widely known by admirers or critics of Popper that he later adopted a more flexible approach to the falsifiability requirement and came to express great enthusiasm for Darwin's ideas Popper (1978), Sonleitner(1986).
In Chapter 2 of my 1978 book The Computer Revolution in Philosophy I claimed that an important type of advance in science could be a theory explaining "How X is possible" for some X, even if the theory did not provide a basis for predicting when instances of X would occur, and was not empirically falsifiable. One of the examples I gave was
The importance of theories that explain "How X is possible" for some X is
further defended in
http://www.cs.bham.ac.uk/research/projects/cogaff/misc/explaining-possibility.html, a paper responding to some criticisms of Chapter 2.
As a result of a recent email exchange with Jose Hernandez-Orallo I realised that the notion of explanations of possibilities could have important applications in designs for methods of assessing competences. This is because the criteria for evaluating theories (or research programmes in the sense of Lakatos(1980)) have some important implications.
Section 2.5.8 of Sloman (1978) states:
(the variety of types of possibility explained)
(the ability to derive that variety from a relatively small explanatory core, often via multiple reasoning steps)
(the amount of detail in the phenomena explained, e.g. not just ability to make discoveries in topology, but abilities to make particular sorts of discoveries in particular situations)
and in their satisfaction of the sorts of criteria given by Lakatos for a research programme to be progressive rather than degenerative.
Such an explanatory theory (e.g. a theory of how human mathematical competences are possible) does not yield predictions regarding who will acquire or demonstrate those competences and under what conditions. Moreover observations of such competences at work does not prove the theory correct. The fact that someone is observed to have made a discovery in geometry proves that making such discoveries is possible, and therefore that there is something to be explained by a theory of how such discoveries are possible. But such a theory may merely demonstrate that if various types of information-processing mechanism and forms of information are assembled then that will make possible certain sorts of mathematical discoveries. But if not enough is known about how to predict when such a mechanism will produce results, then it may be impossible to falsify the theory.
Nevertheless such a theory could have practical utility insofar as it provides the basis of a mode of testing individuals for possession of such competences, as follows:
Suppose T is the best current theory about what makes those competences C1, C2, etc. possible, where T presents a collection of mechanisms (e.g. information processing mechanisms, including forms of representation used, ontologies used, methods of inference, forms of perception, forms of problem-solving, grammars used, etc.)
If we have no other good rival theory explaining the competences demonstrated by individuals P1, P2, ... then we can say that T is the best currently available explanation of how those individuals do what they do.
But if T is an interesting theory with generative power it will typically explain the possibility of far more competences than those demonstrated in the tests, i.e. it can be shown also to explain the possibility of other, related, competences C4,C5,C6...
E.g. the best available explanation of how individuals P1, P2, solve various number-theoretic problems unaided could be that they have something like a grasp of Peano's axioms, including the induction axiom, and various practical skills in the use of those axioms. In that case, we may be able to show that those skills and conceptual achievements can be used to solve many more problems in number theory. This does not imply that everyone who has that combination of abilities will necessarily be able to solve those problems, since there can be many performance errors and lapses, as Chomsky emphasised in the context of linguistic competences).
I suspect that good designers of tests (especially designers of mathematical examinations for advanced students) always have in mind an explicit or implicit theory of what sort of mechanism makes it possible to succeed in those tests. A really good, deep, broad, theory can be used systematically to generate diverse tests that help to rule out alternative explanations of success.
If this is right, then one of the corollaries seems to be that instead of such tests merely producing numerical grades they can produce summaries of mechanisms/competences exhibited by those tested: a potentially far more useful outcome than a grade.
Of course, there are many academic subjects whose practitioners know nothing about cognitive mechanisms or computation, and they may be unable to use the strategy described here. However, there is good reason to believe that gifted teachers have good "implicit" theories related to what they are teaching, and therefore may be able to devise good tests even though they don't have deep theories concerning what they are doing.
Performance in such tests can have positive implications, but cannot have negative implications. The fact that someone tested displays a collection of abilities of the sorts tested for gives strong evidence that that individual has acquired the required mechanisms, but non-performance (e.g. non-answers, or incorrect answers) always leaves open the possibility that something other than lack of competence is to blame. Imaginative readers with a deep understanding of how human minds can vary, or how states of mind in an individual can vary, will be able to think up varied alternative explanations.
TO BE CONTINUED
This paper is
A PDF version may be added later.
A partial index of discussion notes is in