Discussions on the UKCRC and CPHC email lists about research funding, including mention of the high proportion of research time and funding budgets that goes into writing and assessing proposals recently provoked me into reviving an idea that always seems to produce shock and horror, even though I think that (with suitable refinements) it could be an excellent way to fund research, namely, using a properly designed lottery.

There are two key premises

1. When doing real research you often don't know what you are going to do when, because what you do as a researcher exploring unknown territory depends on what you have previously learnt, which keeps changing. So assessing proposals on the basis of what people say they are going to do, forces people to artificially circumscribe research, producing detailed charts and time-tables which make their proposals look professional to people who don't understand what deep science is, and I suspect also reduces depth, originality, and long term importance of much of the research.

   Of course the real function of all that paraphernalia is to enable politicians, funding agencies, and their officers, to proudly claim that they are managing research funding to ensure that tax payers' money is not wasted. And thereby much money gets wasted and some really good research doesn't get funded because it is too speculative to get through the selection process. And researchers learn to think in well-defined grooves because that's what gets them grants, and tenure and kudos in their institutions.

   NOTE:
   This objection to advance planning of research does not apply to industrial or other development projects, or scientific projects that start from a specific practical target to be achieved, which requires working backwards through intermediate stages in order to decide what to do first. That includes 'big science' projects that require building expensive equipment, going on long journeys, e.g. oceanography research, space exploration, building large telescopes, etc. I am not concerned with such research here, Such proposals inevitably require a great deal of prior scrutiny before public resources are allocated to them. The need for detailed planning is also required in research that requires collection and processing of a lot of measurements to fill specific gaps in some region of scientific knowledge, e.g. large scale surveys, long term biological or astrophysical data-collection research, etc.

   In addition to supporting such research we should also encourage high calibre scientists to investigate hard and deep problems in more open-ended, less predictable research projects. Often that involves tasks like:

   • Trying to find a proof for some mathematical theorem
   • Trying to design a formalism, algorithm or architecture that has certain applications or explanatory capabilities
   • Trying to come up with a deep explanatory mechanism unifying a range of previously known phenomena
   • Trying to identify relationships between different disciplines or research areas that are relevant to unsolved problems in one or both
   • Trying to develop a new conceptual framework to make sense of a collection of puzzling facts or unresolved conflicts between theories, or...
   • Surveying existing research publications in order to search for clues to some hard problem
   • Developing a new technique for acquiring information that has so far been hard to acquire
   • Designing a new way of processing old information that can yield important insights previously not accessible or that produces fewer misleading results (e.g. new kinds of data-mining, or new statistical techniques)
   • Designing a new model that can simulate or explain some existing observed behaviours of humans or other animals: typically one does not know in advance what such a model will turn out to be, as the process of designing and testing usually reveals previously unnoticed requirements and options

   Researchers working on such problems may be able to say something fairly detailed about how they are going to start the research, but relatively little about will happen after the early stages, since everything after that is conditional on the early results.

   This contrasts with most of the research proposals I have read which state that the project will do X, then Y, then Z, etc. giving approximate times for all objectives to be achieved.

   Very few of those are research proposals. They are merely attempts to get money by conforming to prescribed application formats.

2. There is too much effort involved both in getting the resources to do research and in evaluating proposals. Both kinds of effort could be reduced by a carefully designed weighted lottery system (given that the current system is often said to be largely a lottery anyway).

In short: (a) spending large resources on assessing research proposals tends to force research into areas where fairly detailed advance planning and prediction of results is possible, so that there is something to assess in advance, leading to too much emphasis on shallow and relatively trivial (and therefore predictable) research, and (b) the processes of generating such proposals and assessing them consume a lot of resources that would be better spent on doing research.

DISCLOSURE:
Most of my own UK research grant proposals have been turned down, though a few were successful. I have had a few funders approach me unasked and offer to fund my research, however. I have also recently found the EC cognitive systems initiative, which is quite visionary, willing to fund my research, though the process is over-managed.

A proposal

1. Universities (and other organisations) that wish to do research should take great care when hiring people -- much more than is normally done in universities in the UK, compared with commercial/industrial research centres, and compared with universities in the USA (at least the ones I know about). Moreover the selection should be based on research potential, originality of ideas, the ability to give exciting presentations, etc. rather than on numbers of publications in journals and previous grant income (factors expected to influence RAE results as opposed to the growth of knowledge).

   If most of the people doing research have already been pre-selected as outstanding researchers, there is less need to assess their specific research in advance of their doing it. So we need a mechanism that gives them high research credibility initially. EPSRC's existing First Grant scheme does that though though it can only benefit a very small minority of new researchers and has the previously mentioned flaws of wasting resources on attempting to assess work that has not been done yet and distorting proposals to fit the grant assessment procedures. It also has a vicious 24 month cut off period (after appointment), which can be grossly unfair, since different amounts of time may be required to identify and clarify different research problems.

2. Individuals appointed as research active academic staff should automatically have the right to take part in the lottery-based research funding process described below, though a university or department might remove or postpone that right in cases of doubt (a decision not to be taken lightly, and which should be subject to appeal). In most cases if universities were properly funded, internal funding could be used to give new researchers a chance to prove themselves.

3. Each individual's lottery bids will be weighted by a factor that starts high, then decays with time (using a backward sigmoid curve), but can be boosted, as described below. This is unlike the 'First Grant' scheme which has a sharp temporal cutoff, as mentioned above.

4. More of the research funding that is now based on competitions run by research councils should instead be allocated directly to research active departments for internal allocation, especially for use by young researchers. Internal allocations should be based on judgements about individual promise, past achievements, and evidence of ability to identify hard problems and investigate them in depth. Some of this funding can take the form of reduced teaching loads or sabbaticals, e.g. for theoretical scientists, for whom there seems to be grossly inadequate provision in the research council funding schemes.

   There should be sufficient direct funding to universities to allow them to give very good research support for all young researchers for the first few years (partly in the form of reduced teaching and admin loads, but also travel, equipment, and in some cases RAs). Mentoring schemes can help nurture young researchers, and also detect whether things are going wrong so that projects should be terminated or redirected -- which research councils don't do, fortunately. (The EC attempts this, at great administrative cost to all concerned.)

   If this reduces the amount allocated by research councils that's not necessarily a bad thing if it transfers resources from administration to research, as well as ensuring better help and guidance for young researchers during their first projects. Of course, university managers would have to be prevented from reallocating that funding for other purposes, which many will try to do. Incidentally I am sure that the most important effect of the 'Full economic costing' scheme over time will be a bad one: namely transferring funding from direct grants to universities to competitively allocated budgets managed by research councils. That is not the intention of the supporters of the scheme, but watch what happens in the next five years. I suspect that very soon it will lead to unfair discrimination within underfunded universities between researchers who get research council funding with FEC and those who don't. (That may look like good business acumen because it motivates people to strive for grants, but is actually bad management because it wastes effort that should be spent on research and is potentially divisive, undermining collaborative and mutually supportive research cultures.)

   NOTE:

   All this assumes that not all university departments should get research funding. We need to revert to a system with a far better division of labour, with a significant subset of higher education done in institutions that are dedicated to high quality education and training such as many polytechnics used to provide, and which both the nation and many school leavers need. Those institutions and the people in them should be well rewarded for what they do, and not punished at all for not doing research. This issue is discussed further in a document pointing out that views on both sides of the debate on top-up fees arise from a failure to plan a well integrated broad spectrum higher education system.

5. Research oriented universities (or departments) should not be punished (as the RAE tends to punish them) for having significant subsets (e.g. as many as 25% or even 30%) of their staff who are not research active but are dedicated to keeping up with research results relevant to their teaching, producing high quality teaching, helping with administration and management and helping research active staff understand requirements for good teaching. All permanent academic staff who do research should also do some teaching since communication of knowledge is as important as discovering it. But some division of labour can be more fruitful than expecting everyone do equal shares of everything.

The lottery [still a rough draft]

Applicants will be people on academic posts who are now eligible to be PIs on EPSRC proposals. Funding from research councils to such individuals who apply for grants will in most cases be allocated in the lottery on the basis of a probabilistic formula which combines elements described below. Very large grant proposals will be dealt with differently, as explained above, using a pre-allocated budget, as will some development proposals aiming at achieving specific practical goals identified by the Research Council. Projects above some size limit will have to be vetted before being accepted for the lottery. Call the proposals eligible to be funded by the lottery 'eligible' proposals.

Every recognised individual researcher N will have at any time a weight W(N) (e.g. between 0 and 100) that will be used in the lottery if N submits a proposal. W(N) will change over time, as explained below.

Some individuals may temporarily contribute some or all of their weight to a virtual individual that is a group of individuals who wish to collaborate and submit a joint proposal to the lottery.

1. For each individual N, an initial weight W(N) will be assigned which starts high at the beginning of N's first appointment to an academic post and drops automatically over time. The rate of fall should be very slow (or possibly the curve should be flat) until the first proposal is submitted by N. After that the weight W(N) should fall more rapidly, though that can be countered by weight-boosting mechanisms described below.

2. Every time a proposal is submitted by N, whether it is proposal fed to the lottery scheme or another mode of selection, W(N) is reduced thereafter temporarily, either for a year or if the project is funded for half (or some other fraction) of the project's length. The amount of reduction of W(N) could depend on the size of the grant awarded. This would also apply to new researchers. So some will be deterred from rushing to submit immature proposals, which is what happens if there are sharp temporal deadlines.

3. If N gets one or more grants, then at intervals the research results produced by N are assessed by some means. The outcome is used to boost or lower W(N). We could have a long debate about details, such as use of citation measures and other things, but the debate can be shortened if we accept that there is no perfect solution and every proposal will include an element of lottery anyway. It is essential to include assessment from outside the individual's subfield, since the ability to communicate should be a requirement for the vast majority of excellent researchers.

   The assessment should take account of both quality (depth, difficulty, theoretical implications, practical implications), and the number/diversity of admirers of the research (as in citation indexes). It might be partly automated and partly based on judging panels, which should change their membership often enough to spread the workload and reduce long term effects of personal bias.

   (Whether W(N) should be reassessed whenever the individual submits a proposal is debatable. I would say not, except for the first few proposals. Various options are possible, e.g. do the assessment when a new proposal arrives if that individual has not been assessed in the past 4 years or some such thing. Individuals who believe they have produced major new results could request a re-assessment by providing prima-facie evidence.)

4. The result of assessment would produce a (possibly zero) increase to W(N). If N's work is judged very poor W(N) will be reduced. In order to allow someone to jump from a low to a high weight on the basis of outstanding new performance, increases should be additive not multiplicative, subject to an upper limit.

5. To prevent abuse of the lottery system, repeated grant applications by N should reduce W(N) by an amount that increases with the frequency.

6. Collaborative proposals could be dealt with by allowing a group to constitute a new virtual individual. The participating individual collaborators could each determine how much of their current W(N) should be allocated to the group.

7. Group projects will always be assessed when complete, except for teams with a well established research record, whose results could be assessed at intervals. When group is assessed, the resulting (positive or negative) boost is divided between the participants in a manner that they can decide. E.g. it could depend on how much of their individual W(N) each one had contributed, as in most investment schemes. Or it may depend on who did what. In case of disputes some adjudication mechanism may be required, but this should be rare if members of groups all sign some appropriate agreement in advance. To reduce administration a group collaborating for a long time could opt to be treated as a virtual individual with intermittent evaluation.

8. It should be possible for people on temporary research posts to request assessment. This will allow their research record to affect proposals involving them, and may also allow their W(N) to be boosted by previous research record if they later take academic posts.
   (This may be unnecessary if the starting weight in point (1) is high enough. But perhaps the default should not be too high, in which case previous research record can boost the starting weight.)
   [I have not thought enough about how to treat people on research posts. These are just first thoughts.]

The formula used in the lottery will need to be reconsidered from time to time by analysing their impact. The previous comments imply that there will be a number of management constants in the formula, and it may be hard to get them right. In fact it is very likely that at first there will be some mistakes that need to be corrected quickly.

Installed: 2005
Last updated: 13 Dec 2005; 12 Jun 2011