How to select research proposals less wastefully
Use a sensibly designed, relatively inexpensive,
School of Computer Science, The University of Birmingham
Discussions on the mailing lists of
UK Computing Research Committee
Council of Professors and Heads of Computing
CPHC email lists about research funding, including mention of the high
proportion of research time and funding budgets that goes into writing and
assessing proposals, recently provoked me into reviving an idea that always
seems to produce shock and horror, even though I think that (with suitable
refinements) it could be an excellent way to fund research, namely, using a
properly designed lottery.
There are two key premises behind this proposal:
When doing real research you often don't know what you are going
to do when, because what you do as a researcher exploring unknown
territory depends on what you have previously learnt, which keeps
changing. So assessing proposals on the basis of what people say they
are going to do, forces people to artificially circumscribe research,
producing detailed charts and time-tables which make their proposals
look professional to people who don't understand what deep science is,
and I suspect also reduces depth, originality, and long term importance
of much of the research.
Of course the real function of all that paraphernalia is to enable
politicians, funding agencies, and their officers, to proudly claim that
they are managing research funding to ensure that tax payers'
money is not wasted. And thereby much money gets wasted and some really
good research doesn't get funded because it is too speculative to get
through the selection process. And researchers learn to think in
well-defined grooves because that's what gets them grants, and tenure
and kudos in their institutions.
In addition to supporting such research on industrial and development projects
we should also encourage high calibre scientists to investigate hard and deep
problems in more open-ended, less predictable research projects. Often that
involves tasks like:
This objection to advance planning of research does not apply to
industrial or other development
projects, or scientific projects
that start from a specific practical target to be achieved, which
requires working backwards through intermediate stages in order to
decide what to do first. That includes 'big science' projects that
require building expensive equipment, going on long journeys, e.g.
oceanography research, space exploration, building large telescopes,
etc. I am not concerned with such research here, Such proposals
inevitably require a great deal of prior scrutiny before public
resources are allocated to them. The need for detailed planning is also
required in research that requires collection and processing of a lot
of measurements to fill specific gaps in some region of scientific
knowledge, e.g. large scale surveys, long term biological or
astrophysical data-collection research, etc.
Researchers working on such problems may be able to say something fairly
detailed about how they are going to start the research, but relatively
little about will happen after the early stages, since everything after
that is conditional on the early results.
Trying to find a proof for some mathematical theorem
Trying to design a formalism, algorithm or architecture that has certain
applications or explanatory capabilities
Trying to come up with a deep explanatory mechanism unifying a range of
previously known phenomena
Trying to identify relationships between different disciplines or
research areas that are relevant to unsolved problems in one or both
Trying to develop a new conceptual framework to make sense of a
collection of puzzling facts or unresolved conflicts between theories,
Surveying existing research publications in order to search for clues
to some hard problem
Developing a new technique for acquiring information that has so far
been hard to acquire
Designing a new way of processing old information that can yield
important insights previously not accessible or that produces fewer
misleading results (e.g. new kinds of data-mining, or new statistical
Designing a new model that can simulate or explain some existing
observed behaviours of humans or other animals: typically one does not
know in advance what such a model will turn out to be, as the process of
designing and testing usually reveals previously unnoticed requirements
This contrasts with most of the research proposals I have read which
state that the project will do X, then Y, then Z, etc. giving
approximate times for all objectives to be achieved.
Very few of those are
research proposals. They are merely attempts to get money by
conforming to prescribed application formats.
There is too much effort involved both in getting the resources to do
research and in evaluating proposals. Both kinds of effort could be
reduced by a carefully designed weighted lottery system (given that the
current system is often said to be largely a lottery anyway).
In short: (a) spending large resources on assessing research proposals
tends to force research into areas where fairly detailed advance
planning and prediction of results is possible, so that there is
something to assess in advance, leading to too much emphasis on shallow
and relatively trivial (and therefore predictable) research, and (b) the
processes of generating such proposals and assessing them consume a lot
of resources that would be better spent on doing research.
Most of my own UK research grant proposals have been turned down, though
a few were successful. However, a few funders have approached me unasked and
offered to fund my research, without asking for anything explicit in return
(except reports in one case).
the EC cognitive systems initiative, which was quite visionary,
was willing to fund my research, though the process was over-managed.
The problems discussed above can be overcome by using a weighted lottery
(whilst leaving the assessment of very large projects unchanged).
The proposal has several interlocking aspects:
Universities (and other organisations) that wish to do research should
take great care when hiring people -- much more than is normally done in
universities in the UK, compared with commercial/industrial research
centres, and compared with universities in the USA (at least the ones I
know about). Moreover the selection should be based on research
potential, originality of ideas, the ability to give exciting
presentations, etc. rather than on numbers of publications in journals
and previous grant income (factors expected to influence RAE
results as opposed to the growth of knowledge).
If most of the people doing research have already been pre-selected as
outstanding researchers, there is less need to assess their specific
research in advance of their doing it. So we need a mechanism that gives
them high research credibility initially. EPSRC's
First Grant scheme
does that though though it can only benefit a very small minority of new
researchers and has the previously mentioned flaws of wasting resources
on attempting to assess work that has not been done yet and distorting
proposals to fit the grant assessment procedures. It also has a vicious
24 month cut off period (after appointment), which can be grossly
unfair, since different amounts of time may be required to identify and
clarify different research problems.
Individuals appointed as research active academic staff should
automatically have the right to take part in the lottery-based research
funding process described below, though a university or department might
remove or postpone that right in cases of doubt (a decision not to be
taken lightly, and which should be subject to appeal). In most cases if
universities were properly funded, internal funding could be used to
give new researchers a chance to prove themselves.
Each individual's lottery bids will be weighted by a factor that starts
high, then decays with time (using a backward sigmoid curve), but can be
boosted, as described below. This is unlike the 'First Grant' scheme
which has a sharp temporal cutoff, as mentioned above.
More of the research funding that is now based on competitions run by
research councils should instead be allocated directly to research
active departments for internal allocation, especially for use by young
researchers. Internal allocations should be based on judgements about
individual promise, past achievements, and evidence of ability to
identify hard problems and investigate them in depth. Some of this
funding can take the form of reduced teaching loads or sabbaticals, e.g.
for theoretical scientists, for whom there seems to be grossly
inadequate provision in the research council funding schemes.
There should be sufficient direct funding to universities to allow them
to give very good research support for all young researchers for
the first few years (partly in the form of reduced teaching and admin
loads, but also travel, equipment, and in some cases RAs). Mentoring
schemes can help nurture young researchers, and also detect whether
things are going wrong so that projects should be terminated or
redirected -- which research councils don't do, fortunately. (The EC
attempts this, at great administrative cost to all concerned.)
If this reduces the amount allocated by research councils that's not
necessarily a bad thing if it transfers resources from administration to
research, as well as ensuring better help and guidance for young
researchers during their first projects. Of course, university managers
would have to be prevented from reallocating that funding for other
purposes, which many will try to do. Incidentally I am sure that the
most important effect of the 'Full economic costing' scheme over time
will be a bad one: namely transferring funding from direct grants to
universities to competitively allocated budgets managed by research
councils. That is not the intention of the supporters of the scheme, but
watch what happens in the next five years. I suspect that very soon it
will lead to unfair discrimination within underfunded universities
between researchers who get research council funding with FEC and those
who don't. (That may look like good business acumen because it motivates
people to strive for grants, but is actually bad management because it
wastes effort that should be spent on research and is potentially
divisive, undermining collaborative and mutually supportive research
All this assumes that not all tertiary education departments should get research
funding. We need to revert to a system with a far better division of
labour, with a significant subset of higher education done in
institutions that are dedicated to high quality education and training
such as many polytechnics used to provide, and which both the nation and
many school leavers need.
Those institutions and the people in them
should be well rewarded for what they do, and not punished at all for
not doing research. This issue is discussed further in
a separate document
pointing out that views on both sides of the debate on top-up fees arise from a
failure to plan a well integrated broad spectrum higher education system.
Research oriented universities (or departments) should not be punished
(as the RAE tends to punish them) for having significant subsets (e.g.
as many as 25% or even 30%) of their staff who are not research active
but are dedicated to keeping up with research results relevant to their
teaching, producing high quality teaching, helping with administration
and management and helping research active staff understand requirements
for good teaching. All permanent academic staff who do research should
also do some teaching since communication of knowledge is as important
as discovering it. But some division of labour can be more fruitful than
expecting everyone do equal shares of everything.
The lottery [still a rough draft]
A few times each year (e.g. at most four times) a limited amount of
funding for eligible proposals will be allocated on the basis of a
lottery. The total amount available to each lottery will depend on how
much overall funding is available and the number of lotteries per year.
It could also depend on the number of proposals (or their total value).
The precise algorithm to be used remains to be considered: this document
is merely about general principles.
Applicants will be people on academic posts who are now eligible to be
PIs on EPSRC proposals. Funding from research councils to such
individuals who apply for grants will in most cases be allocated in the
lottery on the basis of a probabilistic formula which combines elements
described below. Very large grant proposals will be dealt with
differently, as explained above, using a pre-allocated budget, as will
some development proposals aiming at achieving specific practical goals
identified by the Research Council. Projects above some size limit will
have to be vetted before being accepted for the lottery. Call the
proposals eligible to be funded by the lottery 'eligible' proposals.
Every recognised individual researcher N will have at any
time a weight W(N) (e.g. between 0 and 100) that will be used in the
lottery if N submits a proposal. W(N) will change over time, as
Some individuals may temporarily contribute some or all of their weight
to a virtual individual that is a group of individuals who wish to
collaborate and submit a joint proposal to the lottery.
For each individual N, an initial weight W(N) will be assigned which
starts high at the beginning of N's first appointment to an academic
post and drops automatically over time.
The rate of fall should be very slow (or possibly the curve
should be flat) until the first proposal is submitted by N. After that
the weight W(N) should fall more rapidly, though that can be countered
by weight-boosting mechanisms described below.
Every time a proposal is submitted by N, whether it is proposal fed to
the lottery scheme or another mode of selection, W(N) is reduced
thereafter temporarily, either for a year or if the project is funded
for half (or some other fraction) of the project's length. The amount of
reduction of W(N) could depend on the size of the grant awarded. This
would also apply to new researchers. So some will be deterred from
rushing to submit immature proposals, which is what happens if there are
sharp temporal deadlines.
If N gets one or more grants, then at intervals the research results
produced by N are assessed by some means. The outcome is used to boost
or lower W(N). We could have a long debate about details, such as use of
citation measures and other things, but the debate can be shortened if
we accept that there is no perfect solution and every proposal will
include an element of lottery anyway. It is essential to include
assessment from outside the individual's subfield, since the ability to
communicate should be a requirement for the vast majority of excellent
The assessment should take account of both quality (depth,
difficulty, theoretical implications, practical implications), and
the number/diversity of admirers of the research (as in citation
indexes). It might be partly automated and partly based on judging
panels, which should change their membership often enough to spread
the workload and reduce long term effects of personal bias.
(Whether W(N) should be reassessed whenever the individual submits a
proposal is debatable. I would say not, except for the first few
proposals. Various options are possible, e.g. do the assessment when
a new proposal arrives if that individual has not been assessed in
the past 4 years or some such thing. Individuals who believe they
have produced major new results could request a re-assessment by
providing prima-facie evidence.)
The result of assessment would produce a (possibly zero) increase to
W(N). If N's work is judged very poor W(N) will be reduced. In order to
allow someone to jump from a low to a high weight on the basis of
outstanding new performance, increases should be additive not
multiplicative, subject to an upper limit.
To prevent abuse of the lottery system, repeated grant applications by N
should reduce W(N) by an amount that increases with the frequency.
Collaborative proposals could be dealt with by allowing a group
to constitute a new virtual individual. The participating individual
collaborators could each determine how much of their current W(N)
should be allocated to the group.
Group projects will always be assessed when complete, except for
teams with a well established research record, whose results could be
assessed at intervals. When group is assessed, the resulting (positive
or negative) boost is divided between the participants in a manner that
they can decide. E.g. it could depend on how much of their individual
W(N) each one had contributed, as in most investment schemes. Or it may
depend on who did what. In case of disputes some adjudication mechanism
may be required, but this should be rare if members of groups all sign
some appropriate agreement in advance. To reduce administration a group
collaborating for a long time could opt to be treated as a virtual
individual with intermittent evaluation.
It should be possible for people on temporary research posts to
request assessment. This will allow their research record to affect
proposals involving them, and may also allow their W(N) to be
boosted by previous research record if they later take academic
(This may be unnecessary if the starting weight in point (1)
is high enough. But perhaps the default should not be too high, in
which case previous research record can boost the starting weight.)
[I have not thought enough about how to treat people on research
posts. These are just first thoughts.]
The formula used in the lottery will need to be reconsidered from time
to time by analysing their impact. The previous comments imply that
there will be a number of management constants in the formula, and it
may be hard to get them right. In fact it is very likely that at first
there will be some mistakes that need to be corrected quickly.
Added 16 May 2014:
I have found this related discussion.
Monday, March 18, 2013
How to Give $1 Million a Year to Philosophers
I hate grants.
By Eric Schwitzgebel
Last updated: 13 Dec 2005; 12 Jun 2011; 16 May 2014; 8 Feb 2016(re-formatted)