Spitzer Cycle 1 proposal acceptance order
It's sometimes debated whether your chances of getting a proposal approved, such as a telescope observing proposal, are improved by having a low proposal number, which you would usually get by submitting early. For example, see a Cosmic Variance entry and Steinn's blog.
People speculate about possible mechanisms, including both evaluation biases and selection effects. For example, perhaps the TAC (telescope allocation committee) panel will be fresher earlier and more favorable to early proposals, or will be so tired later that a decent proposal will be a welcome breath of fresh air. Both of these assume the TAC reads the proposals more or less in order by number, which often happens in the initial skimming even if it doesn't happen in the actual TAC meeting. Or, perhaps early submissions are done by people who have their act more together; or perhaps late submissions get a bit more time to be cleaned up (Hah!!)
Enough debate. I attempted to measure this effect, if there is one. It is often discussed with respect to HST proposals, but the data are not trivially available for HST, because HST program ID is not the same as proposal number. However, the data are available for the Spitzer Space Telescope proposal process. Spitzer program IDs correspond to the number assigned on first submission - note that a proposal can then be revised before the deadline and still retain an early number. I compiled the IDs of accepted proposals from the Spitzer Science Center website for Guest Observer info.
The following plots show the proposal ID number (chronological among submitted proposals) on the x-axis, and the order among accepted proposals on the y-axis. That is, if there were proposals numbered 1 to 950 of which 220 were accepted, the x-axis is the proposal number out of the 950, and the y-axis is just 1 to 220. Each plot is for one Spitzer proposal cycle.
This curve is a cumulative distribution of the time-order of proposals that succeeded. So if there is no correlation between the time-order and success, the curve is a straight line, and the slope of the curve is the acceptance fraction. If more early proposals succeed, the curve rises and then flattens; it has a decreasing time-derivative.
The short answer from looking at the plots is that there generally is no order-of-submission effect. In 5 of 6 Spitzer cycles, the curve is essentially straight, so that ratio of acceptance is not dependent on order. In cycle 4, the curve rises early, so that more early proposals were accepted.
In cycles 3, 4, and 5, guaranteed-time proposals from the instrument teams were submitted and ranked along the regular GO proposals. The SSC website lists the GTO proposals for cycles 3 and 4 (not 5 for some reason). This may allow us to separate evaluation bias from other factors, because GTO proposals are ranked along the GOs, but almost all GTO proposals are accepted since they come out of guaranteed time. The plots for cycles 3 and 4 show that GTO proposals follow a similar curve to the GO proposals for that cycle: no dependence in cycle 3, rising early in cycle 4 - very few GTO proposals came in late in the cycle 4 process.
I don't know why cycle 4 is different. Cycle 4 was the last full Spitzer cryogenic cycle (cycle 5 was less than a year) and it's possible that people threw in a lot of last-minute proposals that did not fare well. However, this appears not to have happened in cycle 5.
Here are the plots. And yes, I really am procrastinating doing something else that I should be doing.
Ben Weiner
bjw -at- as.arizona.edu
Spitzer Cycle 1 proposal acceptance order
Spitzer Cycle 2 proposal acceptance order
Spitzer Cycle 3 proposal acceptance order
Spitzer Cycle 4 proposal acceptance order
Spitzer Cycle 5 proposal acceptance order
Spitzer Cycle 6 proposal acceptance order
Benjamin Weiner Steward Observatory University of Arizona email: bjw - at - as.arizona.edu November 25, 2009