Best-effort eSWAP analysis

[drphilmarshall]

I talked to @chrislintott about @cpadavis' eSWAP work today - he's very interested in the problem of how to improve the probabilistic analysis, and design task assignments (which classifier should we show this subject to?), in order to optimize a citizen science project. I think this will make a good context for a Human Computation paper on the extensions to SWAP that @cpadavis has been exploring (training on all subjects, offline vs online etc). It will also help focus the program towards the next Space Warps project, where we hope to improve our efficiency by a factor of three by doing the analysis in real time, and perhaps also do some dynamic subject allocation on the basis of the results (building on @anupreeta27 's kick analysis in Paper II, for example). Let's use this thread to discuss this! @aprajita @anupreeta27 and I have tossed around ideas for reducing the false negatives for a while now, and can probably suggest some more good eSWAP experiments to do on the data we have already.

@chrislintott can you say something more about how you think about project optimization, and suggest a very short, focused reading list to give us an idea of where the eSWAP paper will sit in the literature? Thanks!

[drphilmarshall]

@cpadavis and I just discussed the approach to dynamic subject allocation: our plan is to do a best-effort (in the sim/dud ROC curve sense) re-analysis of CFHTLS Stage 1 (which we think will turn out to be offline, using all subjects in the training, and possibly with agents that know about training subject flavors) and then revisiting the Stage 1 False Negatives' (and Positives') trajectories. At this point we should be able to estimate how many dynamic resurrections might be needed to avoid such misclassifications.

[drphilmarshall]

@cpadavis: there may be a possible opportunity to make use of your eSWAP best-effort Stage 1 analysis. We are wondering about re-ingesting a subset of the CFHTLS data to the site for "testing", and figured this may as well be an interesting set - namely, the candidates that we would have got had we run SWAP in unsupervised+supervised and offline mode. If you can run that analysis and produce a catalog with IDs in them, we can ask for those systems to be reingested - perhaps with a new, more difficult set of sims.