Implement Active Learning

[kyledmiller]

Addressed by PR #431

What

Pool-based sampling to train in increments, selectively feeding the network only the worst-predicted examples from the training set.

Workflow

1. Train on N0 training examples.
2. While there are unseen examples left and validation loss is above some threshold:
   • Predict on unseen training examples.
   • Train on N unseen training examples with largest loss

Why

1. LDOS data is plentiful.
2. LDOS data contains lots of redundancy.
3. Prediction is relatively cheap.

Pool-based sampling would help our network focus training on atomic interactions which are rare and/or difficult to capture while avoiding training on as much redundant data. Since prediction is relatively cheap for neural networks, the additional time complexity from selecting the worst-performing examples should be more than recouped by the reduction in redundant training.

Anticipated Issues or Questions

1. How can we efficiently reload each new training/testing data batch?
2. Do we need to train on the cumulative seen dataset each time (to avoid forgetting old knowledge) or just the newest selection of badly-predicted training examples (to maximize speed)? Or maybe something in between?