Smart update of Collection of CompositionalMetrics

[victorjoos]

🚀 Feature

When updating metrics that are composed of other metrics there are two ways of dealing with updating too many times :

• Always update the underlying metric (the pytorch lightning way)
• Never update the underlying metric (the pytorch ignite way: https://pytorch.org/ignite/metrics.html#ignite.metrics.MetricsLambda)

I don't think there is a clean way of only updating the necessary metrics in the general case (when you're just updating all the metrics yourself), but I think that when you combine your metrics in a collection, it could be useful to only update the "base" metric, instead of all metrics.

Motivation

I often want to use a base metric multiple times, and then I have to be careful not to update too many of them. A somewhat convoluted example (because the f1 score is already implemented) :

prec = Precision()
recall = Recall()
f1 = 2 * (prec * recall) / (prec + recall)
prec.update(pred, gt)
recall.update(pred, gt)
f1.update(pred, gt) # Shouldn't do this, because it updates prec and recall twice. 

Pitch

Continuing last example :

collection = MetricCollection([prec, recall, f1])
collection.update(pred, gt)

This should only update prec and recall once.

Alternatives

The alternative is to always define metrics from scratch, but this causes duplication of computation during the update phase.

[Borda]

@justusschock what do you think about it? ;]

[SkafteNicki]

I had a similar proposal here: #76

[justusschock]

@Borda Yes, this is necessary at some point. The only question here is: Should we define "compatible" metrics, that share their internal state or should this be defined by the user?

IF we define it, we have to maintain this list, but we know about the implementation details of the metrics. If the user defines it, we don't need to care about that list but expect the user to know about our internals.