Better dataformat for results and outcomes

[EwoutH]

Currently the results data is dumped into two dataframes:

experiments, outcomes = perform_experiments()

While not necessarily bad, you do lose a bit of information in the process:

• Metadata about the experiment performed (like the ranges of parameters, sampling methods, etc.)
• The difference between model uncertainties and levers (both are just columns in experiments)
• The direct pairing between the experiments and the outcomes

While you can work around everything, it sometimes feels a bit inconvenient. For examples:

• You want to only explore your uncertainties, before exploring your policies, you have to manually select what your uncertainties were
• You can't easily fully reproduce an experiment from an resulting run
• You often have to provide functions with two data objects: Functions like create_pivot_plot, CART and Prim for example require an x and y object.
• "Successfull" outcomes have to be calculated separately

Therefore, I think there should be a more convenient dataformat possible that designs around these limitations.

Moreover, we might find some construct like a "score" or a "success definitions" useful, which can be included in this data structure.

[quaquel]

In my view, the issue is not with the return of perform_experiments, but rather in save_results. If you do perform_experiments, you have the model instance, and so most of the information you describe. This information, however, is lost when you do save_results

You want to only explore your uncertainties, before exploring your policies, you have to manually select what your uncertainties were

Not true, you would do perform_experiments(scenarios=100), and then levers would not be sampled.

You can't easily fully reproduce an experiment from a resulting run

Yes, this is a key issue. First, it inks to spec-7 as well, which should be supported by the workbench. Second, you want metadata regarding the sampling technique that was used.

Moreover, we might find some construct like a "score" or a "success definitions" useful, which can be included in this data structure.

In my personal experience, you generally play around with this during the analysis phase, and you can easily do so with the current dict of numpy arrays for outcomes.

[quaquel]

Curious to see what you come up with. I have been thinking about this on and off for over ten years. I agree that some kind of ExperimentalResults class makes sense, but I never settled on a clean API. Moreover, having just the simple tuple unpacking with the experiments and outcomes has worked well.

The main drawbacks in my view of the current code are:

1. No information on the experimental design used, nor on the seed of the rng. This basically makes the results not reproducible out of the box.
2. No information on the uncertainties and levers, and their data type and ranges.
3. No information on the outcomes and their data type
4. Need to write custom code for the serialization of the results (i.e., save_results).

So any solution should resolve these 4 issues. However, given the proven convenience of experiments, outcomes = perform_experiments, it probably should also support this style of tuple unpacking.

[EwoutH]

Thanks for the additional requirements!

I think I have an even better approach in the works in which we just subclass a Pandas DataFrame, and add some additional functionality to it (just like a GeoDataFrame extends a DataFrame). Then you can do results.uncertainties or results.metrics and only get the columns of that class. While still being able to use the familiar DataFrame interface. The model is also referenced, and for most functions it would require only to input this single Results object.

It can get an experiments_outcomes_tuple attribute for backwards compatiblity.

As for naming hierarchy, I was considering the following:

[quaquel]

I am not sure that subclassing pandas.DataFrame is the best way forward, but curious to see what you can come up with.

The main challenge you will run into is that outcomes can be a mix of 1d, 2d and n-d numpy arrays (and in the future possibly even time series with a dedicated index via a dataframe instead of a numpy array.

By design, a data frame is built around a single array that contains all data, but given the described heterogeneity, you cannot do that here.

Moreover, the memory requirement explodes. A numpy array requires a single block of memory. You can easily require several Gb to store all experiments and their results. If this can be split over multiple blocks of memory, that is easier to allocate.

[EwoutH]

That’s a fair point, hadn’t considered that. Thanks.

[EwoutH]

This might actually be an opportunity to get some aggregation features built in. Outcomes could be stored in raw format, but also be added to an outcomes dataframe:

• single value results get added automatically
• nd arrays can be aggregated to a single value

This way you have one, convenient dataframe for basic usage, and full flexibility to handle the outcomes the way you want if needed.