Budget Allocator - Date Min/Max not working for some media variables

[Tsurty]

Project Robyn

Describe issue

I've this project with Rolling Windows going from 2021-05-25 to 2022-02-28
During Christmas 2021 there has been a huge expense in influencers which is not commonly there, however by running the Budget Allocator we get something like this

For this reason we've been trying to test out the new feature where you set date_min and date_max in the allocator as follows:

AllocatorCollect <- robyn_allocator(
  InputCollect = InputCollect
  , OutputCollect = OutputCollect
  , select_model = select_model
  , scenario = "max_historical_response"
  , channel_constr_low = c(0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7)
  , channel_constr_up = c(1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5)
  , date_min = c('2022-02-01', '2022-02-01', '2022-02-01', '2022-02-01', '2022-02-01', '2022-02-01', '2022-02-01', '2022-02-01', '2022-02-01')
  , date_max = c('2022-02-28', '2022-02-28', '2022-02-28', '2022-02-28', '2022-02-28', '2022-02-28', '2022-02-28', '2022-02-28', '2022-02-28')
)

The console outputs:

>>> Running budget allocator for model ID 12_1023_8 ...
Excluded in optimiser because their coeffients are 0: fb_pr_impressions, fb_re_impressions, g_discovery_clicks, g_display_clicks, g_performance_max_clicks, g_search_clicks, g_video_impressions
Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)Date Window: 2022-02-01:2022-02-28 (28 days)
Error in if (length(metric_value) != 1 | metric_value <= 0 | !is.numeric(metric_value)) { : 
  missing value where TRUE/FALSE needed

I've tested changing the number of provided boundaries, removing the "date_max" which is simply the end of the dataset but nothing changes.

Environment & Robyn version

> packageVersion("Robyn")
[1] ‘3.6.2’

platform       aarch64-apple-darwin20                     
arch           aarch64                                    
os             darwin20                                   
system         aarch64, darwin20                          
status         Patched                                    
major          4                                          
minor          1.1                                        
year           2021                                       
month          09                                         
day            05                                         
svn rev        80862                                      
language       R                                          
version.string R version 4.1.1 Patched (2021-09-05 r80862)
nickname       Kick Things    

[laresbernardo]

Hi @Tsurty thanks for reporting this. Actually, the length of the input for date_min and date_max should be 1. So simply add ..., date_min = "2022-02-01", date_max = "2022-02-28" and let me know if that does the trick. I will def add a check in there so next users don't have this error but a meaningful warning.

[Tsurty]

Hey @laresbernardo I'm pretty sure I had tried that yesterday already and didn't work.
Anyway I tested it out again now and it's still not working but with a different error

> AllocatorCollect <- robyn_allocator(
+   robyn_object = robyn_object
+   #, select_build = 1 # Use third refresh model
+   , scenario = "max_historical_response"
+   , channel_constr_low = c(0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7)
+   , channel_constr_up = c(1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5)
+   , date_min = '2022-02-01'
+   , date_max = '2022-02-28'
+ )
>>> Running budget allocator for model ID 8_1166_5 ...
Excluded in optimiser because their coeffients are 0: fb_pr_impressions, fb_re_impressions, g_discovery_clicks, g_display_clicks, g_performance_max_clicks, g_search_clicks, g_video_impressions
Date Window: 2022-02-01:2022-02-28 (28 days)
Error: vector memory exhausted (limit reached?)

As you can see I'm trying to build it over the robyn_object only as in the meantime I had other stuff running.
Let me know if I should rebuild both InputCollect and OutputCollect and try again.

[Tsurty]

Ok I tried restarting Session, cleaning variables and repeat the process and here's the same error I used to get yesterday

> AllocatorCollect <- robyn_allocator(
+   robyn_object = robyn_object
+   #, select_build = 1 # Use third refresh model
+   , scenario = "max_historical_response"
+   , channel_constr_low = c(0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7)
+   , channel_constr_up = c(1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5)
+   , date_min = '2022-02-01'
+   , date_max = '2022-02-28'
+ )
>>> Running budget allocator for model ID 8_1166_5 ...
Excluded in optimiser because their coeffients are 0: fb_pr_impressions, fb_re_impressions, g_discovery_clicks, g_display_clicks, g_performance_max_clicks, g_search_clicks, g_video_impressions
Date Window: 2022-02-01:2022-02-28 (28 days)
Error in if (length(metric_value) != 1 | metric_value <= 0 | !is.numeric(metric_value)) { : 
  missing value where TRUE/FALSE needed

[laresbernardo]

Hi Cristian.
Can you please check if those 7 excluded media variables have coefficients = 0? Are there non-0 coefficients variables left?
That error comes from robyn_response(), exactly here. Can you please help be debug so we can 1) fix the issue and 2) create a more robust inputs check here?

[Tsurty]

Hey Bernardo, the thing I'm not completely getting is what's inside the metric_value variable.

Anyway these are my paid medias:

,paid_media_vars = c("fb_pr_impressions", "fb_re_impressions", "g_discovery_clicks", "g_display_clicks", "g_performance_max_clicks", "g_search_clicks", "g_video_impressions", "tiktok_impressions", "influencer_spend")
,paid_media_spends = c("fb_pr_spend", "fb_re_spend", "g_discovery_spend", "g_display_spend", "g_performance_max_spend", "g_search_spend", "g_video_spend", "tiktok_spend", "influencer_spend")

So as you can see there are actually non-zero coefs ones.
Moreover, I don't know if that's important, but it talks about the impressions/clicks instead of the spends.

By looking at the one pager model those media do actually have an impact

If those we really 0 coefs wouldn't the effect be 0% for all of them?

[laresbernardo]

Yes, that's correct. Can you please share your print(ExportedModel) (following demo.R nomenclature) or OutputCollect$xDecompAgg[solID == select_model & !is.na(mean_spend), .(rn, coef,mean_spend, mean_response, roi_mean, total_spend, total_response=xDecompAgg, roi_total, solID)]? That way we can check the coefficients.
Also, when you said "it talks about the impressions/clicks instead of the spends", what do you mean? Where? Remember we changed the modeled variable to be spend, instead of impressions, on version 3.6.0.

[Tsurty]

I actually have another thing running in the backgroung and it'll take a few hours still so I can't really share with you that one.
If you want I can share the RDS file and everything with you in private (email maybe).

Anyway, by looking at the code, since I get the error on line 1031 it means that I get past the 1020 -> 1028 lines.
For this reason I believe that I should be passing the first 2 checks of line 1031 (otherwise I wouldn't have come that far) and the error is specifically in the is.numeric part

[Tsurty]

Nevermind I confused media_metric with metric_value my bad

[laresbernardo]

Could you try that change on a new branch and test it with your own data to see if your problem gets fixed? If it does, you are more than welcome to submit a PR so we can merge your fix with master. If not, we can keep brainstorming possible sources for this particular issue. I'd like to understand why this is happening to you and have not occurred to other users or on demo.R. Thanks!

[Tsurty]

Ok so just to be sure you mean changing the metric_value variable on lines 1030 and 1031 with the variable media_metric right?
If so I'll give it a shot tomorrow probably and keep you posted.

Like I said I'm running another project right now and I'll test the budget allocator on this one too once done, maybe it will work on this one.

[Tsurty]

Hello @laresbernardo I finally found some time to test this out but I didn't manage to fix it

By printing out the values in the metric_values variables like so:

This is what I get in the console:

There's a NaN in between the values that may be causing the issue.

I've tested this with a bunch of different models but none worked and even tried to use the date_min/date_max functionality with the exact same timeframe as the Rolling Windows but that didn't work either.

I'm not really sure what's causing the issue but if you have any feedbacks or tests to try out let me know :)

[laresbernardo]

Hi Cristian. Do you have a minimum reproducible example you could share? Can you try changing in line 1043, metric_value <= 0 to any(metric_value <= 0)? What's the length of metric_value? It should be 1 but I don't understand the NaN you mention if the value is 299.9783. I haven't seen this error with other users yet though, might be something with your data.

[Tsurty]

Hey Bernando, I can share with you the robyn_object and everything to reproduce the issue on your side but I'd have to send that via email. Let me know in case and I'll just send it out.

I tested the "any" fix you tiped above but that didn't work and actually introduced 2 warnings:

As you can see by the console metric_value it seems to have both "128.5851" and "NaN" as values inside

[laresbernardo]

Please, share your MRE with me here.

[Tsurty]

@laresbernardo Sent it, let me know if you received correctly the .RDS file since it was to big to load I linked it through the Drive but I had no visual feedbacks whether it was actually selected or not

[laresbernardo]

Note: For the sake of timings, I reduced to 2 trials, which took about 30mins to run.

Note: When running robyn_run() I got the following warning: "This trial contains 4 iterations with all 0 media coefficient. Please reconsider your media variable choice if the pareto choices are unreasonable." Recommendations are:

1. increase hyperparameter ranges for 0-coef channels to give Robyn more freedom
2. split media into sub-channels, and/or aggregate similar channels, and/or introduce other media
3. increase trials to get more samples

Selected model "1_304_13", which had 2 coef == 0 channels. When running the "max_historical_response" scenario, it ran OK, but effectively there was an error in the message displayed. When running the "max_response_expected_spend" scenario, it also ran OK (once I fixed the dimensions on channel_constr_low and channel_constr_up from your script).

When running the "QA optimal response" section, I got his error, which makes sense: " 'media_metric' must be one value from paid_media_spends, paid_media_vars or organic_vars". I noticed that the issue comes from paid_media_vars and organic_vars (OK for paid_media_spends) because none of them have a numerical and positive optimal_spend value. When you run the code in that section, notice the value for optimal_spend is numeric(0). I've updated the demo.R file to be clearer in this point and created new informative errors for when inputs are not acceptable with correct suggestions.

Thanks for the feedback!

[Tsurty]

Alright so if I'm understanding correctly the issue is that at least one of organic_vars or paid_media_vars must be positive, even if paid_media_spends are actually positive.

I'm traveling today so I won't have time to test this out but I will tomorrow. Thanks a lot in the meantime!

[laresbernardo]

By definition, we can't have non-positive values for these curves. And the error occurs when you pass negative or not valid numerical values for your metric_value input. Check this section of the demo.R to test one by one your media variables.

[Tsurty]

Hello @laresbernardo I've built another model, completely different data etc.
The model looks good both business-wise and statistics-wise (Rsq 87% - NRMSE 0.05) and there's no media at all with 0% Share of Effect

Anyway if I try to run the Budget Allocators, even without any warning about 0 coefs, the date min and max won't work

> AllocatorCollect <- robyn_allocator(
+   InputCollect = InputCollect
+   , OutputCollect = OutputCollect
+   , select_model = select_model
+   , scenario = "max_historical_response"
+   , channel_constr_low = c(0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7)
+   , channel_constr_up = c(1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5)
+   , date_min = '2022-03-01'
+   , date_max = '2022-04-25'
+ )
>>> Running budget allocator for model ID 5_1196_4 ...
Date Window: 2022-03-01:2022-04-25 (56 days)
Error in if (length(metric_value) != 1 | metric_value <= 0 | !is.numeric(metric_value)) { : 
  missing value where TRUE/FALSE needed
> 

If I run the allocator without those parameters it works.
However I tried to run the updated QA

> ## QA optimal response
> if (TRUE) {
+   optimal_response_allocator <- AllocatorCollect$dt_optimOut[channels == select_media, optmResponseUnit]
+   optimal_response <- robyn_response(
+     robyn_object = robyn_object,
+     select_build = 0,
+     media_metric = select_media,
+     metric_value = metric_value)
+   plot(optimal_response$plot)
+   if (length(optimal_response_allocator) > 0) {
+     cat("QA if results from robyn_allocator and robyn_response agree: ")
+     cat(round(optimal_response_allocator) == round(optimal_response$response), "( ")
+     cat(optimal_response$response, "==", optimal_response_allocator, ")\n")
+   }
+ }
Error in robyn_response(robyn_object = robyn_object, select_build = 0,  : 
  object 'metric_value' not found

An as you can see it returns me an error

[Tsurty]

Hello @laresbernardo I've built another model, completely different data etc. The model looks good both business-wise and statistics-wise (Rsq 87% - NRMSE 0.05) and there's no media at all with 0% Share of Effect

Anyway if I try to run the Budget Allocators, even without any warning about 0 coefs, the date min and max won't work

> AllocatorCollect <- robyn_allocator(
+   InputCollect = InputCollect
+   , OutputCollect = OutputCollect
+   , select_model = select_model
+   , scenario = "max_historical_response"
+   , channel_constr_low = c(0.7, 0.7, 0.7, 0.7, 0.7, 0.7, 0.7)
+   , channel_constr_up = c(1.5, 1.5, 1.5, 1.5, 1.5, 1.5, 1.5)
+   , date_min = '2022-03-01'
+   , date_max = '2022-04-25'
+ )
>>> Running budget allocator for model ID 5_1196_4 ...
Date Window: 2022-03-01:2022-04-25 (56 days)
Error in if (length(metric_value) != 1 | metric_value <= 0 | !is.numeric(metric_value)) { : 
  missing value where TRUE/FALSE needed
> 

If I run the allocator without those parameters it works. However I tried to run the updated QA

> ## QA optimal response
> if (TRUE) {
+   optimal_response_allocator <- AllocatorCollect$dt_optimOut[channels == select_media, optmResponseUnit]
+   optimal_response <- robyn_response(
+     robyn_object = robyn_object,
+     select_build = 0,
+     media_metric = select_media,
+     metric_value = metric_value)
+   plot(optimal_response$plot)
+   if (length(optimal_response_allocator) > 0) {
+     cat("QA if results from robyn_allocator and robyn_response agree: ")
+     cat(round(optimal_response_allocator) == round(optimal_response$response), "( ")
+     cat(optimal_response$response, "==", optimal_response_allocator, ")\n")
+   }
+ }
Error in robyn_response(robyn_object = robyn_object, select_build = 0,  : 
  object 'metric_value' not found

An as you can see it returns me an error

Don't mind about the last error, it was my fault not defining the metric_value variable

I've fixed that, tested again and all medias return TRUE with matching values like:
QA if results from robyn_allocator and robyn_response agree: TRUE ( 2.759225 == 2.759225 )

I'm not fully getting how this QA section helps, so any clarifications on this would be great.
Besides from this, any idea what may be causing the issue at this point?

[laresbernardo]

This section of the demo.R file gives you an example of how you can calculate and plot the response for a given media variable (paid or not paid). In media_metric you pass the selected media channel and metric_value will be the spend value for paid media variables and the impressions or whatever response for non-paid media variables. We only check for that QA result when optimal_response_allocator > 0 because we automatically use AllocatorCollect to pick the optimal values here.
Note: your error comes from not defining your metric values here before running that section.

[Tsurty]

I'm probably missing out something here but I don't get why I keep getting this error even now that metric_value is defined and positive and all coefficients are positive and non-zero

I'm up to any additional tests/validations you'd like me to do

[laresbernardo]

Hi @Tsurty can you please make sure you're testing the latest version? Can you try narrowing it down to a minimum reproducible example? The metric_value for which metric is negative? That must be happening with a not paid media variable (I guess it'd be useful to have that into that error) because the paid media variables' metric values are the ones you shared in the screenshot.

[laresbernardo]

I've just updated the (internal) check_metric_value() function and it will print which media variable is causing this error.

[Tsurty]

I haven't been able to update the library, I'm getting the following error:

> remotes::install_github("facebookexperimental/Robyn/R")
Downloading GitHub repo facebookexperimental/Robyn@HEAD
Error in utils::download.file(url, path, method = method, quiet = quiet,  : 
  download from 'https://api.github.com/repos/facebookexperimental/Robyn/tarball/HEAD' failed

Anyway I'll look into it later on.

In the meantime I checked the coefs via the all_aggregated.csv file and there's a 0 coefficient organic variable which is "o_google"

I'm not sure why this would break the Allocator, as the date_min/max are supposed to work on the spend variables (I believe) but probably that's the issue.

[laresbernardo]

Please try setting this first options(timeout=400) as mentioned here.

[Tsurty]

Thanks @laresbernardo this did the trick.
I updated and restarted the session, still getting the error ofc but not as expected

>>> Running budget allocator for model ID 8_956_5 ...
Date Window: 2022-04-01:2022-04-15 (15 days)
Error in if (metric_value <= 0) stop(sprintf("Input 'metric_value' for %s must be a positive value",  : 
  missing value where TRUE/FALSE needed

Is it correct that the checks here are done over metric_value but then you print media_metric?

[laresbernardo]

Is it correct that the checks here are done over metric_value but then you print media_metric?

Yes.

Please, try changing that check to the following to show both values and make it easier to debug:

check_metric_value <- function(metric_value, media_metric) {
  if (!is.null(metric_value)) {
    if (length(metric_value) != 1) stop(sprintf(
      "Input 'metric_value' for %s (%s) must be a valid numerical value", media_metric, metric_value))
    if (!is.numeric(metric_value)) stop(sprintf(
      "Input 'metric_value' for %s (%s) must be a numerical value", media_metric, metric_value))
    if (metric_value <= 0) stop(sprintf(
      "Input 'metric_value' for %s (%s) must be a positive value", media_metric, metric_value))
  }
}

[Tsurty]

@laresbernardo The problem was that one of my media channel happens to have all 0 spends in the timeframe used within date min/max, because we chose to turn that off completely.

The check you suggested above didn't work because, when the spends are all 0, metric_value doesn't contain 0 but NaN as you can see down here

In order to print them all I simply commented the last if statement and added a cat at the beginning like so

check_metric_value <- function(metric_value, media_metric) {
  if (!is.null(metric_value)) {
    cat(metric_value)
    cat(" - ")
    if (length(metric_value) != 1) stop(sprintf(
      "Input 'metric_value' for %s must be a valid numerical value", media_metric, metric_value))
    if (!is.numeric(metric_value)) stop(sprintf(
      "Input 'metric_value' for %s must be a numerical value", media_metric, metric_value))
    #if (metric_value <= 0) stop(sprintf(
      #"Input 'metric_value' for %s must be a positive value", media_metric, metric_value))
  }
}

I then tried to change my date_min/max to something where all medias had at least some spends and that finally work.

I'm glad we managed to understand the issue, however I'd still consider this kind of a bug.
It can happen that in the past we used to advertise on a channel we are not using anymore and it would be really useful to be able to use these new params in the Budget Allocator even for managing situations like this.

What do you think?

[laresbernardo]

Yes, that's it! I think you've finally found the bug here. So I'll add a new is.nan() check and let the user know that all metric values are zero for that specific media channel and will be removed/ignored.

[laresbernardo]

@Tsurty I was able to replicate your issue and test my fix. Now we 1. warn the user when there's no spend in a paid media variable (and skip it) and 2. added as skipped media variable in the print method with the coef = 0 media variables.

Can you please update the package and test it out once again? Thanks for catching this case/bug.

[Tsurty]

@laresbernardo Yes! It's working correctly now. I'll test this feature more "in depth" in the upcoming days and I'll report if any new issue comes up.

Thanks a lot for the help :)

[JessicaMDuncan]

I'm having this same issue:
Input 'metric_value' for google_display_s (0) must be a positive value"
In the new date window, spend is 0 for all days for this channel.

Running version 3.7.1 (not sure how since latest documented release appears to be 3.7.0...), and I'm seeing very recent updates to refresh.R(2 hours), checks.R (yesterday), and allocator.R (yesterday) in facebookexperimental/Robyn/R. We're going to pull v3.7.0 and see if that functions with the warning and skipped media variable as mentioned in the fix above by @laresbernardo on May 3. If that doesn't work, I guess we'll try v3.6.3 (May 6 release); will update if either of those solve the issue.

[laresbernardo]

Hi @JessicaMDuncan The demo.R file has the 3.7.1 and it's currently the latest version. As for your issue, I think that's not a bug but rather the result of check_metric_value() which detects your google_display_s channel has 0 spend within the date window (as you mentioned). Try removing it by setting the upper and lower limits to 0 (assuming it's happening in the robyn_allocator() step. Is it?)

[JessicaMDuncan]

@laresbernardo that was the error produced at the end of robyn_refresh; it ran through all trials and then produced that error. What step of the process is this warning supposed to occur?

"Now we 1. warn the user when there's no spend in a paid media variable (and skip it) and 2. added as skipped media variable in the print method with the coef = 0 media variables."

Edited to add screenshot of error.

[laresbernardo]

To help me debug this case, can you please share with me your dataset and model's JSON file to see if I can replicate this error? Thanks!

[JessicaMDuncan]

@laresbernardo I am awaiting confirmation from my leadership team that they will authorize sharing of the data. In the meantime, using placeholder (non-zero spend) data for this channel, I'm now receiving the error discussed in this thread which appears to still be active/ awaiting an update.

[laresbernardo]

Closing given the inactivity. Feel free to open in case this issue persists.