Adding ab-testing vignette

[imadmali]

I'd like to contribute a vignette that outlines how to do A/B testing inference (post-experiment) with rstanarm functions. As discussed here I think it would be useful if it was part of the "Additional tutorials on specific modeling techniques and applications" section on the vignette page of the rstanarm site. @jgabry I've left your comments in and have changed the content based on some of them. Also, regarding Appendix B, I don't fully understand how priors are specified on group specific intercepts when using stan_glmer so I just said something to the effect of "default priors are used".

[jgabry]

Just a few minor comments (I see you already addressed previous comments, thanks!). For appendix B, are you ok leaving it just saying we're using default priors or do you want to add the details about the specification? It's a bit more complicated than the other priors.

[jgabry]

Is it possible to do this without dplyr? I don't think we currently have dplyr as a dependency but we would need to add it to use it in the vignette.

[imadmali]

So I use dplyr here. I don't have to use it and it's straightforward to rewrite this in non-dplyr code. But since it's pretty much a standard for industry workflow maybe it's something to consider using in all the vignettes. But then, like you mentioned, it's not a dependency for the main purpose of the package. Let me know what you want me to do.

[imadmali]

@jonah regarding the hierarchical section I'm fine with saying default priors if there is a place where the information about the priors is clearly defined? If there is such a place then I'll provide a link to it. Let me know.

[imadmali]

@jgabry I made some more edits. Let me know if it's okay to merge or if more changes need to be made.

[bgoodri]

We can't use the params mechanism any more because some OS does not have a sufficiently recent Pandoc or something. The way we are doing the other vignettes now is to put knitr::opts_chunk$set(eval = identical(Sys.getenv("NOT_CRAN"), "true")))

[jgabry]

I got around this in other packages just by checking if (exists("params")) but not using it is also fine.

[imadmali]

Ok, I'll just remove it then and add that line of code.

[ssp3nc3r]

Nice vignette. Would it make sense to call set.seed() for repeated calls to posterior_predict when comparing posteriors, or just run the function once and subtract the groups? For example,

pp_c <- posterior_predict(fit_group_bin)[,1]
pp_d <- posterior_predict(fit_group_bin)[,2]

would be

pp <- posterior_predict(fit_group_bin)
pp_c <- pp[,1]
pp_d <- pp[,2]

[jgabry]

@ssp3nc3r Yeah good suggestion.

[imadmali]

I've made all suggested edits. Let me know if I'm missing anything.

[imadmali]

@jgabry @bgoodri is there anything left that I need to do to get this accepted?

[jgabry]

@imadmali Sorry for the delay. I think this is ready to go, just one quick change: can you remove the uses of autoscale=FALSE? With the latest release that's no longer necessary when specifying non-default priors. It's not wrong, it's just not necessary anymore so might be confusing to keep it.

[jgabry]

I just went through and removed autoscale=FALSE since it's not needed anymore with the latest release. AFAIK that was the last remaining thing to change so will merge this (can revert if necessary).