Release 2.26 checklist

[rok-cesnovar]

With the last release, there were a couple straggling issues that took a long time after the formal release to resolve. In particular the release notes and the docs for lupdf and lupmf came a couple/few weeks after the formal release. Since this release will have a lot of new features, let's plan more explicitly what needs done when so things are not forgotten.

So we've started a checklist here, and part of this checklist will be more checklists. But the idea is that the checkbox is due on the date it is categorized under.

~6 weeks before the next release (halfway between releases -- this is 14th December):

• Create an issue to plan for the release ~6 weeks before the release
• List all known open projects in Math/Stan/Cmdstan/Stanc3 and gather info on what is expected to be merged for release.
  • Expressions
  • OpenCL
  • varmat postponed to 2.27
  • timing
• Create checklists for the open projects -- what needs to be done and how it will be verified that it is done
  • Expressions (expressions done, see here and here)
  • OpenCL 2.26 release OpenCL support meta-issue #959
  • varmat 2.26 release varmat meta-issue #958
  • timing 2.26 timing/profiling meta issue #960
• Make a docs issue to track docs tasks for the next release: Docs for Stan 2.26 release - issue to track progress docs#298

Feature freeze (January 18th -- this is the regular 1 week freeze):

Release candidate stage:

• Create a release candidate tarball. Make sure RC has the new CMDSTAN_VERSION and that RC tarballs have binaries.
• Create Math/Stan RC releases (https://github.com/stan-dev/math/releases/tag/v3.5.0-rc1 and https://github.com/stan-dev/stan/releases/tag/v2.26.0-rc1)
• Create a release candidate feature/bugfix list (this doesn't need to be polished, just so that RC testers know what to test)
• Run CmdstanR tests with the RC tarball
• Make a Discourse RC post
• Make a Twitter RC announcement

Release (January 25th):

• Create the Math Release notes (see Release 2.26 checklist #957 (comment))
• Create the Stan Release notes (see Release 2.26 checklist #957 (comment))
• Create the Cmdstan Release notes (see Release 2.26 checklist #957 (comment))
• Create the Stanc3 Release notes (see Release 2.26 checklist #957 (comment))
• Create the Math release
• Create the Stan release
• Create the Stanc3 release
• Create the CmdStan binaries (make sure the version is incremented and tarball includes binaries)
• Run CmdStanR tests with the release tarball
• Make a Discourse release announcement post
• Cross-post the Discourse announcement on the Stan blog
• Make the Twitter announcement

Ideas for next release:

• thank everyone that funds Stan development and our Github sponsors?

[rok-cesnovar]

This is a checklist attempt for the next release so we get a bit more organized. The first draft was done by me and @bbbales2, feel free to comment/edit/add/remove.

Tagging @stan-dev/math @stan-dev/maintainers

[avehtari]

Looks good and makes it more clear how much work making a release needs!

[bbbales2]

The list of Math pulls that had gone in as of Dec 7th are here. I'm not sure the value of maintaining this list but I'll put this one here since @rok-cesnovar and I discussed it:

General changes:

• Compiles on M1 Macs.(#2208)
• Refactor Jenkins pipeline (#2100)
• Upgrade Sundials to 5.5.0(#2242)
• Added custom_vari class.(#2144)
• Add symmetrize(matrix m) function to stan::math.(#2209)
• Speedup reduce_sum parallelism.(#2162)
• Make gradients for negative binomial and 2F1 function more robust for boundary values.(#2175)
• Fixed problem with integrate_1d tolerances(#2183)
• Modified functions requiring linear indexing of their arguments to remove that requirement.(#2205)
• Updated log-likelihood calculation for ordered_probit_lpmf for increased numerical stability(#2229)

Varmat:

• Add's cholesky decomposition for var types.(#2105)
• Added var_value<Eigen::Matrix<T, R, C>> specializations for columns_dot_self, determinant, dot_self, inverse, log_determinant, matrix_power, multiply_lower_tri_self_transpose, and tcrossprod.(#2106)
• Adds addition overloads for var matrices and matrices of vars(#2115)
• Added transpose for var<mat>(#2163)
• var<mat> versions of quad_form and trace_quad_form(#2164)
• variance modified to work with var<mat>(#2168)
• Updated sd, log_softmax, and log_sum_exp to work fully with var<mat>(#2169)
• Added var<mat> support to divide(#2171)
• Bug fix for assignment to blocks of a var<matrix>(#2172)
• Add more methods for taking blocks of var matrices.(#2188)
• Adds operator- for var<matrix> types(#2196)
• Added var<mat> versions of columns_dot_product and rows_dot_product(#2199)

OpenCL:

• Added OpenCL reverse mode implementation of Cholesky decomposition that works on var_value types. Changed from_matrix_cl to correctly works with triangular matrices.(#2117)
• Implemented bernoulli_lpmf using OpenCL, enabling it to use GPUs. Fixed a bug that caused a generated kernel not to compile if it used both check_cl and any colwise reduction.(#2155)
• Implemented bernoulli_logit_lpmf using OpenCL, enabling it to use GPUs.(#2173)
• Implemented beta_lpdf using OpenCL, enabling it to use GPUs.(#2174)
• Added 4 new OpenCL device functions: lgamma_stirling, lgamma_stirling_diff, lbeta and binomial_coefficient_log.(#2176)
• Added an OpenCL implementation of beta_proportion_lpdf function.(#2180)
• Simplified writing of OpenCL broadcasting tests.(#2177)
• Added OpenCL implementations of functions: sign, cauchy_lpdf, chi_square_lpdf and double_exponential_lpdf.(#2181)
• Added OpenCL implementations of functions exp_mod_normal_lpdf, exponential_lpdf and frechet_lpdf and support for pow function in kernel generator.(#2184)
• Added OpenCL implementations for functions gamma_lpdf and gumbel_lpdf.(#2185)
• Fixed a bug in check_cl, which in case of throwing an exception, caused it to report wrong indices of error.(#2185)
• Added OpenCL implementations for functions inv_chi_square_lpdf, inv_gamma_lpdf and logistic_lpdf.(#2191)
• Added support for device functions with multiple includes to kernel generator. Added functions lbeta and binomial_coefficient_log.(#2201)
• Fixed a bug that would crash a program if two kernels using same operations are run on different set of unique matrices (for example c = a + b and ˙c = a + a).(#2215)
• Added OpencL implementations for distributions lognormal_lpdf, normal_lpdf and pareto_lpdf.(#2216)
• Added OpenCL implementations for functions pareto_type_2_lpdf, poisson_log_plmf and poisson_log_lpmf.(#2217)
• Also added support for function multiply_log to kernel generator.(#2217)
• Added OpenCL implementations for functions rayleigh_lpdf, scaled_inv_chi_square and skew_normal_lpdf.(#2221)
• Added OpenCL implementations for functions std_normal_lpdf, student_t_lpdf and uniform_lpdf.(#2222)
• Added to_matrix_cl(std::vector<var_value>) and reorganized main Stan Math OpenCL headers.(#2226)
• Added OpenCL implementations for functions weibull_lpdf, neg_binomial_2_log_lpmf and neg_binomial_2_lpmf.(#2231)
• Added OpenCL implementations of binomial_lpmf and neg_binomail_lpmf.(#2236)
• Also extends kernel generator element-wise functions to work with integer arguments. Adds option for kernel generator expression to be a scalar, as long as at least one expression in a kernel is not a scalar.(#2236)

Expressions:

• student t distribution related functions accept Eigen expressions.(#2131)
• uniform distribution related functions accept Eigen expressions.(#2133)
• Weibull distribution related functions accept Eigen expressions.(#2136)
• Wiener distribution accept Eigen expressions.(#2138)
• Wishart distribution related functions accept Eigen expressions.(#2139)
• Fixed how dot_product handles expressions.(#2150)
• Generalized remaining functions to accept Eigen expressions (odes, algebra solvers, csr-related functions, multiply_lower_tri_self_transpose, block, map_rect, log_determinant, matrix_exp_multiply, sd).(#2151)
• Removed the file with the list of function signatures, ignored by expression testing framework.(#2186)
• pareto distribution related functions accept Eigen expressions.(#2093)

[avehtari]

"Generalized student t distribution related functions to accept Eigen expressions" etc is confusing as it looks like we would now have new distributions: generalized student t, generalized uniform, generalized Weibull, etc which all are existing distributions. Only the line "Generalizes Wishart distribution" breaks this pattern, and I assume the distributions were not generalized but just the functions.

[bbbales2]

Oh yeah lol. This is correct. "Generalized" means the function supports Eigen expressions, not a new distribution. Let me make a note at the top to edit this on the final versions.

[rok-cesnovar]

This is the unedited version which is just a grouped dump of the release notes written in PRs. In the final version all the “generalize” notes will be something like:

Generalized all remaining C++ functions to accept Eigen expressions.

EDIT: Ben beat me too it :)

[mitzimorris]

in the last release, were there hiccups post-release w/r/t CmdStan makefile?

Rok recently reminded me of #929, which get rid of use of a Boost non-header only library - will try to get this in ASAP

[rok-cesnovar]

The first attempt for the RC tarball was missing CMDSTAN_VERSION somehow, which is why I noted that prominently above. Other than that there were no new issues.

[bbbales2]

Also are expressions just done? The compiler knows how to use them. There are no language changes to account for them. They are used in Math. They are tested, and I remember we added some doc on how to use those tests somewhere.

If anything doesn't exist, it's higher level C++ docs (like are also missing for varmat).

[rok-cesnovar]

Also are expressions just done?

In terms of code I think they are done, at least what was meant for 2.26. At some point, maybe for 2.27, we may attempt also returning and handling expressions in Stan user-defined functions, but that is too big of a project for 2.26.

[bbbales2]

Sounds good!

[avehtari]

@bbbales2 asked me to look different lists, and I'm very happy to see this level of planning and clear checklists. I added couple minor comments in other issues, and here I ask clarification

• If one of the open project checklists is not done January 13th, does it mean that that project will not be included in the next release?
• Is it possible to add new open projects before January 13th as long as there is corresponding checklist for "what needs to be done and how it will be verified that it is done" that is considered to be feasible? Or do you know that there are not such projects that someone might want to include?

[SteveBronder]

Is it possible to add new open projects before January 13th as long as there is corresponding checklist for "what needs to be done and how it will be verified that it is done" that is considered to be feasible? Or do you know that there are not such projects that someone might want to include?

If someone has a project that can be done by Jan 13th then I'm fine with releasing it this version

[bbbales2]

We're calling off varmat for the 2.26 release. There are too many pull requests stacked up at the last minute. Specifically, calling off varmat means we're not going to turn on any varmat stuff in the compiler in 2.26 (there's plenty of varmat stuff in Math).

Since varmat was the reason we were doing the early freeze (today), Rok & I would like to move that freeze to the same day as everything else (the 18th). This does not change the regularly scheduled freeze & release timings (18th and 25th, respectively). We just don't want to do the early Math freeze without a reason. The early Math freeze was to give time for varmat compiler development. Since that is off, no reason to keep our Math repository frozen longer than it needs to be.

If anyone has objections to canceling the early Math freeze, let us know by the end of the day today (13th), otherwise we'll just keep reviewing and merging stuff as planned.

[bbbales2]

Looks like no comments so we'll delay the Math freeze until the 18th (with everything else)

[wds15]

Getting varmat off the ground is definitely worth a week extra time (also two weeks)!

[bbbales2]

We're just gonna delay it to 2.27. Doesn't feel great, but it would be rushed at this point. And no need to clog up the release as is (which still has lots of good stuff in it).

[rok-cesnovar]

@serban-nicusor-toptal https://jenkins.mc-stan.org/blue/organizations/jenkins/Math%20Pipeline/detail/develop/1021/pipeline should be the final develop test before the freeze. Once that finishes feel free to make a Math RC (12-13h from now) and once that propagates upstream you can do RCs for Stan/CmdStan. Current Stanc3 nightly is what will be the RC stanc3. Thanks!

[serban-nicusor-toptal]

Hey @rok-cesnovar RCs are out now!

[rok-cesnovar]

Thanks Nic! I think the tarball is missing binaries for stanc3?

I started the cmdstanr tests with the tarball for a first sanity check of the tarballs https://github.com/stan-dev/cmdstanr/actions/runs/495905352

[serban-nicusor-toptal]

It should contain the stanc3 binaries from nightly release, all together inside /bin

[rok-cesnovar]

I am not seeing it here: https://github.com/stan-dev/cmdstan/releases/download/v2.26.0-rc1/cmdstan-2.26.0-rc1.tar.gz

The tarball is also smaller than for 2.25 which made me check that.

[serban-nicusor-toptal]

@rok-cesnovar you are right, turns out git-archive-all does not include /bin in the final archive. I've re-created it manually and uploaded it again. Should be fine now.

[rok-cesnovar]

Yeah, I think because /bin is in gitignore. Thanks!

[serban-nicusor-toptal]

Hey @rok-cesnovar is everything ready for the release ? Asking just to be sure before starting the proces.

[rok-cesnovar]

Hey @serban-nicusor-toptal, I think everything is ready, but given that we released the RC on Tuesday I think we should do the actual release tomorrow. Does that work for you?

[serban-nicusor-toptal]

Sure that's fine, we'll leave it for tomorrow! Thanks for the quick reply, have a great week!

[rok-cesnovar]

@serban-nicusor-toptal here are all the release notes for the 4 repositories: https://gist.github.com/rok-cesnovar/28750bf5b79026bb2ae4dc37190c744b

Let me know if you need anything from me. We can now proceed with the release whenever you have time.

[wds15]

Ther was a fix for the gradients of the multi variate normal cholesky lpdf...not sure I saw that one.

[rok-cesnovar]

Oh right, good call. Forgot to include the post-freeze stuff (ran the script the day after the freeze). Thanks! Updated now.

[rok-cesnovar]

@serban-nicusor-toptal also listing the version numbers:

Math version should be updated to 4.0
Stan is 2.26 (we need to update the version numbers, we forgot the last time, see https://github.com/stan-dev/stan/pull/2990/files)
Cmdstan and stanc3 also 2.26

Thanks!

[serban-nicusor-toptal]

If there are no objections I will start the release in 1 hour from now.

[serban-nicusor-toptal]

Hey @rok-cesnovar release is now done across all projects!

[rok-cesnovar]

Thanks Nic, will check the tarball and make the announcement. If all is good, will close this issue.

[rok-cesnovar]

Checklist completed, everything released. Closing.

Thanks @serban-nicusor-toptal !