Release CmdStan 2.32

[rok-cesnovar]

Feature freeze (9th or 10th April):

• Ensure all expiring language deprecations have been removed or given new removal dates.
• Create and merge version updating pull requests in Math/Stan/Cmdstan. These should be the last PRs accepted before the freeze.
• Create Math/Stan RC releases.
• Create Stanc3 RC binary.
• Create a release candidate tarball for x86. Make sure RC tarballs include stanc3 binaries.
• Check external links in docs (i.e. TBB docs link) - use https://github.com/tcort/markdown-link-check
• Create a release candidate feature/bugfix list (major features/bugfixes that need testing, link to new docs in Github)
• Run CmdStanR tests with the RC tarball.
• Run CmdStanPy tests with the RC tarball.
• Make a Discourse RC post.
• Post a tweet with a link to the Discourse RC post.

Edit: We did not do an RC announcement this time around.

Release (TBD):

• Create the Math Release notes.
• Create the Stan Release notes.
• Create the Cmdstan Release notes.
• Create the Stanc3 Release notes, include new deprecations and removals.
• Rebuild and publish docs for the new version.
• Check that docs for the previous release links correctly to the newest docs.
• Create the Math release.
• Create the Stan release.
• Create the Stanc3 release.
• Create x86 CmdStan tarballs (check version, check that the extracted folder is in the cmdstan-version format).
• Create non-x86 CmdStan tarballs.
• Run CmdStanR tests with the release tarball.
• Make a Stan blog release announcement post (thank the sponsors and all contributors, mention new devs).
• Link to the blog post in a Discourse thread.
• Make a Twitter announcement.

[WardBrian]

I checked off the first box since we delayed deprecations a version (stan-dev/stanc3#1303). Additionally, no new deprecations this version.

[WardBrian]

CmdStanPy tests passed https://github.com/stan-dev/cmdstanpy/actions/runs/4658834327/jobs/8244978620

I needed to update one test which was probably too brittle for its own good, since log_prob's output file had a column change name

[WardBrian]

@rok-cesnovar @serban-nicusor-toptal is there anything I can do to help with the release?

It doesn't seem like there have been any new issues/reports from the RC

[serban-nicusor-toptal]

I tried to run CmdStanR tests last week but it's failing with

testthat::expect_error(funmod$expose_functions(), "Overloaded functions are currently not able to be exposed to R! The following overloaded functions were found: fun1, fun3") at test-model-expose-functions.R:147:2

https://github.com/stan-dev/cmdstanr/actions/runs/4677937287

If there's nothing pending and the release notes are ready, I can do the release tomorrow or Wednesday. Let's see what Rok has in mind too.

[WardBrian]

That seems unrelated to the new release, but we should check with @andrjohns @jgabry

I’m not sure if Rok has had a chance to draft release notes

[andrjohns]

Ah yep, I know where those failures are coming from. I can fix those tomorrow

[WardBrian]

I ran the release-notes script (which is not as automated as I thought!) and did some light editing. I'd appreciate another pass by @rok-cesnovar if he has time

Math
 - Updated to Eigen 3.4 (#2583)
 - operands_and_partials expanded to accept 8 parameters. (#2833)
 - Added new multidimensional integration routine `hcubature`. (#2838)
 - Added eigenvalues and eigenvectors overloads which accept matrices with complex values. (#2846)
 - Allowed Sundials to be linked to separately installed version. (#2861)
 - Fixed `is_nan` to compile under the Microsoft compiler suite. (#2870)
 - Fixed `csr_to_dense_matrix` to work in case of matrix with empty first row. (#2877)
 - Fixed a bug where `eigenvalues_sym` would return a matrix with a dynamic number of rows equal to 1, rather than a vector. (#2884)
 - Updated `eigenvalues_sym` return type for RcppEigen compatibility. (#2887)
 - Various testing improvements. (#2832, #2864, #2869, #2878, #2880)
 - Fixed an issue with building TBB on gcc 13. (#2890)

Stan
 - Extended `get_dims` and `get_param_names` in model_base with flags for whether to include transformed parameters and generated quantities. (#3139)
 - Fixed an issue where initialization would still retry many times even if a fixed initialization was provided. (#3139)
 - Cleaned up newlines in Laplace service logging. (#3153)
 - make: use SUNDIALS_TARGETS instead of LIBSUNDIALS. (#3162)
 - Updated JSON var context to handle nested structures (tuples). (#3165)
 - Always discard at least 1 entry in new RNGs. (#3168)
 - Relaxed requirements for 0-sized data objects in JSON. (#3170)
 - Deleted unused io/cmd_line.hpp. (#3172)
 - Standalone GQ: Output NaNs when exception is caught. (#3173)
 - Cleaned up dimension and name logic in random_var_context. (#3174)
 - Removed old/unused IO code. (#3177)
 - Made deserializer's map members const. (#3178)

CmdStan
 - Added method `laplace` to sample from a Laplace approximation following optimization. (#1134)
 - Added argument `jacobian` to method `optimize`; default is FALSE (current behavior). (#1134)
 - For method `log_prob`, added functionality to allow sets of constrained parameters as inputs in the form of a Stan CSV file. (#1134)
 - Fixed typo in OpenCL code. (#1141)
 - Updated a test model so it no longer uses the deprecated array syntax. (#1148)
 - Fixed Bernoulli example file mentioning old array syntax in a comment. (#1150)
 - make: use SUNDIALS_TARGETS instead of LIBSUNDIALS. (#1151)
 -  Fixed 'Perhaps you meant' giving invalid suggestions. (#1152) 

Stanc3
 - Syntax removals scheduled for this version have been pushed back to 2.33. We still recommend updating your Stan models as you are able. As a reminder, deprecated syntax which will be removed in minor versions can as a rule be updated automatically using find-and-replace tools or the [Stan compiler's auto-formatting abilities](https://mc-stan.org/docs/stan-users-guide/stanc-pretty-printing.html). (#1303)
 - The deprecated `--include_paths` and `--allow_undefined` arguments have been removed, use `--include-paths` and `--allow-undefined`. (#1303)
 - Recursive user-defined functions no longer need forward declarations. (#1277)
 - Added overloads to `eigenvalues` and `eigenvectors` which accept a complex matrix as input. (#1282)
 - Fixed an issue where `profile` blocks could generate uncompilable C++ when used with `--O1`. (#1281)
 - Fixed an issue where the compiler would generate uncompilable C++ when a matrix expression featured mixed types of elements. (#1292)
 - Fixed an issue where a user-defined-function expecting real arguments could be passed an integer and exhibit integer behavior like truncating division. (#1296)
 - Fixed an issue where external functions could not be used in higher-order functions without the end-user supplying an implementation of our functor structs. (#1298)
 - Fixed a bug in `transform_inits` for models which used complex types (#1305) 
 - Improved the error shown when an `#include`d file cannot be found to list the paths it searched on. (#1273)
 - Improved `--filename-in-msg` for the command line version of `stanc`. Errors now use the requested name, not just warnings. (#1274)
 - Refactored how C++ is code-generated in the compiler. (#1233)
 - Added tests for various parts of the C++ backend which weren't previously tested, and added a `--debug-lir` flag to print out the C++ representation in a s-expression. (#1276)
 - Updated code generate of `get_param_names` (#1245)
 - Switched to `std::endl` instead of `'\n'` to end prints. (#1301)
 - Added a new function `unconstrain_array` to the generated code. This is currently not exposed. (#1305)
 - Added a `--debug-generate-inits` flag to stanc which behaves like `--debug-generate-data` but for declarations in the `parameters` block. (#1306)
 - Sped up some code which is critical to parsing. (#1302)

[rok-cesnovar]

These look great to me. Thanks @WardBrian!

So in terms of user-facing features at the (cmd)stan level we could sum this up to:

• eigenvalues/eigenvectors now accept complex_matrix,
• recursive UDFs no longer need forward declarations,
• Laplace approximation,
• jacobian argument for optimization

[WardBrian]

Yeah, I'd say this update is those 4 features, a ton of bug fixes, and a lot of prep for upcoming features (if we're still doing a "preview" section of the release blog).

[WardBrian]

Since we're planning on merging stan-dev/math#2890, I've updated the above change logs to mention it

[serban-nicusor-toptal]

I see that stan-dev/math#2890 got merged. Is there anything else pending or can I do the release today ?

[WardBrian]

Assuming @rok-cesnovar is ready to make the announcements I think it’s good to go!

[rok-cesnovar]

Lets do it! Will post the blog & tweets in the evening (afternoon US time).

[serban-nicusor-toptal]

Update:

The release is almost entirely done:

• https://github.com/stan-dev/math/releases/tag/v4.6.0
• https://github.com/stan-dev/stan/releases/tag/v2.32.0
• https://github.com/stan-dev/cmdstan/releases/tag/v2.32.0
• https://github.com/stan-dev/stanc3/releases/tag/v2.32.0

In cmdstan we have a warning which failed the quality gate
https://jenkins.flatironinstitute.org/job/Stan/job/CmdStan/job/PR-1157/1/Mac_gcc4/
Will be fixed by https://github.com/stan-dev/cmdstan/pull/1143/files#diff-e6ffa5dc854b843b3ee3c3c28f8eae2f436c2df2b1ca299cca1fa5982e390cf8

There are also some tests failing in the CmdStan Performance Tests
https://jenkins.flatironinstitute.org/job/Stan/job/CmdStan%20Performance%20Tests/job/master/

Thanks to @WardBrian for fixing a test that was causing CI failure during a release.

The last step left is to build the docs. I think the gelman mac updated to a newer os version and somehow messed up the linker.
I'm trying to update gcc and llvm to see if that does it, sadly brew on this machine is super slow. In case it will take too much time, I'll manually build the docs.

Edit: Docs are up! Brian helped me to do it manually for now. We'll migrate this step to Docker so we won't have this kind of issue next time.

[WardBrian]

@rok-cesnovar I think everything is ready for the release posts/announcements when you get a chance!

[rok-cesnovar]

Ah, I see the edit now. Thanks for the bump. Announcements coming up.

[rok-cesnovar]

The blog post is live, posted the link on Discourse. Having some access issues with Twitter, but otherwise, everything else is done. A big thank you to @WardBrian and @serban-nicusor-toptal for all the hard work on getting this over the line!

[rok-cesnovar]

Twitter announcement is done. Posting on mastodon as soon as I figure it out (have the login info for that now).

[WardBrian]

@rok-cesnovar Should we plan on a 2.32.1 from stan-dev/math#2892?

[rok-cesnovar]

Yeah, lets plan. I think we should wait at least a week to seek if any additional reports come up?

[WardBrian]

I'll keep a running list of things merged and things I think we should merge:

Merged

• Add overloads for complex multiply to fix clang16 math#2892
• Add locations to output variables, adjust code generation stanc3#1318
• Validate the dimensionality of variables in transform_inits stanc3#1316
• Add optional --data-file for --debug-generate-inits stanc3#1319
• Laplace sampling: change output columns for consistency stan#3184

Consider for merging

• Use full return type in --standalone-functions definitions stanc3#1313
• Clean and speed up Location.of_string_opt stanc3#1310

[WardBrian]

Unless something else comes up, are we good to do a .1 early next week @rok-cesnovar @serban-nicusor-toptal?

[serban-nicusor-toptal]

Hey @rok-cesnovar @WardBrian can I please ask you to generate and validate the Release Notes for this patch release ? I think that's all I need to continue. Thanks!

[WardBrian]

Here's my go, I'd appreciate a look by @rok-cesnovar

Math

- Fix an issue when compiling with newer versions of LLVM's libc++ (#2892)

Stan

- Fix an issue where the output of Laplace sampling was inconsistent with other outputs and could lead to invalid CSV headers (#3184)

Stanc3

- The shapes and sizes of parameter initial values are now checked against the specified sizes in the model before being read in, matching the existing behavior for data. (#1316)
  **Note:** This change may cause existing code which worked "by accident" to raise new errors about mismatches between the model's declarations and the provided initialization.
- Errors in transform_inits should now properly highlight the location in the original Stan program. (#1318)
- Fixed an issue where the `--debug-generate-inits` debug flag would fail badly if parameter sizes depended on data (#1319)

There are no direct changes to CmdStan besides updating to use the newer versions of stan/math/stanc3, not sure what we put in the release notes in those cases

[rok-cesnovar]

Sorry, it was a holiday yesterday so I was AFK. Looks great to me.

[serban-nicusor-toptal]

np! I will start with the release soon after lunch so we finish when US wakes up. I'll keep you posted.

[serban-nicusor-toptal]

The release 2.32.1 is now up!

• https://github.com/stan-dev/stanc3/releases/tag/v2.32.1
• https://github.com/stan-dev/cmdstan/releases/tag/v2.32.1
• https://github.com/stan-dev/stan/releases/tag/v2.32.1
• https://github.com/stan-dev/math/releases/tag/v4.6.1

I'm currently waiting on the collab build to finish and I have to merge release branch -> master and master -> develop.
Is there anything else to be done ?

Edit: collab build done.
Edit: master/develop up to date.

[rok-cesnovar]

I don't think so. I will publish this on Discourse/Twitter/mastodon/stanblog a bit later.

[WardBrian]

Thanks @serban-nicusor-toptal @rok-cesnovar!