Update versions of LightGBM dependencies

[StrikerRUS]

Refer to #3763.

All dependencies were updated to the latest available stable version.

https://github.com/boostorg/compute/releases/tag/boost-1.78.0
https://gitlab.com/libeigen/eigen/-/releases/3.4.0
https://github.com/lemire/fast_double_parser/releases/tag/v0.5.0 see #4935 (comment)
https://github.com/fmtlib/fmt/releases/tag/8.1.1

[StrikerRUS]

All failing CI jobs are R-package jobs and the error is about hidden files:

* checking for hidden files and directories ... NOTE
Found the following hidden files and directories:
  src/external_libs/fmt/support/bazel/.bazelrc
  src/external_libs/fmt/support/bazel/.bazelversion
These were most likely included in error. See section ‘Package
structure’ in the ‘Writing R Extensions’ manual.

CRAN-pack does not know about
  src/external_libs/fmt/support/bazel/.bazelrc
  src/external_libs/fmt/support/bazel/.bazelversion

[jameslamb]

error is about hidden files

Adding additional rules in https://github.com/microsoft/LightGBM/blob/master/R-package/.Rbuildignore should fix that.

[StrikerRUS]

Here is a description for them in the original PR where they were introduced:

The .bazelrc is kind of optional. It could simply be deleted. In its current form, it just avoids that some temporary Bazel-related folders are created.

The .bazelversion file can also be deleted. It is just a file for Bazelisk and a nice way to document which Bazel version was aimed for/tested with (I am pretty sure that it works with almost every Bazel version).

fmtlib/fmt#2505 (comment)

According to this, I agree that we can safely ignore/remove them.

[StrikerRUS]

Linux_latest regular job is failing (probably segfault):

../tests/python_package_test/test_dual.py s                              [ 64%]
../tests/python_package_test/test_engine.py .....................s...s.. [ 68%]
..................................../__w/1/s/.ci/test.sh: line 212:  7869 Killed                  pytest $BUILD_DIRECTORY/tests
##[error]Bash exited with code '255'.

https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=11943&view=logs&j=275189f9-c769-596a-7ef9-49fb48a9ab70&t=3a9e7a4a-04e6-52a0-67ea-6e8f6cfda74f
Need to identify the failing testcase and try to reproduce the error.

[StrikerRUS]

The latest commit passes all CI checks. I'd like to run CI multiple times to identify flaky test and affected environments.

1. ✔️
2. ✔️
3. ✔️
4. ❌ Linux_latest sdist: ../tests/python_package_test/test_engine.py::test_model_size /__w/1/s/.ci/test.sh: line 155: 7882 Killed
5. ✔️
6. ✔️
7. ❌ Linux_latest regular: ../tests/python_package_test/test_engine.py::test_model_size /__w/1/s/.ci/test.sh: line 212: 7874 Killed
8. ✔️
9. ✔️
10. ✔️

[StrikerRUS]

Seems that failing test is test_model_size

LightGBM/tests/python_package_test/test_engine.py

Line 2285 in ce486e5

def test_model_size():

and I guess that segfault happens here

LightGBM/tests/python_package_test/test_engine.py

Line 2304 in ce486e5

bst.model_from_string(new_model_str)

LightGBM/include/LightGBM/utils/common.h

Lines 359 to 360 in b0137de

	// Use fast_double_parse and strtod (if parse failed) to parse double.
	inline static const char* AtofPrecise(const char* p, double* out) {

[StrikerRUS]

Reverting fast_double_parser update for now.

1. ✔️
2. ✔️
3. ✔️
4. ✔️
5. ✔️
6. ✔️
7. ✔️
8. ✔️
9. ✔️
10. ✔️
11. ✔️
12. ✔️
13. ✔️ (one job times out due to Linux CI jobs hang forever after completing all Python tests successfully #4948 and re-runs don't help)
14. ✔️
15. ✔️

---

Swapped compilers

1. ❌ (only Linux GPU job with Clang compiler failed, expected according to SEGFAULT on GPU version compiled with Clang #3475)
2. ❌ (only Linux GPU job with Clang compiler failed, expected according to SEGFAULT on GPU version compiled with Clang #3475)
3. ❌ (only Linux GPU job with Clang compiler failed, expected according to SEGFAULT on GPU version compiled with Clang #3475)
4. ❌ (only Linux GPU job with Clang compiler failed, expected according to SEGFAULT on GPU version compiled with Clang #3475)
5. ❌ (only Linux GPU job with Clang compiler failed, expected according to SEGFAULT on GPU version compiled with Clang #3475)

[StrikerRUS]

I think this PR is ready for review.

Based on 15 successful CI runs we can hope that this dependencies version upgrade is quite stable. I triggered all 15 runs via with empty commits because I noticed that sometimes if you re-run CI via closing-reopening a PR or by manually clicking rerun UI button (for those CIs where it's possible) new re-triggered CI job may reuse a previous runner, which breaks independency of tests.

[jameslamb]

Looks great! Thanks for the thorough testing and explanation about fast_double_parser.

[StrikerRUS]

Just marked this PR as WIP because I'd like to merge #4953 first. Also, I forgot to run tests with swapped compilers at CI. Sorry for the inconvenience!

[StrikerRUS]

Well, I think this PR is ready for review.
#4935 (comment)

[jameslamb]

Thanks for the very thorough testing! I support merging this.

Pretty exciting, looks like there have been a lot of improvements in Eigen from the version currently shipped with LightGBM to now. For example, https://eigen.tuxfamily.org/index.php?title=3.4.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.