GH-41375: [C#] Move to .NET 8.0

[CurtHagenlocher]

What changes are included in this PR?

Changes to workflow infrastructure and projects to install and target net8.0 instead of net7.0.

Are these changes tested?

Yes

Are there any user-facing changes?

Users will need to install .NET 8 to run tests and examples. No impact on product code.

Closes #41375

• GitHub Issue: [C#] Move to net8.0 for running tests in workflows #41375

[CurtHagenlocher]

I think all the failing checks are due to the recent macos-latest change, but am not confident that it's the case.

[raulcd]

@github-actions crossbow submit nuget

[raulcd]

@github-actions crossbow submit csharp

[github-actions]

Revision: 1439115

Submitted crossbow builds: ursacomputing/crossbow @ actions-a0ad3baf29

Task	Status
nuget

[github-actions]

Revision: 1439115

Submitted crossbow builds: ursacomputing/crossbow @ actions-4f3461e690

Task	Status
verify-rc-source-csharp-linux-almalinux-8-amd64
verify-rc-source-csharp-linux-conda-latest-amd64
verify-rc-source-csharp-linux-ubuntu-20.04-amd64
verify-rc-source-csharp-linux-ubuntu-22.04-amd64
verify-rc-source-csharp-macos-amd64
verify-rc-source-csharp-macos-arm64

[raulcd]

Change looks good to me. There is a single error on CI around the .NET jobs but I think is unrelated:
https://github.com/apache/arrow/actions/runs/8836761043/job/24264073837?pr=41376

Run ci/scripts/csharp_test.sh $(pwd)
+ source_dir=/Users/runner/work/arrow/arrow/csharp
+ '[' -z '' ']'
+ type python3
+ export PYTHON=python3
+ PYTHON=python3
+ python3 -m pip install pyarrow find-libpython
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:
    
    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz
    
    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with
    
    brew install pipx
    
    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.
    
    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.
    
    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

@kou @assignUser is this something you've seen somewhere else?
I am happy to merge this PR as it seems related to the macOS changes for GitHub runners.

[kou]

We need to use venv as the message said or actions/setup-python.

[raulcd]

I've opened #41390 to follow the error

[assignUser]

Yeah, this is a recent change in the brew python. As kou said using venv solves it.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2710626.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 2 possible false positives for unstable benchmarks that are known to sometimes produce them.