Skip to content

Commit

Permalink
Update release scripts now that DataFusion is TLP (apache#10235)
Browse files Browse the repository at this point in the history
  • Loading branch information
@andygrove
andygrove authored Apr 26, 2024
1 parent 4ec3d51 commit 44f1147
Show file tree
Hide file tree
Showing 8 changed files with 61 additions and 150 deletions.
81 changes: 35 additions & 46 deletions dev/release/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ patch release:
- Created a personal access token in GitHub for changelog automation script.
- Github PAT should be created with `repo` access
- Make sure your signing key is added to the following files in SVN:
- https://dist.apache.org/repos/dist/dev/arrow/KEYS
- https://dist.apache.org/repos/dist/release/arrow/KEYS
- https://dist.apache.org/repos/dist/dev/datafusion/KEYS
- https://dist.apache.org/repos/dist/release/datafusion/KEYS

### How to add signing key

Expand All @@ -58,8 +58,8 @@ See instructions at https://infra.apache.org/release-signing.html#generate for g
Committers can add signing keys in Subversion client with their ASF account. e.g.:

```bash
$ svn co https://dist.apache.org/repos/dist/dev/arrow
$ cd arrow
$ svn co https://dist.apache.org/repos/dist/dev/datafusion
$ cd datafusion
$ editor KEYS
$ svn ci KEYS
```
Expand Down Expand Up @@ -128,7 +128,7 @@ release.

See [#9697](https://github.com/apache/datafusion/pull/9697) for an example.

Here are the commands that could be used to prepare the `5.1.0` release:
Here are the commands that could be used to prepare the `38.0.0` release:

### Update Version

Expand All @@ -139,10 +139,10 @@ git fetch apache
git checkout apache/main
```

Update datafusion version in `datafusion/Cargo.toml` to `5.1.0`:
Update datafusion version in `datafusion/Cargo.toml` to `38.0.0`:

```
./dev/update_datafusion_versions.py 5.1.0
./dev/update_datafusion_versions.py 38.0.0
```

Lastly commit the version change:
Expand All @@ -167,7 +167,7 @@ Pick numbers in sequential order, with `0` for `rc0`, `1` for `rc1`, etc.
While the official release artifacts are signed tarballs and zip files, we also
tag the commit it was created for convenience and code archaeology.

Using a string such as `5.1.0` as the `<version>`, create and push the tag by running these commands:
Using a string such as `38.0.0` as the `<version>`, create and push the tag by running these commands:

```shell
git fetch apache
Expand All @@ -181,29 +181,29 @@ git push apache <version>
Run `create-tarball.sh` with the `<version>` tag and `<rc>` and you found in previous steps:

```shell
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 5.1.0 0
GH_TOKEN=<TOKEN> ./dev/release/create-tarball.sh 38.0.0 0
```

The `create-tarball.sh` script

1. creates and uploads all release candidate artifacts to the [arrow
dev](https://dist.apache.org/repos/dist/dev/arrow) location on the
1. creates and uploads all release candidate artifacts to the [datafusion
dev](https://dist.apache.org/repos/dist/dev/datafusion) location on the
apache distribution svn server

2. provide you an email template to
send to dev@arrow.apache.org for release voting.
send to dev@datafusion.apache.org for release voting.

### Vote on Release Candidate artifacts

Send the email output from the script to dev@arrow.apache.org. The email should look like
Send the email output from the script to dev@datafusion.apache.org. The email should look like

```
To: dev@arrow.apache.org
Subject: [VOTE][DataFusion] Release Apache DataFusion 5.1.0 RC0
To: dev@datafusion.apache.org
Subject: [VOTE] Release Apache DataFusion 38.0.0 RC1
Hi,
I would like to propose a release of Apache DataFusion version 5.1.0.
I would like to propose a release of Apache DataFusion version 38.0.0.
This release candidate is based on commit: a5dd428f57e62db20a945e8b1895de91405958c4 [1]
The proposed release artifacts and signatures are hosted at [2].
Expand All @@ -214,16 +214,16 @@ and vote on the release.
The vote will be open for at least 72 hours.
[ ] +1 Release this as Apache DataFusion 5.1.0
[ ] +1 Release this as Apache DataFusion 38.0.0
[ ] +0
[ ] -1 Do not release this as Apache DataFusion 5.1.0 because...
[ ] -1 Do not release this as Apache DataFusion 38.0.0 because...
Here is my vote:
+1
[1]: https://github.com/apache/datafusion/tree/a5dd428f57e62db20a945e8b1895de91405958c4
[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-datafusion-5.1.0
[2]: https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-38.0.0
[3]: https://github.com/apache/datafusion/blob/a5dd428f57e62db20a945e8b1895de91405958c4/CHANGELOG.md
```

Expand All @@ -234,7 +234,7 @@ For the release to become "official" it needs at least three PMC members to vote
The `dev/release/verify-release-candidate.sh` is a script in this repository that can assist in the verification process. Run it like:

```
./dev/release/verify-release-candidate.sh 5.1.0 0
./dev/release/verify-release-candidate.sh 38.0.0 0
```

#### If the release is not approved
Expand All @@ -249,11 +249,11 @@ NOTE: steps in this section can only be done by PMC members.
### After the release is approved

Move artifacts to the release location in SVN, e.g.
https://dist.apache.org/repos/dist/release/datafusion/datafusion-5.1.0/, using
https://dist.apache.org/repos/dist/release/datafusion/datafusion-38.0.0/, using
the `release-tarball.sh` script:

```shell
./dev/release/release-tarball.sh 5.1.0 0
./dev/release/release-tarball.sh 38.0.0 0
```

Congratulations! The release is now official!
Expand All @@ -263,9 +263,9 @@ Congratulations! The release is now official!
Tag the same release candidate commit with the final release tag

```
git co apache/5.1.0-rc0
git tag 5.1.0
git push apache 5.1.0
git co apache/38.0.0-rc0
git tag 38.0.0
git push apache 38.0.0
```

### Publish on Crates.io
Expand Down Expand Up @@ -300,7 +300,7 @@ of the following crates:
Download and unpack the official release tarball

Verify that the Cargo.toml in the tarball contains the correct version
(e.g. `version = "5.1.0"`) and then publish the crates by running the script `release-crates.sh`
(e.g. `version = "38.0.0"`) and then publish the crates by running the script `release-crates.sh`
in a directory extracted from the source tarball that was voted on. Note that this script doesn't
work if run in a Git repo.

Expand Down Expand Up @@ -413,10 +413,9 @@ https://crates.io/crates/datafusion-substrait/28.0.0

### Add the release to Apache Reporter

Add the release to https://reporter.apache.org/addrelease.html?arrow with a version name prefixed with `RS-DATAFUSION-`,
for example `RS-DATAFUSION-14.0.0`.
Add the release to https://reporter.apache.org/addrelease.html?datafusion using the version number e.g. 38.0.0.

The release information is used to generate a template for a board report (see example
The release information is used to generate a template for a board report (see example from Apache Arrow project
[here](https://github.com/apache/arrow/pull/14357)).

### Delete old RCs and Releases
Expand All @@ -431,13 +430,13 @@ Release candidates should be deleted once the release is published.
Get a list of DataFusion release candidates:

```bash
svn ls https://dist.apache.org/repos/dist/dev/arrow | grep datafusion
svn ls https://dist.apache.org/repos/dist/dev/datafusion
```

Delete a release candidate:

```bash
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-7.1.0-rc1/
svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-38.0.0-rc1/
```

#### Deleting old releases from `release` svn
Expand All @@ -447,35 +446,25 @@ Only the latest release should be available. Delete old releases after publishin
Get a list of DataFusion releases:

```bash
svn ls https://dist.apache.org/repos/dist/release/arrow | grep datafusion
svn ls https://dist.apache.org/repos/dist/release/datafusion
```

Delete a release:

```bash
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-7.0.0
svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-37.0.0
```

### Publish the User Guide to the Arrow Site

- Run the `build.sh` in the `docs` directory from the release tarball.
- Clone the [arrow-site](https://github.com/apache/arrow-site) repository
- Checkout the `asf-site` branch
- Copy content from `docs/build/html/*` to the `datafusion` directory in arrow-site
- Create a PR against the `asf-site` branch ([example](https://github.com/apache/arrow-site/pull/237))
- Once the PR is merged, the content will be published to https://datafusion.apache.org/ by GitHub Pages (this
can take some time).

### Optional: Write a blog post announcing the release

We typically crowdsource release announcements by collaborating on a Google document, usually starting
We typically crowd source release announcements by collaborating on a Google document, usually starting
with a copy of the previous release announcement.

Run the following commands to get the number of commits and number of unique contributors for inclusion in the blog post.

```bash
git log --pretty=oneline 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
git shortlog -sn 10.0.0..11.0.0 datafusion datafusion-cli datafusion-examples | wc -l
git log --pretty=oneline 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l
git shortlog -sn 37.0.0..38.0.0 datafusion datafusion-cli datafusion-examples | wc -l
```

Once there is consensus on the contents of the post, create a PR to add a blog post to the
Expand Down
20 changes: 10 additions & 10 deletions dev/release/create-tarball.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@
# Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh

# This script creates a signed tarball in
# dev/dist/apache-arrow-datafusion-<version>-<sha>.tar.gz and uploads it to
# the "dev" area of the dist.apache.arrow repository and prepares an
# email for sending to the dev@arrow.apache.org list for a formal
# dev/dist/apache-datafusion-<version>-<sha>.tar.gz and uploads it to
# the "dev" area of the dist.apache.datafusion repository and prepares an
# email for sending to the dev@datafusion.apache.org list for a formal
# vote.
#
# See release/README.md for full release instructions
Expand Down Expand Up @@ -65,21 +65,21 @@ tag="${version}-rc${rc}"
echo "Attempting to create ${tarball} from tag ${tag}"
release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag})

release=apache-arrow-datafusion-${version}
release=apache-datafusion-${version}
distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc}
tarname=${release}.tar.gz
tarball=${distdir}/${tarname}
url="https://dist.apache.org/repos/dist/dev/arrow/${release}-rc${rc}"
url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}"

if [ -z "$release_hash" ]; then
echo "Cannot continue: unknown git tag: ${tag}"
fi

echo "Draft email for dev@arrow.apache.org mailing list"
echo "Draft email for dev@datafusion.apache.org mailing list"
echo ""
echo "---------------------------------------------------------"
cat <<MAIL
To: dev@arrow.apache.org
To: dev@datafusion.apache.org
Subject: [VOTE][RUST][DataFusion] Release Apache DataFusion ${version} RC${rc}
Hi,
Expand Down Expand Up @@ -124,12 +124,12 @@ echo "Signing tarball and creating checksums"
gpg --armor --output ${tarball}.asc --detach-sig ${tarball}
# create signing with relative path of tarball
# so that they can be verified with a command such as
# shasum --check apache-arrow-datafusion-4.1.0-rc2.tar.gz.sha512
# shasum --check apache-datafusion-38.0.0-rc1.tar.gz.sha512
(cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256
(cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512


echo "Uploading to apache dist/dev to ${url}"
svn co --depth=empty https://dist.apache.org/repos/dist/dev/arrow ${SOURCE_TOP_DIR}/dev/dist
echo "Uploading to datafusion dist/dev to ${url}"
svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist
svn add ${distdir}
svn ci -m "Apache DataFusion ${version} ${rc}" ${distdir}
6 changes: 3 additions & 3 deletions dev/release/publish_homebrew.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ else # Fallback
num_processing_units=1
fi

url="https://www.apache.org/dyn/closer.lua?path=arrow/arrow-datafusion-${version}/apache-arrow-datafusion-${version}.tar.gz"
sha256="$(curl https://dist.apache.org/repos/dist/release/arrow/arrow-datafusion-${version}/apache-arrow-datafusion-${version}.tar.gz.sha256 | cut -d' ' -f1)"
url="https://www.apache.org/dyn/closer.lua?path=datafusion/datafusion-${version}/apache-datafusion-${version}.tar.gz"
sha256="$(curl https://dist.apache.org/repos/dist/release/datafusion/datafusion-${version}/apache-datafusion-${version}.tar.gz.sha256 | cut -d' ' -f1)"

pushd "$(brew --repository homebrew/core)"

Expand All @@ -52,7 +52,7 @@ fi
echo "Updating working copy"
git fetch --all --prune --tags --force -j$num_processing_units

branch=apache-arrow-datafusion-${version}
branch=apache-datafusion-${version}
echo "Creating branch: ${branch}"
git branch -D ${branch} || :
git checkout -b ${branch} origin/master
Expand Down
78 changes: 0 additions & 78 deletions dev/release/rat_exclude_files.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,84 +15,8 @@ ci/etc/*.patch
ci/vcpkg/*.patch
CHANGELOG.md
datafusion/CHANGELOG.md
python/CHANGELOG.md
conbench/benchmarks.json
conbench/requirements.txt
conbench/requirements-test.txt
conbench/.flake8
conbench/.isort.cfg
dev/requirements*.txt
dev/archery/MANIFEST.in
dev/archery/requirements*.txt
dev/archery/archery/tests/fixtures/*
dev/archery/archery/crossbow/tests/fixtures/*
dev/release/rat_exclude_files.txt
dev/tasks/homebrew-formulae/apache-arrow.rb
dev/tasks/linux-packages/apache-arrow-apt-source/debian/apache-arrow-apt-source.install
dev/tasks/linux-packages/apache-arrow-apt-source/debian/compat
dev/tasks/linux-packages/apache-arrow-apt-source/debian/control
dev/tasks/linux-packages/apache-arrow-apt-source/debian/rules
dev/tasks/linux-packages/apache-arrow-apt-source/debian/source/format
dev/tasks/linux-packages/apache-arrow/debian/compat
dev/tasks/linux-packages/apache-arrow/debian/control.in
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-cuda-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-arrow-dataset-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-gandiva-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-parquet-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/gir1.2-plasma-1.0.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libarrow-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-cuda400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-dataset400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-flight-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-flight400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-flight-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python-flight400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow-python400.install
dev/tasks/linux-packages/apache-arrow/debian/libarrow400.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libgandiva-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libgandiva400.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libparquet-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libparquet400.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-dev.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.doc-base
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib-doc.links
dev/tasks/linux-packages/apache-arrow/debian/libplasma-glib400.install
dev/tasks/linux-packages/apache-arrow/debian/libplasma400.install
dev/tasks/linux-packages/apache-arrow/debian/patches/series
dev/tasks/linux-packages/apache-arrow/debian/plasma-store-server.install
dev/tasks/linux-packages/apache-arrow/debian/rules
dev/tasks/linux-packages/apache-arrow/debian/source/format
dev/tasks/linux-packages/apache-arrow/debian/watch
dev/tasks/requirements*.txt
dev/tasks/conda-recipes/*
pax_global_header
MANIFEST.in
__init__.pxd
Expand All @@ -109,8 +33,6 @@ requirements.txt
.gitattributes
rust-toolchain
benchmarks/queries/q*.sql
python/rust-toolchain
python/requirements*.txt
**/testdata/*
benchmarks/queries/*
benchmarks/expected-plans/*
Expand Down
Loading

0 comments on commit 44f1147

Please sign in to comment.