Enable duckdb option partitioned_write_flush_threshold=1024 to reduce memory

[Taepper]

resolves spurious OOM crashes when preprocessing large datasets

Summary

We now set the flag partitioned_write_flush_threshold to 1024, whereas its default value is 524288. Initial tests did not indicate a big slowdown of the preprocessing times.

We found this flag in a discussion of a similar issue here.

PR Checklist

- [ ] All necessary documentation has been adapted or there is an issue to do so.
- [ ] The implemented feature is covered by an appropriate test.

[github-actions]

This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog.

Note that this might contain changes that are on main, but not yet released.

Changelog:

0.2.20 (2024-09-20)

Features

• make duckdb memory limit configurable via preprocessing config (c112eb1)

Bug Fixes

• preprocessing: resolves spurious OOM crashes when handling large datasets (79658d5)

[fengelniederhammer]

Please adapt the commit message (with an eye on the changelog).

And maybe also put the link to that GitHub comment in the commit description? That would probably be valuable for someone trying to figure out in a year from now in the commit history, why we set that value.