Draft: Aiokafka kip 345 #13
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-streaming#294) * re-import orjson module for json serialization/deserialization * fix test Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com>
* fix typos in README.md * fix other typos in README.md
…ng#312 (faust-streaming#313) * Faust commits the wrong offset in case of a gap in acks faust-streaming#312 Faust commits the wrong offset in case of a gap in acks faust-streaming#312 Co-Authored-By: ekerstens <49325583+ekerstens@users.noreply.github.com> * kljlk jhkjhj * Revert "kljlk" This reverts commit 8b487fc. * update fix faust-streaming#313 Co-Authored-By: ekerstens <49325583+ekerstens@users.noreply.github.com> * update test case Co-Authored-By: ekerstens <49325583+ekerstens@users.noreply.github.com> Co-Authored-By: Leo <47164895+LeohuangLeo@users.noreply.github.com> Co-authored-by: Leo Huang <s3563602123@gmail.com> Co-authored-by: ekerstens <49325583+ekerstens@users.noreply.github.com> Co-authored-by: Leo <47164895+LeohuangLeo@users.noreply.github.com>
faust-streaming#289) * fix: pessimistically lookup the earliest key The earliest_offset function in the consumer seems to not always return all tps which have been passed in but presumabely only the ones it has data for. After longer runtime it seems that the response MAY return a dict not filled with all topic partitions which have been added. To bypass this and not lead faust to crash, use the .get function on the returned mapping and default to None on return. * chore: flake8 fixes Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com>
* Fix flake warning with bound loop var * fix variable names
faust-streaming#325) * If changelog topic partitions are only listed in the actives_tps set and not in standbys, then those partitions need to be resumed to ensure GlobalTable stays in sync for all workers * fix recovery linting * pass flake8 test Co-authored-by: lmetzger <lorinmetzger@gmail.com> Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com>
Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com>
* Fixed filter not acking filtered out messages. * Removed debug print from test. * Added Cython implementation for the filter fix. Co-authored-by: Matthew Drago <matthew.drago@gig.com> Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com> Co-authored-by: Taybin Rutkin <taybin@users.noreply.github.com>
* create method for backing up partition * if we're not flushing, just directly read the db * annotate backup method * Define backup_partition in StoreT baseclass and derivatives * change partition to tp * change partition to union tp or int since all we care about is partition index * fix error log * add method to restore backups * add forgotten ellipses * remove misleading docstring * Check if backup path is directory and make paths * Convert partition paths used in restoration to str * dedicate backup path by tablename * update backup docstring * dont import BackupEngine to fix linting * commit lint changes * reformat docstrings * add general Exception * add backup_partition and restore_backup to MyStore test class * add backup_partition and restore_backup to MySerializedStore test class * check permissions to create dirs and write to backup dir before spawning backupengine * remove redundant exception handle * add backup methods to ChangeloggedObjectManager Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com>
* Remove a broken link in Readme.md The link for open file limit was dead. The solution is elaborated in a short, compact way. * Reformat code * Fix formatting * Move noqa note to propagate * Fix formatting in streams.py Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
Correct pypi link, tar filename and path
* commit inital fix for faust-streaming#333 * manually lint comment * allow multiple partitions to resume * minor cleanup
Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
…ng workers (faust-streaming#331) * allow 'read_only' to be passed through options * only apply read_only if the file doesn't already exist * pass read_only kwarg to test to fix test bug * call options.pop for brevity * add note to docs for read_only option
* add dist.yml for uploading sdists to pypi * update dist workflow for macos
…reaming#282) * adding intervaltree to manage gaps in topics to prevent OOM * adding intervaltree to manage gaps in topics to prevent OOM * remove old consumer test TPs Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
* build wheels for more platforms * enforce 64-bit only builds
Co-authored-by: Vikram Patki <54442035+patkivikram@users.noreply.github.com> Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
* point mode to faust screaming fork * update other links
…ssage (faust-streaming#342) * Address topics.py error in faust-streaming#175 * manually fix linting
* add noqa: B024 to base classes to pass lint check * append noqa to type ignore
Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
…ming#145) * rename default.txt requirements to requirements.txt in Makefile * fix docs.txt and dist.txt requirements files * add pre-commit support with flake8 back Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
* cleanup requirements and references to authors and changelog * update sphinx requirements * disable flake8 to minimize inconvenience for devleopers for now * change Signature import * update AUTHORS.md * use m2r2 and mdinclude * upgrade docs for new version of sphinx * create test branch for gh-pages * disable make build call * fix pages branch * disable building * correct inputs * specify inputs properly * add six to dependencies * change branch * add requirements.txt to docs.txt * try another approach * add test branch * update token permissions * add persist-credentials * update action versions * never mind * enable GITHUB TOKEN * enable contents:write * force for now * force for now * try deploy-pages * reorganize * remove sphinx-notes * fix path to Documentation * cleanup * revert title reformat * update all references from fauststreaming/latest to faust-stream.github.io * fix typos to faust-streaming * fix travis links * update release procedure on contributing.rst
Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
* Fix races in conductor with dynamic agents * Add documentation for `_resubscribe_sleep_lock_seconds` --------- Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
…etadata definitions into pyproject.toml (faust-streaming#574) * Consolidate dynamic attributes for setuptools v69.0.0 * commit functioning build * forgot to lint setup.py lol * Update authors and maintainers metadata * remove redundant name metadata definition * Revert "remove redundant name metadata definition" This reverts commit 0263fa6. * bring back long_description in setup.py
* Update setup.py to use setuptools_scm * Update setup.py
* add fastapi dir and change dep to rocksdict * commit inital example * update example * use lifespan * reorganize to have an endpoint trigger a producer * reorganize timer to be above fast api decorator
Bumps [actions/deploy-pages](https://github.com/actions/deploy-pages) from 2 to 3. - [Release notes](https://github.com/actions/deploy-pages/releases) - [Commits](actions/deploy-pages@v2...v3) --- updated-dependencies: - dependency-name: actions/deploy-pages dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Rename kafka to aiokafka kafka-python is no longer to be a dependency of aiokafka Following aio-libs/aiokafka#915 Signed-off-by: Anton Melser <anton.melser@outlook.com> * Force aiokafka version >=0.9.0 Signed-off-by: Anton Melser <anton.melser@outlook.com> --------- Signed-off-by: Anton Melser <anton.melser@outlook.com> Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
…aming#586) * Replace called_once_with test_objects.py for 3.12 support * just replace with assert_called_once
* Add official support for Python 3.12 * commit remaining lint changes
… /requirements (faust-streaming#583) * Update aiokafka requirement in /requirements Updates the requirements on [aiokafka](https://github.com/aio-libs/aiokafka) to permit the latest version. - [Release notes](https://github.com/aio-libs/aiokafka/releases) - [Changelog](https://github.com/aio-libs/aiokafka/blob/master/CHANGES.rst) - [Commits](aio-libs/aiokafka@v0.8.0...v0.9.0) --- updated-dependencies: - dependency-name: aiokafka dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update requirements.txt --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 3 to 4. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3 to 4. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: William Barnhart <williambbarnhart@gmail.com>
Bumps [actions/deploy-pages](https://github.com/actions/deploy-pages) from 3 to 4. - [Release notes](https://github.com/actions/deploy-pages/releases) - [Commits](actions/deploy-pages@v3...v4) --- updated-dependencies: - dependency-name: actions/deploy-pages dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [actions/upload-pages-artifact](https://github.com/actions/upload-pages-artifact) from 2 to 3. - [Release notes](https://github.com/actions/upload-pages-artifact/releases) - [Commits](actions/upload-pages-artifact@v2...v3) --- updated-dependencies: - dependency-name: actions/upload-pages-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update agent.py to call loop.create_task rather than asyncio.Task Credit goes to @samtx for finding this solution for faust-streaming#175 (comment) * Update agent.py Co-authored-by: Sam Friedman <samtx@outlook.com> * Fix linting in agent.py * Update unit test to verify slurp is awaited * Update unit test to check awaitable task creation * Linting and formatting --------- Co-authored-by: William Barnhart <williambbarnhart@gmail.com> Co-authored-by: Sam Friedman <sam.friedman@he360.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On 7.12.23 the merge request regarding static-partition-assignem (aio-libs/aiokafka#941) was merged to master by aio-kafka. This allows us to remove our aio-kafka-1 fork.
This PR also contain the newest version of faust-streaming (09.01.24) 0.10.22