Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simplify, explicit lock is not necessary in this design #8

Merged
merged 3 commits into from
Oct 11, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
name: Tests and Coverage

on:
push:
branches: ['testing']
pull_request:
branches: ['main', 'master', 'testing']
branches: ['main', 'master']
workflow_dispatch:

jobs:
tests:
name: Python ${{ matrix.python-version }} on ${{ matrix.os }}
name: Python ${{ matrix.python_version }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
strategy:
matrix:
python-version:
python_version:
- '3.12'
- '3.11'
- '3.9'
@@ -22,61 +24,73 @@ jobs:
timeout-minutes: 10
steps:
- uses: actions/checkout@v4

- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
python-version: ${{ matrix.python_version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade coverage

- name: Lint
run: |
python -m pip install --upgrade bandit
python -m bandit --recursive awaiter
python -m pip install --upgrade flake8
python -m flake8 .
if: ${{ matrix.os == 'ubuntu-20.04' && matrix.python-version == '3.12' }}
if: ${{ matrix.os == 'ubuntu-20.04' && matrix.python_version == '3.12' }}

- name: Run tests
run: python tests.py
if: ${{ matrix.python-version != '3.12' }}
if: ${{ matrix.python_version != '3.12' }}

- name: Run tests with coverage
run: |
python -m coverage run tests.py
mkdir artifact && mv .coverage artifact/.coverage.${{ matrix.os }}
if: ${{ matrix.python-version == '3.12' && !startsWith(matrix.os, 'windows-') }}
if: ${{ matrix.python_version == '3.12' && !startsWith(matrix.os, 'windows-') }}

- name: Run tests with coverage on Windows
run: |
python -m coverage run tests.py
mkdir artifact && move .coverage artifact\.coverage.windows
shell: cmd
if: ${{ matrix.python-version == '3.12' && startsWith(matrix.os, 'windows-') }}
if: ${{ matrix.python_version == '3.12' && startsWith(matrix.os, 'windows-') }}

- uses: actions/upload-artifact@v3
with:
path: artifact
if: ${{ matrix.python-version == '3.12' }}
include-hidden-files: true
if: ${{ matrix.python_version == '3.12' }}
report:
name: Upload Coverage to Codecov & SonarCloud Scan
needs: ['tests']
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Install dependencies
run: |
python -m venv --system-site-packages .local
echo "$HOME/.local/bin" >> $GITHUB_PATH
python -m pip install --upgrade pip
python -m pip install --upgrade coverage
- uses: actions/download-artifact@v3

- name: Combine and view report
run: |
python -m coverage combine artifact
python -m coverage report --show-missing --skip-covered
python -m coverage xml

- uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
- uses: sonarsource/sonarcloud-github-action@master

- uses: sonarsource/sonarcloud-github-action@v3
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # needed to get PR information, if any
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
46 changes: 19 additions & 27 deletions awaiter/__init__.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Copyright (c) 2024 nggit

__version__ = '0.1.1'
__version__ = '0.1.2'
__all__ = ('ThreadExecutor', 'MultiThreadExecutor')

import asyncio # noqa: E402

from functools import wraps # noqa: E402
from inspect import isgeneratorfunction # noqa: E402
from queue import SimpleQueue # noqa: E402
from threading import Thread, current_thread, Lock # noqa: E402
from threading import Thread, current_thread # noqa: E402


def set_result(fut, result):
@@ -122,19 +122,17 @@ def shutdown(self):


class MultiThreadExecutor(ThreadExecutor):
def __init__(self, size=10, loop=None, name='MultiThreadExecutor'):
def __init__(self, size=5, loop=None, name='MultiThreadExecutor'):
super().__init__(loop=loop, name=name)

self.size = size
self._threads = {}
self._delete_lock = Lock()
self._shutdown = None

def is_alive(self):
with self._delete_lock:
for thread in self._threads.values():
if thread.is_alive():
return True
for thread in self._threads.values():
if thread.is_alive():
return True

return False

@@ -149,30 +147,24 @@ def run(self):
super().run()
self.size -= 1
finally:
with self._delete_lock:
if current_thread().name in self._threads:
del self._threads[current_thread().name]

if self.is_alive():
# exited normally. signal the next thread to stop as well
self.queue.put_nowait((None, None, None, None))
else:
self.loop.call_soon_threadsafe(current_thread().join)
self.loop.call_soon_threadsafe(set_result, self._shutdown, None)

if current_thread().name in self._threads:
del self._threads[current_thread().name]

# exited normally. signal the next thread to stop as well
self.loop.call_soon_threadsafe(self.shutdown)

def submit(self, *args, **kwargs):
fut = super().submit(*args, **kwargs)
num = len(self._threads)

with self._delete_lock:
num = len(self._threads)

if num < self.size:
thread = Thread(
target=self.run,
name=f'{self.name}.{num}.{self.loop.time()}'
)
thread.start()
self._threads[thread.name] = thread
if num < self.size:
thread = Thread(
target=self.run, name=f'{self.name}.{num}.{self.loop.time()}'
)
thread.start()
self._threads[thread.name] = thread

return fut

Loading