Skip to content

Commit

Permalink
v0.4.0 (#137)
Browse files Browse the repository at this point in the history
* Change README logo if in dark mode (#95)

* Start transform tests & minor `RandomTimeShift` optimization (#94)

* Start transform tests & slight optimization in RandomTimeShift

* Add new length check for TimeCrop tests

* Code style

* Fixed name mangling.

* QAM/PSK Pulse shaping filter transition bandwidth corrected (#98)

* excess bandwidth is defined in porportion to signal bandwidth, not sampling rate, thus needs to be scaled by the samples per symbol

* filling in a comment to describe modification to code

* QAM/PSK Pulse shaping filter transition bandwidth corrected (#98)

* excess bandwidth is defined in porportion to signal bandwidth, not sampling rate, thus needs to be scaled by the samples per symbol

* filling in a comment to describe modification to code

* OFDM Modulator filter lengths estimated and bandwidth randomized (#99)

* * cutoff frequency for LPF now randomized when using 'rand_lpf'
* derives a transition bandwidth from the cutoff frequency
* uses filter length approximating function for the randomized LPF

* using filter estimation function for pre-computed LPF taps

* Tests for visual inspection. (#103)

* 91 create generation performance benchmarks for each modulation type (#104)

* Initial benchmarking code.

* Some benchmarks

* Adding initial benchmarks.

* Fix action.

---------

* 75 examine ofdm generation for potential speedups for sig53 (#105)

* Tests for visual inspection of modulation generation. (#102)

* Optimizations show significant improvement in generation speed.

* Nominal behavior after using scipy.

* Adding initial Dockerfile (#108)

* Incrementing version

* Fix float issue (#111)

* Initial draft of restructuring transforms (#106)

* Flatten transforms to mirror torchvision/audio structure & add reprs

* Address flake8 errors

* Update transform imports with restructuring

* Formatting.

---------

* Fixing broken examples, formatting files, etc.

* Adjustments for generation speed

* Wideband generation working.

* 45 consider compatibility with torch 20 (#115)

* Seems to work

* Workflow is broken

* Python 3.7 not supported for Torch 2.0

* Adjusting test workflow

* Extracted out a method for filter design. Put DSP-only things in utils/dsp (#116)

* Benchmark and visualize wideband dataset generation. (#118)

* Extracted out a method for filter design. Put DSP-only things in utils/dsp

* New tests.

* Migrated to pytest. (#119)

* Added model instantiation tests for narrowband signals. (#120)

* 85 gmskgfsk also needs faster filtering due to convolution of long signals with gaussian pulse shape (#121)

* Change all references to convolution to scipy-based convolutions.

* Missed one.

* Add mypy workflow check and fix all mypy-found bugs (#123)

* Fix mypy in target_transforms

* Add mypy workflow for static type checking

* Fix typo in mypy.yml workflow

* Fix mypy in torchsig/transforms/functional.py

* Fix mypy in utils/types.py

* Fix mypy in torchsig/utils/

* Fix mypy in torchsig/transforms/

* Fix mypy in torchsig/datasets/

* Fix mypy in torchsig/models/

* Format with pyfmt

* Fix isinstance(x, Callable)

* Add PR template

* Adjusting module (#130)

* 131 configure package for pypi release (#132)

* Adjusting installation

* More information on pyproject.toml

* More information on pyproject.toml

* I guess some keywords in current documentation do not work :/.

* Ignore distributable artifacts

* More documentation and examples (#134)

* More documentation and examples

* Added script to train

* Remove duplicate script

* 126 create dockerfile and script for generating all versions of widebandsig53 (#135)

* Added generation script.

* Adding scripts

* Fixing mypy issue.

* 127 create test suite for visually validating transforms (#136)

* Some transforms here

* Most transforms included

---------

Co-authored-by: lboegner <luke.boegner@gmail.com>
Co-authored-by: Garrett Vanhoy <gmvanhoy@gmail.com>
Co-authored-by: MattCarrickPL <120057274+MattCarrickPL@users.noreply.github.com>
  • Loading branch information
3 people authored Jun 13, 2023
1 parent f6edab8 commit c9b1193
Show file tree
Hide file tree
Showing 77 changed files with 9,034 additions and 8,521 deletions.
24 changes: 24 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
## Summary

Describe your changes here specifying if the change is a bug fix, enhancement, new feature, etc.

## Test Plan

Describe how you tested and verified your changes here (changes captured in existing tests, built and ran new tests, etc.).

## Before Submitting
- [ ] Check mypy locally
- `pip3 install mypy==1.2.0`
- `mypy --ignore-missing-imports torchsig`
- Address any error messages
- [ ] Lint check locally
- `pip3 install flake8`
- `flake8 --select=E9,F63,F7,F82 torchsig`
- Address any error messages
- [ ] Run formatter if needed
- `pip3 install git+https://github.com/GooeeIOT/pyfmt.git`
- `pyfmt torchsig`
- [ ] Run test suite locally
- `pytest --ignore-glob=*_figures.py --ignore-glob=*_benchmark.py`
- Ensure tests are successful prior to submitting PR

30 changes: 30 additions & 0 deletions .github/workflows/mypy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
name: mypy

on: [push]

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.10"]

steps:
- uses: actions/checkout@v3
- name: Install Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- uses: actions/cache@v3
with:
path: ${{ env.pythonLocation }}
key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
- name: Build package
run: |
python -m pip install --upgrade pip
python -m pip install .
- name: Static type check with mypy
run: |
pip install mypy==1.2.0
mypy --ignore-missing-imports torchsig
3 changes: 2 additions & 1 deletion .github/workflows/pip_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.7", "3.10"]
python-version: ["3.8", "3.10"]

steps:
- uses: actions/checkout@v3
Expand All @@ -21,4 +21,5 @@ jobs:
key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
- name: Build Package
run: |
python -m pip install --upgrade pip
python -m pip install .
3 changes: 2 additions & 1 deletion .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.7", "3.10"]
python-version: ["3.8", "3.10"]

steps:
- uses: actions/checkout@v3
Expand All @@ -21,6 +21,7 @@ jobs:
key: ${{ env.pythonLocation }}-${{ hashFiles('pyproject.toml') }}
- name: Build package
run: |
python -m pip install --upgrade pip
python -m pip install .
- name: Lint with flake8
run: |
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,4 @@ lightning_logs/
*.pt
*.jpg
*.benchmarks/
dist/
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime
FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime

ENV DEBIAN_FRONTEND=noninteractive

Expand Down
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,35 @@ cd torchsig
pip install .
```

## Generating the Datasets
If you'd like to generate the named datasets without messing with your current Python environment, you can build the development container and use it to generate data at the location of your choosing.

```
docker build -t torchsig -f Dockerfile .
docker run -u $(id -u ${USER}):$(id -g ${USER}) -v `pwd`:/workspace/code/torchsig torchsig python3 torchsig/scripts/generate_sig53.py --root=/workspace/code/torchsig/data --all=True
```

For the wideband dataset, you can do:

```
docker build -t torchsig -f Dockerfile .
docker run -u $(id -u ${USER}):$(id -g ${USER}) -v `pwd`:/workspace/code/torchsig torchsig python3 torchsig/scripts/generate_wideband_sig53.py --root=/workspace/code/torchsig/data --all=True
```

If you do not need to use Docker, you can also just generate using the regular command-line interface

```
python3 torchsig/scripts/generate_sig53.py --root=torchsig/data --all=True
```

or for the wideband dataset:

```
python3 torchsig/scripts/generate_wideband_sig53.py --root=torchsig/data --all=True
```

Then, be sure to point scripts looking for ```root``` to ```torchsig/data```.

## Using the Dockerfile
If you have Docker installed along with compatible GPUs and drivers, you can try:

Expand Down
1 change: 1 addition & 0 deletions examples/00_example_sig53_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ def __len__(self) -> int:

data_loader = DataLoader(dataset=plot_dataset, batch_size=16, shuffle=True)


# Transform the plotting titles from the class index to the name
def target_idx_to_name(tensor: np.ndarray) -> list:
batch_size = tensor.shape[0]
Expand Down
2 changes: 1 addition & 1 deletion examples/01_example_modulations_dataset.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ def __len__(self) -> int:
# ### Save Data to LMDB
# As a final exercise for this example notebook, the dataset can be saved to an LMDB static dataset for offline use. Note this is similar to how the static Sig53 dataset is generated and saved to serve as a static performance evaluation dataset.

env = lmdb.open("dataset", max_dbs=3 if include_snr else 2, map_size=int(1e12))
env = lmdb.open("examples/dataset", max_dbs=3 if include_snr else 2, map_size=int(1e12))

iq_sample_db = env.open_db("iq_samples".encode())
modulations_db = env.open_db("modulations".encode())
Expand Down
20 changes: 11 additions & 9 deletions examples/02_example_sig53_classifier.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,19 @@
# ### Import Libraries
# First, import all the necessary public libraries as well as a few classes from the `torchsig` toolkit. An additional import from the `cm_plotter.py` helper script is also done here to retrieve a function to streamline plotting of confusion matrices.

from torchsig.transforms.target_transforms.target_transforms import DescToClassIndex
from torchsig.transforms.target_transforms import DescToClassIndex
from torchsig.models.iq_models.efficientnet.efficientnet import efficientnet_b4
from torchsig.utils.writer import DatasetCreator
from torchsig.transforms.wireless_channel.wce import RandomPhaseShift
from torchsig.transforms.signal_processing.sp import Normalize
from torchsig.transforms.expert_feature.eft import ComplexTo2D
from torchsig.transforms.transforms import Compose
from torchsig.transforms.transforms import (
RandomPhaseShift,
Normalize,
ComplexTo2D,
Compose,
)
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning import LightningModule, Trainer
from sklearn.metrics import classification_report
from cm_plotter import plot_confusion_matrix
from torchsig.utils.cm_plotter import plot_confusion_matrix
from torchsig.datasets.sig53 import Sig53
from torchsig.datasets.modulations import ModulationsDataset
from torch.utils.data import DataLoader
Expand Down Expand Up @@ -112,17 +114,17 @@
# ### Format Dataset for Training
# Next, the datasets are then wrapped as `DataLoaders` to prepare for training.

# Create dataloaders
# Create dataloaders"data
train_dataloader = DataLoader(
dataset=sig53_clean_train,
batch_size=16,
batch_size=8,
num_workers=8,
shuffle=True,
drop_last=True,
)
val_dataloader = DataLoader(
dataset=sig53_clean_val,
batch_size=16,
batch_size=8,
num_workers=8,
shuffle=False,
drop_last=True,
Expand Down
165 changes: 116 additions & 49 deletions examples/03_example_widebandsig53_dataset.ipynb

Large diffs are not rendered by default.

13 changes: 8 additions & 5 deletions examples/04_example_wideband_modulations_dataset.ipynb
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -11,6 +12,7 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -24,17 +26,16 @@
"outputs": [],
"source": [
"from torchsig.utils.visualize import MaskClassVisualizer, mask_class_to_outline, complex_spectrogram_to_magnitude\n",
"from torchsig.transforms.target_transforms.target_transforms import DescToMaskClass\n",
"from torchsig.transforms.target_transforms import DescToMaskClass\n",
"from torchsig.datasets.wideband import WidebandModulationsDataset\n",
"from torchsig.transforms.expert_feature.eft import Spectrogram\n",
"from torchsig.transforms.signal_processing.sp import Normalize\n",
"from torchsig.transforms.transforms import Compose\n",
"from torchsig.transforms.transforms import Spectrogram, Normalize, Compose\n",
"from torch.utils.data import DataLoader\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down Expand Up @@ -74,6 +75,7 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"tags": []
Expand Down Expand Up @@ -114,6 +116,7 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {
"tags": []
Expand Down Expand Up @@ -186,7 +189,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.0"
"version": "3.8.10"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit c9b1193

Please sign in to comment.