URS v3.3.1 | Bug Fixes/Refactoring, a New Utility, and GitHub Actions

[JosephLai241]

Overview

Summary

URS v3.3.1's primary focus is refactoring/fixing open issues. This release also includes a new utility that I thought was fun and useful to implement.

Additionally this is a reintroduction to GitHub Actions, which replaces Travis-CI since it is unfortunately no longer free. URS is now tested on all operating systems (see the new GitHub Actions YAML in the Test Configuration section).

The New Utility

Here is the new flag:

[-t [<optional_date>]]

When used without the optional date, this flag will display a visual tree of the current date's scrape directory. This makes it quick and easy to see what scrapes you have recorded for today and is an alternative to the tree command found on Linux distributions. You can also include a date following the -t flag, which will display the visual tree for the specified date.

Refactoring

The previous codebase utilized Python's old-school method of string formatting. Here is an example of the previous string formatting that was implemented:

some_string = "string"
a_string = "A %s here" % some_string 

I went through every module to replace the outdated method with the superior f-string introduced in Python 3.6:

some_string = "string"
a_string = f"A {some_string} here"

There should not be issues with implementing f-strings since URS requires Python 3.7+ - this is because numpy, a dependency, dropped support for Python 3.6 after releasing v1.20.0 on January 30, 2021.

Demo GIFs for the Utilities section in the README will be added to the demo-gifs branch shortly after this pull request is merged.

Motivation/Context

Two issues are fixed with this pull request (see Issue Fix or Enhancement Request section).

I have been considering adding additional tools/scrape options to URS. Adding the new tree utility seemed like a good addition. I recently discovered Rich and immediately realized I need to take full advantage of this library - the next iteration will include yet another UI overhaul.

I noticed Travis-CI was not triggering builds as I was pushing to the dev branch. I looked into the issue and discovered their business structure has changed - there is now a cap for the number of free builds you can run. I do not believe I would run out of free builds anytime soon if it continued to be the primary CI provider for this project. However I want to avoid the looming anxiety of counting the number of builds I use every time I push to this repository, so I am re-activating GitHub Actions. It was previously disabled since Travis-CI provided the necessary CI features, which is simply running pytest. It is now enabled, the exclusive CI provider for this project, and tests URS on all platforms (previously only tested on Ubuntu, the Linux distribution provided by most CI solutions).

New Dependencies

commonmark==0.9.1
Pygments==2.9.0
rich==10.4.0

Issue Fix or Enhancement Request

• Fixes Bug Report | Scrape data is not located within the scrapes directory when generating frequencies #40.
  • Bug: Analytical tools do not work when attempting to run them on Windows machines.
    • This error was raised because Windows handles file paths differently compared to Unix-based operating systems. The previous code assumes the user is running URS on a Unix-based machine.
  • Fix: Refactored GetPath.get_scrape_type().
    • This method now uses a platform-agnostic algorithm to determine whether the file is eligible for analysis.
• Fixes Bug Report | IndexError raised in test_Export.py when running pytest on Windows machine #30.
  • Bug: IndexError raised in test_Export.py when running pytest on Windows machine.
    • This error was raised because CSV files are written differently on Windows machines.
  • Fix: Refactored two tests in test_Export.py:
    • These tests now account for the CSV format difference between platforms.

Type of Change

• Bug Fix (non-breaking change which fixes an issue)
• Code Refactor
• New Feature (non-breaking change which adds functionality)
• This change requires a documentation update

Breaking Change

N/A

List All Changes That Have Been Made

Added

• User interface
  • Added a new utility:
    • -t/--tree - display the directory structure of the current date directory. Or optionally include a date to display that day's scrape directory.
• Source code
  • Added a new file Utilities.py to the urs/utils module.
    • Added a class DateTree which contains methods to find and build a visual tree for the target date's directory.
      • Added logging when this utility is run.
  • Added an additional Halo to the wordcloud generator.
• README
  • Added new "Utilities" section.
    • This section describes how to use the -t/--tree and --check utility flags.
  • Added new "Sponsors" section.
• Tests
  • Added test_Utilities.py under the test_utils module.

Changed

• Source code
  • Refactored the following methods within the analytics module:
    • GetPath.get_scrape_type()
    • GetPath.name_file()
    • FinalizeWordcloud().save_wordcloud()
      • Implemented pathlib's Path() method to get the path.
  • Upgraded all string formatting from old-school Python formatting (using the % operator) to the superior f-string.
  • Updated GitHub Actions workflow pytest.yml.
    • This workflow was previously disabled. The workflow has been upgraded to test URS on all platforms (ubuntu-latest, macOS-latest, and windows-latest) and to send test coverage to Codecov after testing completes on ubuntu-latest.
• README
  • Changed the Travis-CI badge to a GitHub Actions badge.
    • Updated badge link to route to the workflows page within the repository.
• Tests
  • Upgraded all string formatting from old-school Python formatting (using the % operator) to the superior f-string in the following modules:
    • test_utils/test_Export.py
    • test_praw_scrapers/test_live_scrapers/test_Livestream.py
  • Refactored two tests within test_Export.py:
    • TestExportWriteCSVAndWriteJSON().test_write_csv()
    • TestExportExportMethod().test_export_write_csv()
• Community documents
  • Updated PULL_REQUEST_TEMPLATE.md.
    • Removed Travis-CI configuration block.
    • Added GitHub Actions pull request badge.

Deprecated

• Source code
  • Removed .travis.yml - URS no longer uses Travis-CI as its CI provider.

How Has This Been Tested?

• Ran pytest on local machine - all tests passed
• Passed all tests in Pytest workflow.

Test Configuration

• Python version: 3.9.5

New GitHub Actions configuration:

name: Pytest

on:
  push:
    branches-ignore:
      - 'demo-gifs'
      - 'samples'
  pull_request:
    branches-ignore:
      - 'demo-gifs'
      - 'samples'

jobs:
  pytest:
    runs-on: ${{ matrix.os }}
    
    strategy:
      matrix:
        os: [ubuntu-latest, macOS-latest, windows-latest]

    steps:
      - uses: actions/checkout@v2

      - name: Set up Python 3.9
        uses: actions/setup-python@v2
        with:
          python-version: 3.9
        
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install pytest
          pip install -r requirements.txt
        
      - name: Install URS
        run: |
          pip install .
      
      - name: Run Pytest
        env:
          CLIENT_ID: ${{ secrets.CLIENT_ID }}
          CLIENT_SECRET: ${{ secrets.CLIENT_SECRET }}
          USER_AGENT: ${{ secrets.USER_AGENT }}
          REDDIT_USERNAME: ${{ secrets.REDDIT_USERNAME }}
          REDDIT_PASSWORD: ${{ secrets.REDDIT_PASSWORD }}
        run: |
          pytest --cov=./
            
      - name: Send coverage data to Codecov
        uses: codecov/codecov-action@v1
        if: matrix.os == 'ubuntu-latest'

Dependencies

astroid==2.5.6
attrs==21.2.0
certifi==2021.5.30
chardet==4.0.0
colorama==0.4.4
commonmark==0.9.1
coverage==5.5
cycler==0.10.0
halo==0.0.31
idna==2.10
iniconfig==1.1.1
isort==5.9.1
kiwisolver==1.3.1
lazy-object-proxy==1.6.0
log-symbols==0.0.14
matplotlib==3.4.2
mccabe==0.6.1
more-itertools==8.8.0
numpy==1.21.0
packaging==20.9
Pillow==8.2.0
pluggy==0.13.1
praw==7.3.0
prawcore==2.2.0
prettytable==2.1.0
py==1.10.0
Pygments==2.9.0
pylint==2.8.3
pyparsing==2.4.7
pytest==6.2.4
pytest-cov==2.12.1
python-dateutil==2.8.1
python-dotenv==0.18.0
requests==2.25.1
rich==10.4.0
six==1.16.0
spinners==0.0.24
termcolor==1.1.0
toml==0.10.2
update-checker==0.18.0
urllib3==1.26.6
wcwidth==0.2.5
websocket-client==1.1.0
wordcloud==1.8.1
wrapt==1.12.1

Checklist

• My code follows the style guidelines of this project.
• I have performed a self-review of my own code, including testing to ensure my fix is effective or that my feature works.
• My changes generate no new warnings.
• I have commented my code, providing a summary of the functionality of each method, particularly in areas that may be hard to understand.
• I have made corresponding changes to the documentation.
• I have performed a self-review of this Pull Request template, ensuring the Markdown file renders correctly.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 77.73852% with 63 lines in your changes missing coverage. Please review.

Project coverage is 85.82%. Comparing base (158efd6) to head (d1516ea).
Report is 262 commits behind head on master.

Files with missing lines	Patch %	Lines
urs/utils/Logger.py	22.72%	17 Missing ⚠️
tests/test_utils/test_Utilities.py	84.61%	10 Missing ⚠️
urs/analytics/Wordcloud.py	11.11%	8 Missing ⚠️
urs/utils/Utilities.py	93.97%	5 Missing ⚠️
tests/test_utils/test_Export.py	81.81%	4 Missing ⚠️
urs/praw_scrapers/live_scrapers/Livestream.py	72.72%	3 Missing ⚠️
urs/praw_scrapers/static_scrapers/Basic.py	0.00%	3 Missing ⚠️
urs/praw_scrapers/static_scrapers/Comments.py	0.00%	3 Missing ⚠️
urs/utils/Titles.py	50.00%	3 Missing ⚠️
urs/utils/Cli.py	75.00%	2 Missing ⚠️
... and 4 more

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
- Coverage   86.56%   85.82%   -0.74%     
==========================================
  Files          33       42       +9     
  Lines        3453     4028     +575     
==========================================
+ Hits         2989     3457     +468     
- Misses        464      571     +107     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.