MyNintendo Scraper

A webscraper that checks My Nintendo rewards store every hour for changes.
Explore the docs »

View App · Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. Contact
7. Acknowledgments

About The Project

The motivation for this project was automate the process of checking My Nintendo rewards store for new items to reduce the chance of missing out on new listings.

Built With

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• BeautifulSoup
• Fly.io
• Axios
• HeroIcons

(back to top)

Getting Started

To get a local copy up and running follow these simple example steps.

Installation

1. Development Environment Setup

   • You can skip these steps below if you don't plan on submitting changes or features to the repository

   • npm

     • In the app/ directory, install the required npm packages:

     npm install

   • pip

     • In the server/ directory, install the required python packages:

     source venv/bin/activate
     pip3 install -r requirements.txt

2. Discord Bot messaging

   • This application utilizes a Discord Bot to send an automated message when changes are detected on the store front, which has private keys that cannot be shared. If one wants to replicate this in their copy of this repository, one can start here.

3. Enter your secrets in a .env file

   DATABASE_URL=YOUR_DATABASE_URL
   WEBHOOK_ID=YOUR_WEBHOOK_ID_FROM_DISCORD
   WEBHOOK_TOKEN=YOUR_WEBHOOK_TOKEN_FROM_DISCORD
   DISCORD_USER_ID=YOUR_DISCORD_USER_ID

4. Starting the application

    (in the app directory) npm start
    (in the server directory with venv activated) flask run

(back to top)

Usage

Scraping

Upon accessing the web app, a useEffect hook is ran which scrapes the My Nintendo rewards page for its listings. Then the API checks to see if there are any changes between the items from the most recent scrape result and the previous. If there is a change then:

1. The web app will display the changes and store the difference as a new record in the changes table of the database.
2. Then it'll send a message via the Discord bot to notify those who are connected to the bot about the changes

Regardless of a change, the most recently scraped result will be added as a new record in the listings table of the database.

Automated Scraping

The whole motivation for creating this web app is that the web scraping is automated. To achieve this, hourly requests are made to the API via cron-job.org.

Automated Clean Up

Since so much data is being stored in the database, a weekly cron job request is made to delete records that have been saved in the database for a week or more. This is handled by keeping track of the timestamps when a listing record is stored and having an additional column, expiration, for that record of seven days after its insertion timestamp. Then the request just queries based on the expiration column to see if that value is a date older than the time of the deletion API request.

Roadmap

Completed Features

• Start a CI/CD Pipeline
  • Utilize Github Actions to deploy on main branch merge to Fly.io
• Scrape My Nintendo rewards page
  • Get current item listings
• Display any changes to My Nintendo rewards listings
  • Display what has changed
  • Display timestamp of when change occurred
• Display current changes if any
• Utilize a useEffect to scrape as soon as web app is accessed
  • Include a timestamp to show when scrape occurred at time of web app loading
• Improve frontend visuals so it isn't so plain
  • Added Mario inspired background animation
  • Include the image of items currently listed on MyNintendo

Features To Implement

• [] Add tests
  • [] Test routes
  • [] Test models
• [] Distribute this app for public use
• [] An account system
• [] Refine Discord Bot
  • [] Add an option/method to add users to be mentioned when a change is detected
  • [] Add an option to choose which types of changes to be notified for
  • [] Add a command to call the scrape functions

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the repository a star! Thanks again!

1. Fork the repository
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Contact

Project Link: https://github.com/samau3/mynintendo-scraper

(back to top)

Acknowledgments

• Thanks to FreeCodeCamp for the BeautifulSoup Webscraping tutorial that helped get this project started
• Thanks to Hyperplexed for the Mario UI background idea, check out his tutorial!

(back to top)