Skip to content

Latest commit

 

History

History
55 lines (38 loc) · 2.78 KB

README.md

File metadata and controls

55 lines (38 loc) · 2.78 KB
Raw

2-step verification for verifying a website's contents before scraping

This repository contains a simple 2-step verification process for verifying a website's contents before scraping.

Functionality

  1. User Authentication: Users can register and login to the website, using Firebase Authentication.
  2. Website verification: Users will be displayed an IFRAME of a random website for verification.
  3. Feedback: Users can provide feedback on the website's contents, including a "accept" or "reject" tab where they categorize the website into a categoary such as "news", "blog", "e-commerce", etc. or provide a reason for rejection.The feedback is stored in a Firebase Firestore database along with the metadata.

Pre-requisites

  1. Google Cloud Account: The dashboard uses Google Cloud Firestore to store feedback data. You will need to have a Google Cloud account and a Firestore database to run the dashboard.
  2. Firebase Project: The dashboard uses Firebase authentication to authenticate users. You will need to have a Firebase project and a service account key to run the dashboard.
  3. Service Account Key: The dashboard uses a service account key to authenticate with Google Cloud Firestore. You will need to have a service account key to run the dashboard.
  4. Google Cloud SDK: If you want to deploy the dashboard to the Google Cloud App Engine, you will need to have the Google Cloud SDK installed on your machine.
  5. Anaconda : The dashboard is built using the Streamlit framework. You will need to have Anaconda installed on your machine to run the dashboard.
  6. Docker: The dashboard can be run in a Docker container. You will need to have Docker installed on your machine to run the dashboard in a container.

Setup

To run the application, you will need to have a Firebase service account key. This key should be stored in the root directory as serviceAccountKey.json.

docker build -t <target-name> . 
docker run <target-name>

If you want to deploy to the Google cloud App engine, install the gcloud cli and run the following command:

gcloud app deploy

Akter to run the app locally, run the following command:

conda create -n url_verification python=3.9
conda activate url_verification

pip install -r requirements.txt
streamlit run app.py

Getting DefaultCredentialsError ?

You need to set the environment variable GOOGLE_APPLICATION_CREDENTIALS to the path of the service account key. You can do this by running the following command:

export GOOGLE_APPLICATION_CREDENTIALS="path/to/serviceAccountKey.json"