NLP and Sentiment Analysis on Stock News and Price Monitoring

• Proposal link (CLAAT): https://codelabs-preview.appspot.com/?file_id=1x528Ez7oU1SrBOJBMSHLpZ-h7VmKvegiq2tCO8b5fQw#1

• FastAPI link (hosted on App Engine): https://big-data-final-project-347804.ue.r.appspot.com

• Streamlit Cloud link: https://share.streamlit.io/krishna-aditi/nlp-sentiment-analysis-on-stock-news-and-price-monitoring/main/src/data/streamlit-app.py

  • To run the project, click on the above Streamlit link and login with the following credentials
    • UserID: admin
    • Pwd: admin
  • The funtionality to fetch news on-demand takes one or two clicks, since the lambda functions take some time to start and might end up in a timeout error on first attempt.

• Summarization serverless lambda function endpoint link: https://j3btan6btg.execute-api.us-east-1.amazonaws.com/dev/summary

• Sentiment Analysis serverless lambda function endpoint link: https://n9h06pjw6f.execute-api.us-east-1.amazonaws.com/dev/sentiment

• Documentation link (Google Doc): https://docs.google.com/document/d/1AbperRxJHq3DNUMaQ18v6EQXjUE_nIiD1WQ-OZDkRds/edit?usp=sharing

• Documentation CLAAT link: https://codelabs-preview.appspot.com/?file_id=1AbperRxJHq3DNUMaQ18v6EQXjUE_nIiD1WQ-OZDkRds#0

  • The functional documentation in detail can be found on the above CLAAT Link

• Youtube Demo link: https://youtu.be/2-ybk5bAaL4

Overview

Stock market analysis is a difficult task to execute because of market volatility and a multitude of other dependent and independent factors that influence the market value of a particular stock. One of these factors - investor sentiment, is very much capable of influencing the stock price. News and social media are few of the ways to capture it. We aim to bring together Natural Language Processing and Sentiment Analysis of the stock related news to better understand the stock price trends.

Goals

Goal is to create a platform with curated information about a company's profile, stock specific news (from 24 hours), and the stock price trend from the past year. We focus on summarizing the news using NLP models, and labeling it as positive or negative using sentiment analysis. The platform aims to help the user make informed trading decisions.

1. Create a workflow to web-scrape company specific information using YFinance library in Python
2. Build a Streamlit interface for the FastAPI to render stock price trends, company’s financials, and description of the business
3. Create an Airflow DAG to do the following:
4. Web-scrape company specific stock news with the help of NewsAPI and Newspaper3K API
5. Call the Summarization API (Lambda function) for summarization on each news article’s content
6. Call Sentiment Analysis API (Lambda function) for extracting the sentiment of each news article
7. Summarized news, along with the associated sentiment will be rendered on the Streamlit

Main Requirements

To test pretrained models and train API requires

• Python 3.8
• pandas
• numpy
• Streamlit
• Python-jose
• Passlib
• Transformers 3.4
• Torch
• GCSFS
• Plotly
• Yfinance
• NewsAPI
• Newspaper3K

Use cases

1. Single hub for stock trends and associated news to help the investor to make informed decisions
2. Platform for learning about company stocks, or, stock market analysis

Data

Following libraries will be potentially used to web-scrape data off Yahoo Finance:

1. yfinance
2. BeautifulSoup (bs4)
3. news-api
4. newspaper3k

Process Outline

• FastAPI for data scraping for stock prices
• JWT Authentication for FastAPI
• Data scraping for stock news
• Dashboard creation using Google Data Studio
• Building lambda functions for news summarization
• Building lambda functions for sentiment analysis
• Deploying Serverless lambda
• Developing UI using Streamlit
• Deploy on Streamlit Cloud

Deployment Details:

1. Language: Python
2. Container: Docker, AWS ECR
3. Cloud ecosystems: AWS, Google cloud platform
4. Cloud tools: AWS Lambda, AWS Cloudwatch, AWS Gateway, GCP Cloud Storage, Google BigQuery, Google App Engine, Google DataStudio
5. Tools for analysis: Google Colab, Anaconda Spyder, Microsoft Azure Visual Studio
6. Other tools and frameworks: FastAPI, Pytest, Streamlit, Airflow

Workflow

Fig. Proposed workflow

References:

1. https://python.plainenglish.io/build-a-stock-data-api-using-web-scraping-and-fastapi-dcbcdbd3d2ec
2. https://zzhu17.medium.com/web-scraping-yahoo-finance-news-a18f9b20ee8a
3. https://www.analyticsvidhya.com/blog/2021/06/download-financial-dataset-using-yahoo-finance-in-python-a-complete-guide/
4. https://www.parsehub.com/blog/scrape-yahoo-finance/
5. https://newspaper.readthedocs.io/en/latest/
6. https://huggingface.co/blog/sentiment-analysis-python
7. https://newsapi.org/docs/client-libraries/python
8. https://fastapi.tiangolo.com/tutorial/first-steps/
9. https://www.youtube.com/watch?v=1zMQBe0l1bM&ab_channel=AbhishekThakur
10. https://docs.streamlit.io/streamlit-cloud/get-started/deploy-an-app
11. https://www.philschmid.de/serverless-bert-with-huggingface-aws-lambda-docker
12. https://pypi.org/project/yfinance/
13. https://www.thepythoncode.com/article/text-summarization-using-huggingface-transformers-python
14. https://huggingface.co/blog/sentiment-analysis-python
15. https://newspaper.readthedocs.io/en/latest/

Project Organization

├── LICENSE
├── Makefile           <- Makefile with commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── docs               <- A default Sphinx project; see sphinx-doc.org for details
│
├── models             <- Trained and serialized models, model predictions, or model summaries
│
├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
│                         the creator's initials, and a short `-` delimited description, e.g.
│                         `1.0-jqp-initial-data-exploration`.
│
├── references         <- Data dictionaries, manuals, and all other explanatory materials.
│
├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures        <- Generated graphics and figures to be used in reporting
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
│
├── setup.py           <- makes project pip installable (pip install -e .) so src can be imported
├── src                <- Source code for use in this project.
│   ├── __init__.py    <- Makes src a Python module
│   │
│   ├── data           <- Scripts to download or generate data
│   │   └── stock_main.py
|   │   ├── news_scrapper.py
|   │   ├── streamlit-app.py
|   │   ├── test.py    
|   │   ├── airflow_dag
|   │   │   └── news_airflow.py
│   │   ├── news_summary
|   |   │   ├── handler.py
|   |   │   ├── get_models.py
|   │   |   ├── dockerfile
|   |   │   ├── serverless.yaml
|   |   │   └── requirements.txt
│   │   ├── news_sentiment
|   |   │   ├── handler.py
|   |   │   ├── get_models.py
|   │   |   ├── dockerfile
|   |   │   ├── serverless.yaml
|   |   │   └── requirements.txt
|   |   |
│   ├── features       <- Scripts to turn raw data into features for modeling
│   │   └── build_features.py
│   │
│   ├── models         <- Scripts to train models and then use trained models to make
│   │                     predictions
│   │
│   └── visualization  <- Scripts to create exploratory and results oriented visualizations
│       └── visualize.py
│
└── tox.ini            <- tox file with settings for running tox; see tox.readthedocs.io

---

Project based on the cookiecutter data science project template. #cookiecutterdatascience

Submitted by: