Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GSoC] 210-Video Recognition using SlowFast #922

Merged
merged 17 commits into from
Jul 4, 2023
Merged

[GSoC] 210-Video Recognition using SlowFast #922

merged 17 commits into from
Jul 4, 2023

Conversation

rajatkrishna
Copy link
Contributor

Video Recognition using SlowFast and OpenVINO

Added the notebook for Video Recognition using SlowFast model in this PR. Demonstrates steps to achieve the following:

  • Run inference on a sample video using a ResNet based SlowFast model pretrained on the Kinetics 400 dataset and downloaded from PyTorchVideo repository
  • Convert above model to OpenVINO Intermediate Representation (IR) format and serialize the same for later use
  • Use OpenVINO Runtime to read, compile and run inference on the serialized model

Issue: #832

@rajatkrishna
Merge branch 'main' into slowfast-notebook
d707b7b
@rajatkrishna
Added notebook for SlowFast video recognition
eeee410 
Notebook demonstrating conversion and inference of the SlowFast model for Action Classification using OpenVino.
@rajatkrishna
Add README for SlowFast Video Recognition notebook
d5f63fe 
Added README describing the objectives, contents and installation steps for the SlowFast video recognition notebook
@rajatkrishna
Update README
c57afa6 
Added entries for the SlowFast video recognition notebook in notebooks overview tables
@rajatkrishna
Add binder badge for SlowFast video recognition notebook
2bc9d58
@rajatkrishna
Styling changes
bc1f489
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@eaidova eaidova added new notebook new jupyter notebook gsoc Google Summer of Code (Prerequisite Task) labels Mar 16, 2023
@adrianboguszewski
Copy link
Contributor

adrianboguszewski commented Mar 16, 2023 •
edited
Loading

Thank you for your contribution. Before we do the review please do the self-check first (tick the item when it's done):

  • each function is described by docstrings and type hints
  • notebook contains explicit descriptions and explanatory diagrams
  • the notebook doesn't use any data (image, video, etc.) that is not CC4.0 licensed
  • there is a README.md file in consistent style (look at other notebooks)
  • the notebook is added to the main README and notebooks/README files
  • there are no grammar, interpunctions or typo issues (use any free tool for that e.g. Grammarly)
  • there are no committed files besides notebook and readme (please use images or videos from data dir)
  • all CI checks passed

We will review only if all the above items are checked.

@rajatkrishna
Fix failed build- missing dependency
0151b4d 
- Added missing dependency causing build to fail
- Updated README
@rajatkrishna
Use proper type classes for type hinting
ae82558 
Use classes from the typing library for type hinting generics to ensure backward compatibility with earlier Python versions
@rajatkrishna
Text and styling changes
602e840 
Fix typos and resolve styling issues in the SlowFast Video Recognition notebook.
@rajatkrishna
Potential fix for failing build- skip torch.hub repo validation
04f8af8 
Skip repo validation for TorchHub models due to known bug in `torch.hub` with Github CI (pytorch/vision#4156)
@rajatkrishna
Copy link
Contributor Author

rajatkrishna commented Mar 17, 2023 •
edited
Loading

Hello @adrianboguszewski

Thank you for the feedback. Please find the status of the self-check items below:

  • each function is described by docstrings and type hints
  • notebook contains explicit descriptions and explanatory diagrams
  • the notebook doesn't use any data (image, video, etc.) that is not CC4.0 licensed
  • there is a README.md file in consistent style (look at other notebooks)
  • the notebook is added to the main README and notebooks/README files
  • there are no grammar, interpunctions or typo issues (use any free tool for that e.g. Grammarly)
  • there are no committed files besides notebook and readme (please use images or videos from data dir)
  • all CI checks passed

@rajatkrishna
Copy link
Contributor Author

@adrianboguszewski

I would also like to bring to your attention that one of the CI tests has failed, which appears to be unrelated to the notebook I am currently adding. Upon further investigation, I have discovered an issue with one of the libraries utilized in that notebook to download a video from YouTube. As a result, I have submitted a fix for the problem (#929). However, the fix involves the addition of the mentioned video to the notebooks/data/ folder.

I would appreciate it if you could take a moment to review the fix and provide any feedback or suggestions you may have.

@adrianboguszewski
Copy link
Contributor

Thanks, @rajatkrishna for the fix :) Please rebase your work now.

We will review your PR when we have some resources. Please note we're reviewing PRs in FIFO style.
Note: The PR must be opened before April 4th, it doesn't have to be reviewed or merged.

@rajatkrishna
Copy link
Contributor Author

Hey @adrianboguszewski,

Just FYI, all changes are in and rebased. CI tests are still failing due to a dependency/notebook unrelated to mine.

@raymondlo84 raymondlo84 added RFR2 Ready for review tech_review: ready Ready for technical review labels Apr 26, 2023
openvino-dev-samples
openvino-dev-samples reviewed Apr 27, 2023
View reviewed changes
notebooks/210-slowfast-video-recognition/210-slowfast-video-recognition.ipynb
@@ -0,0 +1,567 @@
{
Copy link
Collaborator

@openvino-dev-samples openvino-dev-samples Apr 27, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

Line #38.        post_act = torch.nn.Softmax(dim=1)

I dont know can we use numpy instead of torch in this data process pipeline, so we can deploy openvino model without torch dependency


Reply via ReviewNB

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced torch with numpy in the data process pipeline.
The only use of torch.tensor is to feed input to the PyTorch model.

Additionally, CI tests are failing due to unrelated notebooks.

@rajatkrishna
Merge branch 'main' into slowfast-notebook
e06bd18
@rajatkrishna
Removed torch dependency in the data processing pipeline
98a2706 
Use numpy instead of pytorch for processing input data to avoid torch dependency during model deployment.
@rajatkrishna
style fix
5d12987 
fix missing whitespace around arithmetic operator
@eaidova
Copy link
Collaborator

eaidova commented Jul 3, 2023

@rajatkrishna thank you for your contribution, it is great notebook example. Please fix my small remarks and we are ready to merge your notebook :)

@eaidova eaidova merged commit 022480f into openvinotoolkit:main Jul 4, 2023
eaidova pushed a commit to eaidova/openvino_notebooks that referenced this pull request Jul 19, 2023
@rajatkrishna @eaidova
[GSoC] 210-Video Recognition using SlowFast (openvinotoolkit#922)
5f265a3 
* Added notebook for SlowFast video recognition

Notebook demonstrating conversion and inference of the SlowFast model for Action Classification using OpenVino.

* Add README for SlowFast Video Recognition notebook 

Added README describing the objectives, contents and installation steps for the SlowFast video recognition notebook

* Update README

Added entries for the SlowFast video recognition notebook in notebooks overview tables

* Add binder badge for SlowFast video recognition notebook

* Styling changes

* Fix failed build- missing dependency

- Added missing dependency causing build to fail
- Updated README

* Use proper type classes for type hinting

Use classes from the typing library for type hinting generics to ensure backward compatibility with earlier Python versions

* Text and styling changes

Fix typos and resolve styling issues in the SlowFast Video Recognition notebook.

* Potential fix for failing build- skip torch.hub repo validation

Skip repo validation for TorchHub models due to known bug in `torch.hub` with Github CI (pytorch/vision#4156)

* Removed torch dependency in the data processing pipeline

Use numpy instead of pytorch for processing input data to avoid torch dependency during model deployment.

* style fix

fix missing whitespace around arithmetic operator

* Install dependencies in quiet mode

* Add link to dataset
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@openvino-dev-samples openvino-dev-samples openvino-dev-samples left review comments

@eaidova eaidova eaidova approved these changes

Assignees
No one assigned
Labels
gsoc Google Summer of Code (Prerequisite Task) new notebook new jupyter notebook RFR2 Ready for review tech_review: ready Ready for technical review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rajatkrishna @adrianboguszewski @eaidova @openvino-dev-samples @raymondlo84