Download and Kinetics 400/600/700 Datasets #3680

bjuncek · 2021-04-16T20:38:28Z

Addresses #3452.

bjuncek · 2021-04-16T22:49:46Z

@pmeier is there a nice/neat way to read in CSV annotation file and have it be queryable without using pandas?

pmeier

Hey Bruno, thanks for the PR! I did an initial pass. Comments below.

torchvision/datasets/kinetics.py

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

bjuncek · 2021-04-22T21:51:08Z

@pmeier I've fixed all the issues you mentioned, plus removed pandas dependency. Can you please check if all looks well.
Also, additionally, should we add a download test?

pmeier

Mostly style related comments left.

Also, additionally, should we add a download test?

Yes, please. If you need help to set them up, please let me know. Happy to help.

Plus, since this adds new parameters (n_classes and split), we should also include them in our dataset test

vision/test/test_datasets.py

Line 1067 in d7ad62f

class Kinetics400TestCase(datasets_utils.VideoDatasetTestCase):

You can do so by adding a ADDITIONAL_CONFIGS class variable like

ADDITIONAL_CONFIGS = datasets_utils.combinations_grid(
        n_classes=("400", "600", "700"), split=("train", "val")
    )

torchvision/datasets/kinetics.py

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

… into bkorbar/datasets/kinetics

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

pmeier

Looks awesome, thanks Bruno!

fmassa

This looks good to me, thanks Bruno!

I only have one comment regarding the order of the dimensions for a frame. If we are to be deprecating Kinetics400, I think we might as well change the format to perform a .transpose() if we use the legacy format

torchvision/datasets/kinetics.py

fmassa

Thanks!

Summary: * Initial commit * pmeiers comments * pmeiers changes * pmeiers comments * replace pandas with system library to avoid crashes * Lint * Lint * fixing unittest * Minor comments removal * pmeier comments * remove asserts * address pmeier formatting changes * address pmeier changes * pmeier changes * rename n_classes to num_classes * formatting changes * doc change to add ".mp4" to backported class * formatting to correct line length * adding **kwargs to Kinetics400 class * remove urlib request and download the file directly * annotations and files can be already downloaded * test fix * add download tests for Kinetics * users now dont need to provide full path within the root for new Kinetics dataset * linter * Update test/test_datasets_download.py * Update torchvision/datasets/kinetics.py * revert whitespace (3680#discussion_r626382842) * addressing annotation_path parameter which is unnecessary * Update torchvision/datasets/kinetics.py * Update torchvision/datasets/kinetics.py * kwargs update * expose num_download_workers as public * swap os.isfile with check_integrity * nit on private things * special case if there are no default arguments * revert changes to kinetics400 test case for BC * add split_folder changes and support for legacy format * pmeiers suggestions * pmeiers suggestions - root comment * pmeiers comments - annotation attribute remmoved * pmeiers suggestion * pmeiers suggestion * pmeiers suggestion * pmeiers suggestion * Update torchvision/datasets/kinetics.py * Update torchvision/datasets/kinetics.py * Update torchvision/datasets/kinetics.py * Update torchvision/datasets/kinetics.py * Update torchvision/datasets/kinetics.py * Update torchvision/datasets/kinetics.py * minor debugging * nit picks * only include public kwargs into defaults * add _use_legacy_structure in favour of **kwargs * add type hints for Kinetics400 * flake8 * flake8 * flake8 * rename to make thigs clearer * permuting the output Reviewed By: fmassa Differential Revision: D29097736 fbshipit-source-id: 1de2119e82eadbbba682f0897f03aba3929e3604 Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Philip Meier <github.pmeier@posteo.de> Co-authored-by: Francisco Massa <fvsmassa@gmail.com>

Initial commit

5e60e01

facebook-github-bot added the cla signed label Apr 16, 2021

pmeier changed the title ~~#3452 Download and Kinetics 6/700 Datasets~~ #3452 Download and Kinetics 400/600/700 Datasets Apr 19, 2021

pmeier requested changes Apr 19, 2021

View reviewed changes

bjuncek and others added 9 commits April 20, 2021 14:48

pmeiers comments

62c5e05

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

pmeiers changes

090e526

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

pmeiers comments

9403bcf

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

replace pandas with system library to avoid crashes

e08cf09

Lint

29a4f03

Lint

8cd5209

fixing unittest

a6d2490

Merge branch 'master' into bkorbar/datasets/kinetics

c2076a7

Minor comments removal

93d1444

pmeier requested changes Apr 23, 2021

View reviewed changes

bjuncek and others added 14 commits April 30, 2021 10:26

pmeier comments

9e3d3f3

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

remove asserts

139ec6d

address pmeier formatting changes

33a9f98

address pmeier changes

abdd2f6

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

pmeier changes

1460886

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

rename n_classes to num_classes

7b06906

Merge branch 'bkorbar/datasets/kinetics' of github.com:bjuncek/vision…

62f7fd5

… into bkorbar/datasets/kinetics

formatting changes

e76f4ab

doc change to add ".mp4" to backported class

0a8f216

formatting to correct line length

94a40aa

adding **kwargs to Kinetics400 class

c585a5f

remove urlib request and download the file directly

8cacd80

annotations and files can be already downloaded

802f8f9

test fix

af70e5f

bjuncek and others added 10 commits May 11, 2021 19:16

Update torchvision/datasets/kinetics.py

5a7db27

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

Update torchvision/datasets/kinetics.py

44030ee

Co-authored-by: Philip Meier <github.pmeier@posteo.de>

minor debugging

ce5f80b

nit picks

803bab1

only include public kwargs into defaults

6e64bb6

add _use_legacy_structure in favour of **kwargs

8b64d1d

add type hints for Kinetics400

94b21cc

flake8

f803946

flake8

b39646a

flake8

c47c309

pmeier mentioned this pull request May 18, 2021

video datasets (UCF101, Kinetics400) download error (downloading nothing) #3854

Closed

bjuncek and others added 2 commits May 24, 2021 10:47

rename to make thigs clearer

18ad36d

Merge branch 'master' into bkorbar/datasets/kinetics

d0fa6f4

pmeier approved these changes May 25, 2021

View reviewed changes

bjuncek mentioned this pull request Jun 4, 2021

Add verbose mode for dataset constructors to print directory scan stats + add a warning when video loading fails #3282

Closed

Merge branch 'master' into bkorbar/datasets/kinetics

61334f0

fmassa reviewed Jun 7, 2021

View reviewed changes

torchvision/datasets/kinetics.py Show resolved Hide resolved

bjuncek and others added 3 commits June 8, 2021 15:00

permuting the output

12b76d7

Merge branch 'master' into bkorbar/datasets/kinetics

89b9bee

Merge branch 'master' into bkorbar/datasets/kinetics

1ee00b2

fmassa approved these changes Jun 10, 2021

View reviewed changes

Merge branch 'master' into bkorbar/datasets/kinetics

a22e4e7

fmassa added enhancement module: datasets new feature and removed enhancement labels Jun 10, 2021

fmassa merged commit 8ea04d1 into pytorch:master Jun 10, 2021

pmeier mentioned this pull request May 20, 2022

Use Kinetics instead of Kinetics400 in references (#5787) #5952

Merged

NicolasHug mentioned this pull request May 20, 2022

Add output_format do video datasets and readers #6061

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download and Kinetics 400/600/700 Datasets #3680

Download and Kinetics 400/600/700 Datasets #3680

bjuncek commented Apr 16, 2021 •

edited by pmeier

Loading

bjuncek commented Apr 16, 2021

pmeier left a comment

bjuncek commented Apr 22, 2021

pmeier left a comment

pmeier left a comment

fmassa left a comment

fmassa left a comment

Download and Kinetics 400/600/700 Datasets #3680

Download and Kinetics 400/600/700 Datasets #3680

Conversation

bjuncek commented Apr 16, 2021 • edited by pmeier Loading

bjuncek commented Apr 16, 2021

pmeier left a comment

Choose a reason for hiding this comment

bjuncek commented Apr 22, 2021

pmeier left a comment

Choose a reason for hiding this comment

pmeier left a comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

bjuncek commented Apr 16, 2021 •

edited by pmeier

Loading